What's up with perforce integration? Works bad for me.

I would have to agree. I've been using CVS and SVN for years under IDEA and have been thrilled with it. Not unexpectedly, Perforce just blows.

I understand why, it's because there is no demand for it. Perforce is an SCM that time has passed by. Anyone still using it is either a) stuck with it because they have this feeling that they need the history in their files (they don't, they just need the relationships between the files and a means of tracking future changes for merges), or b) stuck with it because someone in their company has a poor understanding of priorities and thinks that they own SCM, they can't do their work without it, and the rest of engineering be damned. I've never seen an engineering organization where a majority of the individual contributors prefer Perforce. It will likely be a snowy day in hell when such a team ever raises it's head, but I'll be packing my snowboard for that day, just waiting for some freshies.

Having said that, for as long as I have been sitting here typing, the IDEA Perforce integration has been in the background doing some kind of a complete scan of the directory tree, sending out a command to the server every ten seconds or so, and otherwise using 100% of my processor so I can't get anything done.

Finally, it's done. Better go get something done before a whole hour is lost for a 30-second change.

Hello Brian,

I would have to agree. I've been using CVS and SVN for years under
IDEA and have been thrilled with it. Not unexpectedly, Perforce just
blows.

I understand why, it's because there is no demand for it. Perforce is
an SCM that time has passed by. Anyone still using it is either a)
stuck with it because they have this feeling that they need the
history in their files (they don't, they just need the relationships
between the files and a means of tracking future changes for merges),
or b) stuck with it because someone in their company has a poor
understanding of priorities and thinks that they own SCM, they can't
do their work without it, and the rest of engineering be damned.

Well, in fact JetBrains uses Perforce for developing IntelliJ IDEA and TeamCity,
and we're using it not because we're stuck with it, but because it matches
our needs very well. (Some of more well-known companies which use Perforce
are Microsoft and Google.)

Having said that, for as long as I have been sitting here typing, the
IDEA Perforce integration has been in the background doing some kind
of a complete scan of the directory tree, sending out a command to the
server every ten seconds or so, and otherwise using 100% of my
processor so I can't get anything done.

Any details? Which build of IDEA does this happen with? Did this happen when
you just opened the project, or after
some other operations? Which exactly commands does it execute? (You can enable
logging in the Perforce settings to see this.)

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Hi Dmitry,

The problems I was seeing with the Perforce integration are incredibly numerous, but shortly after posting my little love letter there, it struck me that maybe it boiled down to the size of the changelist I was working on. There was about 3500 files in it, most of them moves. After committing it, things got more manageable from a time perspective, but prior to that, I would wait about 30 seconds to a minute for just about anything that touched the filesystem. An example would be 30 seconds to create a new folder, an event that Perforce typically doesn't care about.

Unfortunately, I've been doing a lot of these large-scale directory layout changes lately and I've never had issues like this with SVN or CVS in the past.

I didn't know about the logs and I've turned that on. When I start having more problems, I'll start monitoring it and filing bugs. When things get bad, I have noted that the network traffic is mostly dead with bursts at regular intervals (maybe every ten seconds) during these periods of high CPU. Sometimes I have to walk away for 30 minutes to let the thing catch up. My feel for it is that the integration has some hotspots with large changesets, but I'll post more when I get it.

Thanks for the concern and interest.

Brian

Hello Brian,

The problems I was seeing with the Perforce integration are incredibly
numerous, but shortly after posting my little love letter there, it
struck me that maybe it boiled down to the size of the changelist I
was working on. There was about 3500 files in it, most of them moves.
After committing it, things got more manageable from a time
perspective, but prior to that, I would wait about 30 seconds to a
minute for just about anything that touched the filesystem. An
example would be 30 seconds to create a new folder, an event that

I'll admit that we didn't optimize the plugin specifically for the case of
having a large number of files opened for editing - here we usually have
at most a few hundred files opened. It may well be possible to optimize the
plugin to handle this better.

Also, you can try the new "offline mode" feature added in recent EAP builds.
Even when you are not offline, it should provide much better performance
(at the cost of losing a few features like tracking unversioned/locally deleted
files).

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Hi Dmitry,

Since I've been dead for about a half hour waiting for this thing to catch up, I might as well write with some new information I've got on this.

At this point, I've got all this running on a faster machine (2.4GHz quad desktop) and it's had all four cores at 30% for the whole half hour. I started watching the log and it was spraying out fstat log entries. I tried checking in one directory, and it sat with a dialog open for about 20 minutes saying it was waiting for VCS synchronization. At this point I have about 3,500 files open in perforce.

I'm honestly in a really bad place with this thing right now. In order to get around the issue of the VCS time, I checked in all the files, even though they were nowhere near ready, since I am the only one working on this branch. Now I find I need to roll back, and I have no clue what is in there.

I understand that your organization as well as other organizations may rely on perforce, but that doesn't make it a solid or desirable architecture. Back when it was the only VCS around that could do atomic commits, the issues of being tethered to the server were manageable because people did not have this much code to work on. Ironically, tools like IDEA with their excellent refactoring capabilities expose the weaknesses in perforce. While there may be better merge tools for perforce than there are for Subversion, this is not a strength for perforce, it is a market opportunity for Subversion and tools vendors that work with it.

For those tuning in late or hitting this through a search engine, the damning architectural issue with perforce is that it is tethered to the server, and like any network application, serialization is always the slower to the network than it is to disk by orders of magnitude. To boot, if you don't have network access (like in a cafe or on an airplane), you need to jump through hoops to get your files synchronized when you get back to a connection. Perforce inc says that they have "features" that others don't have like "remote proxy servers" for distant sites, but the reality is, because one isn't tethered to the server, there is no need for a proxy server. It goes on and on. Do yourself a favor and avoid it!

Hello Brian,

At this point, I've got all this running on a faster machine (2.4GHz
quad desktop) and it's had all four cores at 30% for the whole half
hour. I started watching the log and it was spraying out fstat log
entries. I tried checking in one directory, and it sat with a dialog
open for about 20 minutes saying it was waiting for VCS
synchronization. At this point I have about 3,500 files open in
perforce.

Which build does that happen with? Selena M1 and later EAP builds no longer
use fstat to update the changes view. Please try them if you haven't yet.

For those tuning in late or hitting this through a search engine, the
damning architectural issue with perforce is that it is tethered to
the server, and like any network application, serialization is always
the slower to the network than it is to disk by orders of magnitude.
To boot, if you don't have network access (like in a cafe or on an
airplane), you need to jump through hoops to get your files
synchronized when you get back to a connection.

Once again, let me repeat my recommendation to try the Perforce offline mode
added in recent Selena EAP builds. It solves both of these problems (performance
and lack of connection), and I believe that the solution works quite well.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Once again, let me repeat my recommendation to try
the Perforce offline mode
added in recent Selena EAP builds. It solves both of
these problems (performance
and lack of connection), and I believe that the
solution works quite well.

Hi thanks for the suggestion. The last time I tried Selena, the IDEA plugin for Maven was not so hot. Maybe I am missing something and it's not necessary to use it with IDEA any more because of the built-in integration with Maven, but I haven't figured out how to use it yet, having a hard time finding docs on it. But I will give it a try again soon, thanks for your suggestion! :)

Yes, Selena is faster when it works, but sometimes it just goes out to lunch, churning away on processor (~110% of one core, 27% of all four cores) and not responding. Occasionally, I get the spinning beach ball and no processor usage (looks like a deadlock). If I kill the process and restart it, things work for a while, then go back to neverland after a few transactions (rename file, replace a string in some other files.)

Older EAP builds used to have YourKit integrated for measuring statistics, but I guess nobody used it enough to make it worthwhile. Could use it now though, would be nice to supply some more data. I'm not doing anything else with my time but reading blogs.

I've never had this kind of problem with CVS or Subversion.

Hello Brian,

Yes, Selena is faster when it works, but sometimes it just goes out to
lunch, churning away on processor (~110% of one core, 27% of all four
cores) and not responding. Occasionally, I get the spinning beach
ball and no processor usage (looks like a deadlock). If I kill the
process and restart it, things work for a while, then go back to
neverland after a few transactions (rename file, replace a string in
some other files.)

Older EAP builds used to have YourKit integrated for measuring
statistics, but I guess nobody used it enough to make it worthwhile.

It's definitely used enough - it's just disabled by default in release builds
(milestone and final release). Please add the following line to idea.exe.vmoptions
to enable YourKit:
-agentlib:yjpagent

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"