Abysmal synchronization performance needs attention

Synchronization performance in IDEA is abysmal but the official line seems to be: "Unfortunately there is nothing we can do to speed up IDEA working on network drives. Our official recommendation is to avoid them at all costs."

Many corporate environments dictate network drives. This is such a serious issue that JetBrains needs to fundamentally re-examine their approach on this issue. I'm contracting at a company where people are fee to use any IDE they want including having the company pay for IDEA. Well, guess what-- Eclipse performance is fantastic on exactly the same project. IDEA performance is so bad that even IDEA fans-- much less people willing to explore IDEA-- are against using running it. Having the project local isn't an option as the build and deployment both have UNIX dependencies.

Here is a typical issue file on this:
http://www.jetbrains.net/jira/browse/IDEA-10407

You will be getting new issues forever on this issue if you don't address it and you'll lose lots of IDEA sales and even lose some existing mindshare due to this problem.

Can you not come up with a faster alternative? A different way? Anything? Perhaps only immediately synch the files actually in the editor then do a low priority background thread synch on the rest?

I know JetBrains can solve this problem if they have the support of management and if they set their minds to it. This is a blocker of sales and erodes mindshare.

13 comments
Comment actions Permalink

I haven't noticed that synchronization performance is 'absymal' but I don't have Eclipse configured for the same project so I can't make a direct comparison.

I agree 100% that the synchronization should be in the background. The 'Synchronize on frame activation' should really be off by default since I think many are annoyed by the behavior since it locks the IDE.

With dual-core desktops coming into the mainstream -- in 2007 I would expect most professional developers to be using dual core machines -- IDEA itself will start looking unprofessional if long running tasks are running in the foreground locking up the GUI.
All long running tasks should default to the background.

In the case of synchronization, the cpu isn't even at 100% since much of time is the I/O wait time communication with the drive, so I would think that even single-core desktops, esp. those with hyperthreading, would still be responsive while synchronization is happening in the background.

My personal biggest issue with the file synchronization is that I every time you execute an Ant task, it synchronizes. This wastes alot of my time and is completely pointless most of the time. I don't use IDEA to compile or deploy my code; Everything is defined in the ant build.xml files. The only time I need IDEA to sync up is if I want to Run/Debug from within IDEA.

I have mentioned before about giving more control over what is synchronized. For instance, I would like to synchronize all open editor tabs on frame activation. Also, during the full synchronization, I would like for IDEA to sync up all open Editor tabs plus all files mentioned in any open Results pane, e.g. search/inspect results, first before synchronizing the rest of the workspace. This will given the appearance of speed, even if it takes the same time as before.

Reference:
IDEA Background Tasks (As of 5261)
http://intellij.net/forums/thread.jspa?threadID=222675

0
Comment actions Permalink

Hi Alex. The abysmal performance is for projects with network drives. It can be painstaking-- Idea frozen for over a minute!-- while Eclipse hums along without skipping a beat. Imagine a situation where Corporate dictates networked drives & anti-virus and won't give users admin rights to their Windows dev machines and the projects must live on the network drives.

JetBrains can easily solve this by pushing the synch into the background and as you mentioned too, go ahead and do the files open in the editor immediately (but even that should be non-blocking). JB, what is preventing you? Is the current approach something designed and written by a JB founder and considered sacred, untouchable? JB, please, for your own sake, address this.

0
Comment actions Permalink

It's fine with me if Idea can be made more performant on network drives.
I consider this however as a workaround for an insane practice that might be mandated by companies where developer productivy is less important that rigid bureaucracy - which is traditionally not the market that Idea aimed for.
But of course with 6.x Idea's target shifted to the big enterprise...

My only plead is that Idea should use file watching just like it does now for local file systems.
I consider it as one of the biggest Eclipse shortcomings that I constantly have to refresh the project view, rebuild my workspace, update VSS status, get bothered with "File has changed do you want to reload" and other such stupid idiosyncracies.

Idea just picks up whatever changes occured at whatever files and is happy with that.

Of course I have the "luxury" of a local hard drive.

0
Comment actions Permalink

I don't disagree with you about it being a less than desirable corporate policy, but these companies purchase bundles of IDEA licenses in 50+ sizes so their needs are important too. JetBrains can solve this problem for the corporate behemoths as well as make it continue to work great for a nimble user like you.

0
Comment actions Permalink

Hi Alex. The abysmal performance is for projects with
network drives. It can be painstaking-- Idea frozen
for over a minute!-- while Eclipse hums along without
skipping a beat.


That's because Eclipse doesn't automatically synchronize with the filesystem. Try using eclipse on a network filesystem, and asking it to refresh the whole workspace. Including subversion administrative files, my project contains around ~200,000 files spread over ~60,000 directories. Synchronizing the whole workspace in eclipse over a network drive can take 20 minutes or more.

Anyways, I'm with you here. Jetbrains should try to solve this problem more elagantly. For instance, IDEA could start a background update, as you suggested, and present the user a "File changed in memory and on disk" dialog, should this situation happen.

0
Comment actions Permalink

Background synchronization is there for ages. The issue with locking upon network drives synchronization should be fixed in 6.0.3.

0
Comment actions Permalink

My personal biggest issue with the file
synchronization is that I every time you execute an
Ant task, it synchronizes. This wastes alot of my
time and is completely pointless most of the time. I
don't use IDEA to compile or deploy my code;
Everything is defined in the ant build.xml files. The
only time I need IDEA to sync up is if I want to
Run/Debug from within IDEA.


Completely agree. I am also working for a "behemoth" corporation and we are using Clearcase with dynamic views (essentially network drives). Every time after I run an Ant target, I have to have a tea break, because it takes about 5 minutes to sync my project and most of this time IDEA is unusable. Please allow us to configure sync settings on a "per Ant file", or better "per target" basis.

0
Comment actions Permalink

I would really appreaciate your comments on 6.0.2 performance with CC dynamic views: I've discovered file watcher is able to watch MVFS, so things should speed up.

0
Comment actions Permalink

Background synchronization is there for ages.


That explains why I been experiencing abysmal performance synching for so long then.

Taking a tea break every time you run an ant script and then switch back into IDEA should really should not be the work around for this problem - especially if you prefer coffee.

It is definitely not just networked drives that are the issue. It has been a big problem on Mac for since 5.x using local drives but having a project that is modified via Ant.

Hopefully, the solution won't be to make synchronization manual, but rather have options that reduce what goes on during synchronization so that it goes faster if you want less. For example, I definitely still want file changes to be synched and compile errors to be detected, but other code suggestions and features I can live without unless I explicitly ask for them.

Spencer

0
Comment actions Permalink

Is it still an issue with 6.0.2?

0
Comment actions Permalink

Unfortunately, CC support in IDEA 6.0 deteriorated to the level, where I considered it inadequate and rolled back to 5.5. I am simply not going to take those pains again and observe exceptions in CC plugin on each and every check-out or check-in. I shall wait until a properly working CC plugin is available for IDEA 6. Sorry, guys. Using ClearCase is a pain in itself.

0
Comment actions Permalink

First, I love IDEA. I've been using it since 3.0.

This thread summarizes my experience and disappointments well.
And thanks to the "UI locking" fix in 6.0.3 (http://intellij.net/forums/thread.jspa?messageID=5177270??), IDEA 6.0 on windows now behaves as it did in 4.5 and as it behaves on linux. File sync still takes a very long time across network drives but at least I can keep working while it does it in the background. Thanks!

To give JetBrains an idea of the pain I had, file sync takes over 2 minutes on Windows with my project of over 5,000 files. That's largely because CIFS is so inefficient. On linux/NFS, the project takes less than 30 seconds to sync. (Both of my win & linux PCs mount the same network drive from a NAS server).

But I bet more can be done to speed up file synchronization. I did some network captures and it appears that IDEA is checking each file in my project tree one-at-a-time i.e. it appears that IDEA drills down each directory from the root then goes to the next directory, serially.

So how about if IDEA uses multiple threads to synchronize multiple directories in parallel? Does that sound feasible?

Theoretically, 10 parallel threads could synchronize my project on windows in about 12 seconds (versus 2 minutes)! And my NAS server could easily handle parallel file requests.

0

Please sign in to leave a comment.