5594 feedback Project file scanning still an a big problem

See http://www.jetbrains.net/jira/browse/IDEABKL-2735 for my request to treat generated code differently than 'normal' source code.

In my case, the generated code is only part of the problem. There's lots of scanning going on including all the jar files our ant build process touches.

Same problem here.

After each build, failed or not, I'm waiting 15ish minutes for every library jar to be scanned multiple times. This isn't generated code, I'm taking struts.jar binaries and the like.

Even a cancel button on the 'Loading Files' box would be vastly better.

While I can't promise there won't be any scanning phase, showing multiple progress dialogs is definitely not normal. I'll try to see if I can reproduce this.
BTW, are you sure that the same jars are scanned several times?

BTW, is it tld index that is frequently rebuilt?

yes, it's the tld, then after each one, it digs back into the libraries, then back to tlds, then back libraries

I'm trying to get a more specific story, but after downloading build 5622, my ant tool window is gone!

Does it dig into jar files without tlds?

well, no, doesn't seem to.

the pattern looks to be that it scans the generated code libraries, then starts digging into the binary jars, looking at the tlds, then back to my project code, looking through tlds, then binary jar tlds, project code tlds, then quickly scan some recently compiled class files, then start back to the project code tlds, binary tlds, etc

it looks through the same binary tlds and same project code tlds MANY times.

I'm having the same issue as well. I've got too very large multi module projects. Each module may reference the same jar(s) that are in use by other modules. Any change can create a huge storm of file/jar scanning that is definately repeated over and over again. TLD s are the most obvious rescan, though I'm unsure if there are others being scanned as well.

Fixed.

Sweet! I look forward to using it.

Thanks,
MIke

I'm glad to see this topic, because I was about to start my own. Hopefully I'm not hijacking it. We too have a very large project, with 40+ modules. We have several hundred core code-generated objects. But that is beside the point...the code generation amplifies the problem, but it does not cause it.

Here is my real problem: we have paid for IntelliJ licenses. Money is not really the issue. Yet, I have about half of my team, none of them being idiots, are now willingly choosing Eclipse over IDEA. Why? Prior familiarity is only part of it. I would force them to use IntelliJ if it were just a matter of re-learning keyboard shortcuts. But that is not the problem. The problem is simply the foreground scanning/indexing. Eclipse itself starts up very quickly and all build/indexing operations can be sent to the background. This is a very old topic in the IntelliJ community I think. And I am afraid that most of you and most of your customers do not organize their code, or do not have such a large project, like we do, and thus do not experience this amount of pain. The indexing process does not just pop up once in a while. It happens all of the time.

Really, it would be very beneficial if
a) JetBrains acknowledged in some marketing material or here in the forums that very large projects are a main focus and something is being done to satisfy that use case, and
b) You implemented background indexing, pure and simple.

Not long ago, I posted that for IntelliJ to continue to be on
the cutting edge of a response UI and optimized for dual-core
and quad-core workstations, they must change IDEA so that
alll long running tasks are run in the background by default
and you should be able to have multiple tasks running in the background.

Demetra EAP has a few improvements along these lines, but it seems
Eclipse is ahead of them, which is disappointing.

IDEA Background Tasks (As of 5261)
http://www.intellij.net/forums/thread.jspa?messageID=5107349

-

All long-running tasks should:
1) Run in the background by default.
2) Don't steal the focus or interfere with user while running or after completion
3) Don't show progress bar covering the editor.
progress should be shown the status bar or in a status/progress toolwindow
4) If the results window for the task is being updated to reflect the currently
found results, it should also give indication that the command is still running
and have toolbar button to stop it.

Users never want to see a modal progress bar; And the users with dual core
workstations are even more annoyed because they have one cpu sitting idle
while IDEA is blocking input! I personally use the Inspect Code feature less
than I would like because I can't launch it in the background while I do other
work.

I have seen Jetbrains make some positive changes in this area in Demetra, and I hope
more are coming.

Demetra has a new option General->Search in the background. If enabled, when you do
a Search, Search & Replace, Structural Search, or Structural Search & Replace (but not
a Find Usages), the modal progress dialog is not shown. Instead, the progress is displayed
in the status bar along with a red button to cancel the operation.

Currently, it looks like IDEA is behind Eclipse in this area. Eclipse did a major overhaul
in 3.0 M9 to add a new UI framework for background operations. See
http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.platform.doc.user/gettingStarted/qs-responsiveUI.htm
In particular, Eclipse properly shows two or more background tasks in the Progress Dialog or Progress
View.
-

Click for more: http://www.intellij.net/forums/thread.jspa?messageID=5107349

JetBrains: No comment on this?

Would just like to add my vote - this is a major issue and imho makes Eclipse a better choice right now

The issue with multiple progress dialogs is fixed in 5661.
Please check the next EAP, where I hope I've fixed the exception on project synchronization.

Thanks Eugene, that is good news. But I think we are all still more interested in the bigger picture: background indexing. I understand it is far too late in the game for the current EAP but we haven't heard ANYTHING from JetBrains about this. People are writing to you telling you that they are switching to Eclipse solely because of this. What is your position?

Thanks.

I can verify that the issue with multiple progress dialogs is fixed in 5661.
But the overall problem still persists. Project scanning performance is a big problem, both in terms of the time it takes how how IDEA behaviors when it is happening.

When you change the file system of a project with, say and Ant task executed outside of IDEA, IDEA is it extremely unresponsive. It is so bad sometime it looks like IDEA has hung. You need a dialog in this case, but the real issue is not the dialog or lack of them (although the multiple dialogs where really bad) but the overall responsiveness or the lack of it.

• I'd be happy to be able to turn a bunch of features for faster response. Maybe there could be a fast mode setting or something that make it easy to trade features for speed.

BTW, I am MacBookPro with 2 gigs of memory - not an old clunker and performance is still poor.

In general the performance in the betas has made it really tough to work with. But I recognize that they are betas and that's part of the deal. On the other hand, I am using the beta primarily because 5.1 project scanning issues made it almost unusable in my environment.

I am hoping that the performance continues to improve as the march to the release continues. However, I worry that performance on the Mac might be overlooked as it is a minority platform. I hope that does not end up being the case.

I can't speak about company position here, but my personal feelings are that it is possible to make at least some of the indexing background after project startup. This should not certainly indicate we are going to do it in 6.0, but I strongly think some work in this direction must be planned for future release.

I would agree that project scanning is still an issue in the latest EAP release (5692) that I downloaded and installed today.

I think it is actually even worse than it was with the previous EAP now. I know that I had stepped away from my dev box for about a half hour and I came back and the idea.exe process had over a 100,000 handles open. Idea had become unresponsive and required to kill the process.

I will try and continue to use 5692 for the rest of today, but if things don't get better with it, I don't know what to do anymore with Idea 6.0. :(

Plus this is even on a pretty beefy Windows dev box with 2 gigs and 3Ghz HT P4. This is on one of our larger projects that has about 2000 class files and say 400 jsps and 20 libraries. VCS is Source Safe 6.0c

The issue you face is the bug in jdk, not in IDEA. It is fixed in jdk 5.0 update 9 that we are eager to update to. Meanwhile you can verify the problem does not persist on latest jd6 build (it does on beta 2 though).

That is good to hear since I just always download the JDK with the windows version since I already have 4 other JDKs on my box and really hate to have to point Idea to one of those since they can change weekly and hate to break my ide just because the JDK changed on my box for something else. :)

I will go an manully point it to J6 and try.

Something I (re)discovered today is that Anti-Virus apps which do automatic file scanning can completely kill IDEA's performance. That whole shutting down take 30 seconds problem was because my Anti-Virus was scanning every one of the thousands of little files that IDEA pokes as it shuts down. Once I excluded my .IntellJIdea directories from the Anti-Virus file scanning, IDEA's shutdown and subsequent startup was much quicker.

I could have sworn that I had done this before but it looks like my Anti-Virus settings may have been reset. Since it's a corporate setting, I wouldn't be surprised if this was the case. If you're experiencing slowdown problems like this, you may want to double-check that your Anti-Virus is configured to exclude these directories.

Yes, it makes a big difference if ALL IDEA's files are exempted from virus scanning.

At work, we use an exempted DEV directory for all Java stuff - and in Windows it's worth remembering to move the IDEA system and config folders from their default Documents and Settings location to an exempt directory (I put them into the IDEA directory). This can be done by editing the IDEA properties file. It is important because the IDEA plugin and cache files are stored there and are being read/written almost continuously.

I have just tried Idea 6.0Beta 5706 today with Mustang (1.6.0.b99) build and I still get the huge handle usage as it launches a new SS.EXE and kills it as part of opening a project (apparently idea checks each file in source safe or something)?

Anyhow, in my one project it went up to 20,000 handles. I thought 1.5.09 (which I have not tried) or Mustang latest build was suppose to fix this problem?

I must be missing something.

Is it the following JVM bug? The fix appears to be postponed to java 5 update
10. (And I assume you tried an update 8 build, since update 9 is not out
of the doors yet). However, it's marked as fixed for Mustand b90, so perhaps
you experience something else.

"Event Handle Leak while using JNI"
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6399321

I think it is, but I thought it was suppose to be fixed in Mustang B90+ (I am using b99)?

It definitely exhibits the signs explained in the Bug ID you listed. I don't have 1.5 update 8 (or 9) on this machine. I just happened to have the latest Mustang binary build on here and tried it with the new idea release and I saw that it was no better. :(

Hopefully update 9 is released soon and it can just be bundled with a new EAP build! That would be the best solution. ]]>

1 on this issue for me. I will say that it has been around since the 4.x days for our build environment - 3 medium-to-large size projects. I have at times turned off file synchronization because of the "take over" of the editor window by IDEA upon syncrhonization. There seems to be several levels of scanning that takes place:

1. Source code
2. Javadocs
3. JAR files

All of which cause IDEA to become unsable during sync updates.

-Joe

1 on this issue for me. I will say that it has been around since the 4.x days for our build environment - 3 medium-to-large size projects. I have at times turned off file synchronization because of the "take over" of the editor window by IDEA upon syncrhonization. There seems to be several levels of scanning that takes place:

1. Source code
2. Javadocs
3. JAR files

All of which cause IDEA to become unsable during sync updates.

BTW - Here are my JVM settings: -Xms256m -Xmx512m -XX:MaxPermSize=256m -Xbootclasspath/p:../lib/boot.jar -ea -agentlib:yjpagent

Please check the next build for better performance of external changes.