Memory usage

I know this has been questioned many times but I'm hoping I can get some insight here. I spend my time on a large commercial application. Customers are generally customized so we maintain multiple CVS branches. I have an IDEA project set up for each. The project is large - around 400 classes. I have IDEA start up with a minimum of 64M and max 256M heap. Now, I can open-up one project and watch memory go up to around 100-120M. Open up a second in another frame and watch memory grow to almost 200M. Now, of course, we get into the misery of client-side java - GC which basically consumes my machine. But, if I close one project (and it's frame) I only free up a small amound of memory - maybe 20-30M. In the remaining frame I can close the project, so all that's left is an IDEA frame with no project. At best I'll go down to 160-175M. Why is IDEA holding on to so much memory with no open projects? I can hit the GC button all I want - but IDEA will not let go of the memory. Does anyone have any insight? I'm running a new Latitude D600 with Win2K - 512M - and with 2 projects open - it's gets really bad, really fast - constantly lagging being my key/mouse movement and garbage collecting continuously. Any info or suggestions would be greatly appreciated.
Thanks,
Scott

21 comments
Comment actions Permalink

Which build are you using?

Bas

0
Comment actions Permalink

It's not just you. I see the same behavior.

Norris Shelton
Sun Certified Java Programmer




Scott Johnson wrote:

>I know this has been questioned many times but I'm hoping I can get some insight here. I spend my time on a large commercial application. Customers are generally customized so we maintain multiple CVS branches. I have an IDEA project set up for each. The project is large - around 400 classes. I have IDEA start up with a minimum of 64M and max 256M heap. Now, I can open-up one project and watch memory go up to around 100-120M. Open up a second in another frame and watch memory grow to almost 200M. Now, of course, we get into the misery of client-side java - GC which basically consumes my machine. But, if I close one project (and it's frame) I only free up a small amound of memory - maybe 20-30M. In the remaining frame I can close the project, so all that's left is an IDEA frame with no project. At best I'll go down to 160-175M. Why is IDEA holding on to so much memory with no open projects? I can hit the GC button all I want - but IDEA will not let go of the memory. Does anyone have any insight? I'm running a new Latitude D600 with Win2K - 512M - and with 2 projects open - it's gets really bad, really fast - constantly lagging being my key/mouse movement and garbage collecting continuously. Any info or suggestions would be greatly appreciated.
>Thanks,
>Scott

>

0
Comment actions Permalink

I'm using 4.5.4 build 2253 - but I've seen this behavior since I started with IDEA - 3.0. It just continues to get worst as the product grows.

0
Comment actions Permalink

I see this all the time too. In my opinion this is the biggest problem of IDEA right now. It frequently goes into garbage collection mode because it runs out of memory very quickly. And garbage collection sometimes takes minutes. Waiting for it to "wake up" while I'm in the middle of typing, or compiling, or something else is very annoying.

0
Comment actions Permalink

Excactly - I have to shut down IDEA several times a day, just to get memory back. As I've indicated above, closing all projects doesn't matter - you have to shut down IDEA completely. And the GC does take your entire machine, sometimes for a minute or two - you can't even type. When under a dealine and making mods to multiple branchs, which to me are multiple projects, it can be extremely frustrating. I'm hoping this is something that can become a top priority in future releases. I thoroughly understand the nature of Java and GC but with all projects closed, it seems as though most memory should be released, which it clearly is not. Glad to hear it's not must me anyway.

0
Comment actions Permalink

That's obvious memory leak aka bug...
In Irida EAP (http://www.intellij.net/eap) we have embedded YourKit memory
and CPU profiling into production thus if anyone feels IDEA has a memory
leak or performs ceratain operation for way too long it's enough to press
the button on the IDEA toolbar and send created profiling snapshot to the
developers. That's great of help for us indeed. It makes solving many of
such problems just "open, see what the garbage is going on, fix" instead
of suffering guessing and poor assumptions about what may be happening at
client side.

-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

I know this has been questioned many times but I'm hoping I can get
some insight here. I spend my time on a large commercial application.
Customers are generally customized so we maintain multiple CVS
branches. I have an IDEA project set up for each. The project is
large - around 400 classes. I have IDEA start up with a minimum of
64M and max 256M heap. Now, I can open-up one project and watch
memory go up to around 100-120M. Open up a second in another frame
and watch memory grow to almost 200M. Now, of course, we get into the
misery of client-side java - GC which basically consumes my machine.
But, if I close one project (and it's frame) I only free up a small
amound of memory - maybe 20-30M. In the remaining frame I can close
the project, so all that's left is an IDEA frame with no project. At
best I'll go down to 160-175M. Why is IDEA holding on to so much
memory with no open projects? I can hit the GC button all I want -
but IDEA will not let go of the memory. Does anyone have any insight?
I'm running a new Latitude D600 with Win2K - 512M - and with 2
projects open - it's gets really bad, really fast - constantly lagging
being my key/mouse movement and garbage collecting continuously. Any
info or suggestions would be greatly appreciated.Thanks,Scott




0
Comment actions Permalink

OK, great and thanks - I'm assuming you work for JB - but, as I've said above, this is not new. Search the forums for memory usage - you'll find many reports going way back. I've just given you a very simple scenario. Open up two large projects (particularly if the last time you closed them you had open source files, so they reopen when you reopen the project), watch the memory usage - close them - look at the usage - memory not released. Should be pretty easy for you to attach the profiler and get a handle on what's not being dereferenced properly, right? Maybe you could try and dup it and get it fixed? If you need anything specifc, like project size, numbers, etc., I'd be happy to provide it but, as we see from other posts, it's well known.
Scott

0
Comment actions Permalink

If things were that simple...
As the matter of fact memory leaks won't happen to every project no matter
how often they're swapping. Those leaks are happening because of some project
or IDE or application server or whatever configuration, which is specific
to your configuration. That's why client side profile snapshots are cool.
BTW you can take one even with your 4.5.x. You just need to download (evaluation?)
YourKit profiler and tweak idea.bat launcher. If you can afford some of your
time to configure all that stuff to help us find the leak I'll come up with
detailed instructions shortly.

-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

OK, great and thanks - I'm assuming you work for JB - but, as I've
said above, this is not new. Search the forums for memory usage -
you'll find many reports going way back. I've just given you a very
simple scenario. Open up two large projects (particularly if the last
time you closed them you had open source files, so they reopen when
you reopen the project), watch the memory usage - close them - look at
the usage - memory not released. Should be pretty easy for you to
attach the profiler and get a handle on what's not being dereferenced
properly, right? Maybe you could try and dup it and get it fixed? If
you need anything specifc, like project size, numbers, etc., I'd be
happy to provide it but, as we see from other posts, it's well known.

Scott




0
Comment actions Permalink

I think there might be memory leaks but another problem with IDEA is that it creates a lot of intermediary objects all the time, which do get garbage collected so they are not technically a leak, but the result is that garbage collection has to run a lot more often than it should and takes a long time. Simply clicking around a file, I see the memory counter in the bottom right go up by megabytes very quickly. I think this is a big problem.

When we develop applications we test with profilers to make sure we don't generate a lot of objects while the user is doing simple tasks as part of memory testing.

0
Comment actions Permalink

sure, I'll be happy to though I can't guarantee a really snappy turnaround - I haven't been using any of the EAP builds this time around - too many customer deadlines to deal with - though I did you EAP builds through most of the 4.0 beta. If you want to get me instructions I'll be happy to give it a shot. Nothing really complicated here. No app server - just IDEA and 2 large projects with CVS interaction. Basic CVS update/edit/unedit/commit/diff, java edit/compile/debug - that will cause me to restart Idea a couple of times a day. Let me know and I'll do what I can.
And thanks for the response.
Scott

0
Comment actions Permalink

Hello All,

I have tried to start IntelliJ with VM 1.5 so it seems
a little faster and garbage collecting works better.
(The times of frozen IntelliJ hase been reduced)

My settings in the idea.lax are:

lax.nl.current.vm=C:
jdk1.5.0
bin
java.exe

and

lax.nl.java.option.additional=-Xms312m -Xmx512m -Didea.no.jdk.check=true ...

0
Comment actions Permalink

Scott Johnson wrote:

I'm using 4.5.4 build 2253 - but I've seen this behavior since I
started with IDEA - 3.0. It just continues to get worst as the
product grows.


Ok thanks, at first I thought you were talking about the EAP version,
which is pretty leaky right now.
Do you use any plugins, besides the bundled ones?

Bas

0
Comment actions Permalink

Thanks - I'll try this - completely forget about the jvm packaged with IDEA. And while I wish I could give it 512, that's pushing my machine. Unfortunately for me, I am required to use a company machine - not allowed to use my own - 512 is what I have - though I can probably make an argument for more. I'll let you know if 1.5 helps out. Are you seeing any issues running IDEA with 1.5? Anyone know if there are any known issues?
Thanks,
Scott

0
Comment actions Permalink

well, at first glance, jdk 1.5 does make a difference in memory usage and gc performance. Here's what I can duplicate easily, however. Maybe someone can give some insight. My main project is about 430 java files. I have several projects defined, each of which corresponds to a different branch of the project in CVS - generally the same amount of source - just different customizations. Now, if I start up IDEA, I'm using about 12M of memory per the memory meter in IDEA. If I open one of my projects (which opens with 2 files open as per my last close), I jump up to about 45M which I can force with the GC button down to about 36. Now, if I just sit and watch, it will grow from 36 to about 40 then GC back to 36 - repeatedly - presumably some background threads doing something. Now, open another project in it's own frame (ie. different CVS branch) which opens with 5 files open and memory grows to about 65M, then (again, with no usage on my part) cycle between 65 and 70. Now, close the second opened project and memory will decrease slightly, to around 60, but no more. Reopen the second project in it's own frame and memory jumps up to about 90, then cycles between 90 and 95 - close it, memory only goes down to about 85. Pushing GC over and over does nothing. So, it appears to me, that significant memory allocated at project open time is not released at project close time. It is routine for me to be opening and closing multiple projects during the day - this probably accounts for much of the memory usage. Again, in this test scenario, I didn't do anything other than open and close projects - no CVS interaction, no edit/save, no compile, etc. - just open projects, close projects and sit and watch memory. Should be relatively easy for JB to dup - I would assume there are various projects that can opened and closed for different versions or builds of IDEA that would probably replicate this.
Scott

0
Comment actions Permalink

Scott,
A couple of workarounds that are possible in a crunch: You can set up
two installations of IDEA with different system and config folders. That
will allow you to run two instances of IDEA in separate processes. You
can probably tweak the memory settings to work well on your machine.
Running multiple instances has other minor benefits too (can open modal
dialogs without blocking the other window), as well as some minor
drawbacks (any changes to IDE settings need to be changed for each
instance).
Secondly, you could splurge on an extra 512mb for about $100.
Personally, I would do this anyway, since it will benefit other apps and
overall performance (less page faults).

--
Rob Harwood
Software Developer
JetBrains Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

Thanks, I'll try - but I still think you've got some memory management problems and they should be addressed. With respect to going to 1G, well, yeah, if I had any say that would be a given - my own personal machines are maxed - always. But the company owns the laptop and I'm required to use it so that's that - money is hardly the issue - if I can't get them to do it then it won't happen. Besides, the problem will still exist, just take a little longer to become annoying, so that's really not a solution at all. Point of my post was pretty straightforward - memory isn't being released. Just wondering why. Again, you can find many complaints of this in these forums - just hoping it could be addressed or at least explained. I figure if I've closed my projects and all I have is an IDEA frame, most of the memory allocated for the projects should have been freed up - it's not - just hoping that might give you some information. I've lived with it since the beginning and I'll continue living with it - it's not really any more of a crunch than it's ever been - just as annoying as it's ever been - would be awesome if it got better.
Scott

0
Comment actions Permalink

Scott Johnson wrote:

... but I still think you've got some memory management problems ...


I know, just giving some workarounds. :)

--
Rob Harwood
Software Developer
JetBrains Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

I'm not sure you understood what Maxim was saying... he's suggesting that
you take a snapshot using your existing 4.5.4 installation and send them
the snapshot for analysis.

To do this, grab a trial version of YourKit from http://www.yourkit.com/download/index.jsp
and install it. Then adjust your idea.bat (or idea.lax?) file by adding -Xrunyjpagent:port=10100
to Java's startup params. Now fire up IDEA and, once you've opened then closed
a few projects to trigger the leak, take a snapshot with YourKit. Send that
snapshot through to JetBrains and hopefully they'll have a fix for you (and
everyone else affected by this problem!) in 4.5.5 :)

(Apologies if the above instructions aren't 100% accurate BTW! I haven't
tried snapshotting IDEA myself however the info should be enough to get you
on the right track at least).

sure, I'll be happy to though I can't guarantee a really snappy
turnaround - I haven't been using any of the EAP builds this time
around - too many customer deadlines to deal with - though I did you
EAP builds through most of the 4.0 beta. If you want to get me
instructions I'll be happy to give it a shot. Nothing really
complicated here. No app server - just IDEA and 2 large projects with
CVS interaction. Basic CVS update/edit/unedit/commit/diff, java
edit/compile/debug - that will cause me to restart Idea a couple of
times a day. Let me know and I'll do what I can.

And thanks for the response.

Scott




0
Comment actions Permalink

Yes, I did get that - just didn't know if there was anything specific. I use JProfiler myself and was simply going to run IDEA and attach using it. Haven't used YourKit yet - I know it uses the newer interface but I didn't see that it was as complete as JProfiler and sells for the same price so I haven't bothered with it. I'll give it a shot quickly as I can with JProfiler - if I see something odd I'll send it in - if for some reason I must use YourKit, I'll do it as I have time.
Thanks,
Scott

0
Comment actions Permalink


The memory snapshot is almost 29MB if that's any indication. At the
time, I had shutdown my server and only had 1 file open. I had hit the
GC several times and minimized. I even cleared out the error messages.
Still in the neighborhood of 170MB per Task Manager.

Norris Shelton
Sun Certified Java Programmer




Maxim Shafirov wrote:

That's obvious memory leak aka bug...
In Irida EAP (http://www.intellij.net/eap) we have embedded YourKit
memory and CPU profiling into production thus if anyone feels IDEA has
a memory leak or performs ceratain operation for way too long it's
enough to press the button on the IDEA toolbar and send created
profiling snapshot to the developers. That's great of help for us
indeed. It makes solving many of such problems just "open, see what
the garbage is going on, fix" instead of suffering guessing and poor
assumptions about what may be happening at client side.

>

-------------------
Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

>
>> I know this has been questioned many times but I'm hoping I can get
>> some insight here. I spend my time on a large commercial application.
>> Customers are generally customized so we maintain multiple CVS
>> branches. I have an IDEA project set up for each. The project is
>> large - around 400 classes. I have IDEA start up with a minimum of
>> 64M and max 256M heap. Now, I can open-up one project and watch
>> memory go up to around 100-120M. Open up a second in another frame
>> and watch memory grow to almost 200M. Now, of course, we get into the
>> misery of client-side java - GC which basically consumes my machine.
>> But, if I close one project (and it's frame) I only free up a small
>> amound of memory - maybe 20-30M. In the remaining frame I can close
>> the project, so all that's left is an IDEA frame with no project. At
>> best I'll go down to 160-175M. Why is IDEA holding on to so much
>> memory with no open projects? I can hit the GC button all I want -
>> but IDEA will not let go of the memory. Does anyone have any insight?
>> I'm running a new Latitude D600 with Win2K - 512M - and with 2
>> projects open - it's gets really bad, really fast - constantly lagging
>> being my key/mouse movement and garbage collecting continuously. Any
>> info or suggestions would be greatly appreciated.
>

Thanks,
Scott

>
>>
>
>
>



Attachment(s):
moz-screenshot.jpg
0
Comment actions Permalink

I see it in ftp.intellij.net/.uploads. We'll look into shortly.
-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"




0

Please sign in to leave a comment.