Getting more available memory in IDEA without a 64 bit VM on VISTA

I have apparently solved/mitigated the problem that I and others have been having for quite some time where we cannot allocate more than 600M of memory to IDEA in idea.exe.vmoptions and
have IDEA startup reliably, I thought I would restate it here in case the IDEA people wanted to pursue this. Though many people don't care about having that much memory
and some people apparently can just allocate as much as they want without any problem, I figured JETBRAINS would want to know about the trivial fix required to allow 1G to be
allocated reliably which for me seems to do the trick where 600M simple was unuseable.
 
]]>I would love to know why some people on many different
instances of PC hardware  and VISTA instalations over a long period of time
can't allocate more than 600M and some with apparently quite similar hardware
and software installations can allocate more than 1G, it doesn't seem to be
that easy to answer that question and frankly now that I have a solution I can't
really justify the time to figure it out.
 
The answer is a bit in the windows EXE binary object file called the LAA  (LARGEADDRESSAWARE ) bit. Its not a big secret and you can find lots of info on it such as:
 
If you set that flag on idea.exe then those of us with that problem can allocated 1G, if you flip it back, we can only allocated 600M. I can go back and forth all day with the exact same result. I am quite convinced that it is a fix though admittedly I can't explain why a 32 bit JAVA VM on a 64 bit VISTA install performs differently when its set or unset.
 
So I am requesting that this  bit  get set on idea.exe in future releases. It would make some of our lives easier.  I would far prefer that you support a full 64 bit version of idea but
I doubt that will happen anytime soon. If my project starts to need more than 1G I guess I will have to solve the teeny font problem that happens to me when I run IDEA on a 64 bit SUN VM.
 
So how about it JETBRAINS? It couldn't be easier to do this...
 
thanks!
 
Erik
 

]]>

6 comments
Comment actions Permalink

Thanks for this Chris -- Did you get out of this article a plausible
scenario that would explain why some people running a vanilla Idea release
with its built in 32 bit JVM on Vista 64 can easily allocate considerably
over 1G of memory but others can't get more than 600M without setting the
LARGEADDRESS bit and then only a max of 1G? I could not see it in there -
but there was a lot there. I wonder if something like AMD vs INTEL
architectures or motherboard chip set issues, or installation of particular
driver software could do this. I have experienced this myself personally on
at least four totally different hardware setups but all were INTEL CPUs and
chipsets... My experience with this scenario is extremely repeatable and
consistent - it happens immediately from a fresh install of Vista -- and
won't change.

thanks again Chris - I wish Idea would take some interest in something so
basic.

ejf


"Chris Miller" <chris.miller@kbcfp.com> wrote in message
news:a6939bdae4d838cba29965c07981@news.jetbrains.com...

Hi Erik,

>

Given the nature of your problem, you might find this article quite
interesting:

>

http://www.ibm.com/developerworks/linux/library/j-nativememory-linux/index.html?ca=dgr-jw22Linux-JVM&S_TACT=105AGX59&S_CMP=grjw22

>


0
Comment actions Permalink

According to my understanding this is not IDEA's fault and as far as I can
see there's not a lot Jetbrains can do about it. In particular, setting the
LARGEADDRESS bit does not really solve the problem, plus it potentially
introduces other nasties due to the reduced kernel size. The problem is that
Sun's VM (I'm not sure about other VMs though?) requires a contiguous memory
region to allocate it's heap due to the way they optimise some of their internal
indexes and data structures, and on your particular system you don't have
as much contiguous memory as you might like. Note that a fragmented heap
would be possible to implement in the VM, but at a performance cost.

Now as to why you're seeing less available contiguous space than other people
comes down to which libraries are being loaded before the JVM starts, and
where they are loaded. A DLL has a default base location where it prefers
to load. If it's not able to load at that location, it dynamically gets relocated
by the OS but that can slow down startup time and prevent it being shared
by other applications (thus hurting performance, increasing swap size etc).
For whatever reason, you have a DLL that is plonking itself in the middle
of your user memory and hence limiting the maximum contiguous JVM heap that
can be allocated. The /3GB switch can help in that the maximum contiguous
size might end up being bigger, but it's not really solving the true problem
which is the inconveniently-located DLL.

You can use listdlls (http://technet.microsoft.com/en-us/sysinternals/bb896656.aspx)
or Process Explorer (also from sysinternals) to see what DLLs are loaded,
and what their base memory addresses are. Try it both before and after you
start the VM (the JVM loads some DLLs too...). That should provide some indication
of which DLL is the culprit on your system. It could be device drivers, anti-virus
software, etc... Obviously the DLLs that are loaded can vary significantly
between different PCs depending on combinations of both hardware and software,
and it only takes one DLL with a bad base location to chop your heap in half.
(Note that the DLL developer may not have considered that a problem since
most apps don't mind having non-contiguous memory).

Obviously depending on the culprit you have a few courses of action, eg uninstall/disable
the offending driver/software. If that's not an option you could 'rebase'
the DLL using a free tool from Microsoft, however you might want to be a
bit careful if you do so and do a bit of research into the consequences.
It's not something I've ever tried, but if you give it a shot I'd be interested
to hear how it goes!

Typically the best thing to do in this situation is to move to a 64bit VM,
though in this case I can appreciate why that's not ideal. Have you looked
at all the options for overcoming the 64bit small font problem? You could
also try running IDEA on non-Sun JVMs, they might not require a contiguous
heap or they may load their DLLs in different locations that help mitigate
the problem.

Also, here's a bugreport you might find interesting: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4358809

Hope that helps!

Chris

Thanks for this Chris -- Did you get out of this article a plausible
scenario that would explain why some people running a vanilla Idea
release with its built in 32 bit JVM on Vista 64 can easily allocate
considerably over 1G of memory but others can't get more than 600M
without setting the LARGEADDRESS bit and then only a max of 1G? I
could not see it in there - but there was a lot there. I wonder if
something like AMD vs INTEL architectures or motherboard chip set
issues, or installation of particular driver software could do this.
I have experienced this myself personally on at least four totally
different hardware setups but all were INTEL CPUs and chipsets... My
experience with this scenario is extremely repeatable and consistent -
it happens immediately from a fresh install of Vista -- and won't
change.

thanks again Chris - I wish Idea would take some interest in something
so basic.

ejf

"Chris Miller" <chris.miller@kbcfp.com> wrote in message
news:a6939bdae4d838cba29965c07981@news.jetbrains.com...

>> Hi Erik,
>>
>> Given the nature of your problem, you might find this article quite
>> interesting:
>>
>> http://www.ibm.com/developerworks/linux/library/j-nativememory-linux/
>> index.html?ca=dgr-jw22Linux-JVM&S_TACT=105AGX59&S_CMP=grjw22
>>


0
Comment actions Permalink

Hi Chris - I am stil digesting the majority of this response but wanted to add some references I found that seem relevant...
Out of these MS references I get that the /3GB switch   (or 4-Gigabyte Tuning)  is only applicable to  32bit Windows versions.  As I mentioned before, I am using VIsta 64 which has a rather robust 8TB space for the Kernel. I also see that in VIsta 64, the user process space is limited to 2GB  without the IMAGE_FILE_LARGE_ADDRESS bit set but doubles to 4Gb with the IMAGE_FILE_LARGE_ADDRESS set.  That would probably explain why I can get more VM allocated with that bit set. I see nothing about this causing any problems with Kernel allocations. The IMAGE_FILE_LARGE_ADDRESS is a per-executable binary file option.  While it may be true that a driver or DLL is screwing around with the first 2GB, apparently it does not screw around with the second 2GB. If this is all the way it seems then it certainly seems to make more sense to have Jetbrains set a flag in idea.exe than to have any users like me who need more memory to search for and either remove or rearrange drivers or DLLs on their system.  I don't see any indication of it creating problems on 32 bit windows versions. You could test for that pretty easily I would imagine.
 
I guess I am growing more convinced that my little workaround should be in all releases - at least it deserves their attention... Does that make sense?
 
Again thanks for your help with this!
 
Erik
 
 
 
> According to my understanding this is not IDEA's fault and as far as I can
> see there's not a lot Jetbrains can do about it. In particular, setting the
]]>&gt; LARGEADDRESS bit does not
really solve the problem, plus it potentially > introduces other nasties due to the reduced kernel size. The problem is that
> Sun's VM (I'm not sure about other VMs though?) requires a contiguous memory
> region to allocate it's heap due to the way they optimise some of their internal
> indexes and data structures, and on your particular system you don't have
> as much contiguous memory as you might like. Note that a fragmented heap
> would be possible to implement in the VM, but at a performance cost.
>
]]>

0
Comment actions Permalink

Sorry yes my mistake, I meant to say the LARGEADDRESS flag not /3GB. The
/3GB switch is indeed for 32bit OSes only, and works in conjunction with
the LAA flag. My only real experience with this flag (and /3GB) has been
with 32bit MS Analysis Services where in the past we had to enable them both
on a few of our servers (we're 64 bit now so it's no longer an issue), hence
some of my confusion wrt the 64bit behaviour. You're right, the kernel space
problem only affects 32bit OSes.

I don't see any indication of it creating problems on 32 bit windows versions.


Given the first article you reference says "Setting this flag and then running
the application on a system that does not have 4GT support should not affect
the application.", it does indeed sound like there's no harm in setting that
bit in all Windows builds. It'll only help people in the exact same situation
as you though, and given you seem to be the first person to speak up about
it I guess it's not a common problem, which is why Jetbrains haven't done
anything about it yet.

You could test for that pretty easily I would imagine.


I can't see how it would be possible to test this other than by trial and
error on many different hardware and software configurations. Better to just
trust the docs quoted above I guess :)

I think it would be an interesting exercise for you to try and find what
the problem DLL is on your system still, and see if the contiguous memory
available corresponds to what you see as your maximum heap size. Shouldn't
take too long to figure out from the listdlls output. And of course the problem
DLL(s) could still be rebased to give you even more heap if you need it!

Better yet of course would be for the 64bit VM to work as expected, then
all this mess goes away...

Chris


0
Comment actions Permalink

I do appreciate your time on this Chris - I wish Jetbrains would take this
as seriously...


-erik


"Chris Miller" <chris.miller@kbcfp.com> wrote in message
news:a6939bdae51148cba60ce276a060@news.jetbrains.com...

Sorry yes my mistake, I meant to say the LARGEADDRESS flag not /3GB. The
/3GB switch is indeed for 32bit OSes only, and works in conjunction with
the LAA flag. My only real experience with this flag (and /3GB) has been
with 32bit MS Analysis Services where in the past we had to enable them
both on a few of our servers (we're 64 bit now so it's no longer an
issue), hence some of my confusion wrt the 64bit behaviour. You're right,
the kernel space problem only affects 32bit OSes.

>
>> I don't see any indication of it creating problems on 32 bit windows
>> versions.
>

Given the first article you reference says "Setting this flag and then
running the application on a system that does not have 4GT support should
not affect the application.", it does indeed sound like there's no harm in
setting that bit in all Windows builds. It'll only help people in the
exact same situation as you though, and given you seem to be the first
person to speak up about it I guess it's not a common problem, which is
why Jetbrains haven't done anything about it yet.

>
>> You could test for that pretty easily I would imagine.
>

I can't see how it would be possible to test this other than by trial and
error on many different hardware and software configurations. Better to
just trust the docs quoted above I guess :)

>

I think it would be an interesting exercise for you to try and find what
the problem DLL is on your system still, and see if the contiguous memory
available corresponds to what you see as your maximum heap size. Shouldn't
take too long to figure out from the listdlls output. And of course the
problem DLL(s) could still be rebased to give you even more heap if you
need it!

>

Better yet of course would be for the 64bit VM to work as expected, then
all this mess goes away...

>

Chris

>


0

Please sign in to leave a comment.