Advice on using very large memory jvm

What type of experience do folks have with very large memory footprint jvm sessions (large -Xmx) for servers? Someone here seems to be under the impression that going over 2G is a universally bad idea due to inherent garbage collection issues regardless of what version of java (even 1.6). Is anyone running jvm's successfully with >2G or >4G on 64-bit servers? Were you forced to change default gc settings to make it workable?

Thanks,
Jon

P.S. Sorry about the off topic post, but you guys are just too smart for me to resist.

8 comments
Comment actions Permalink

Our server has a couple java proceses. One of them uses 4 GB to 6 GB of memory. This is running on a Sun T2000 box which I think has 4 cpus but each core has 8 'execution' threads, so it looks like it has 32 cpus.

Anyway, using JDK 1.6 the performance has been fine. On a multi-cpu box, the GC automatically uses parallel GC, so the stop-the-world time is shorter.

If you're concerned you should just run your application with the GC debugging on,
-XX:PrintGCTimeStamps -XX:PrintGCDetails, and look to see how long the stop-the-world time is.

0
Comment actions Permalink

I remember reading an interview with Joshua Bloch (of Sun and now Google
fame) not too long ago, and he mentioned he's working with extremely
large datasets in Java at Google these days; such as array with more
than 1 billion Objects, etc. It might be worth making a couple of
Google searches with that lead...

Jon Steelman wrote:

What type of experience do folks have with very large memory footprint jvm sessions (large -Xmx) for servers? Someone here seems to be under the impression that going over 2G is a universally bad idea due to inherent garbage collection issues regardless of what version of java (even 1.6). Is anyone running jvm's successfully with >2G or >4G on 64-bit servers? Were you forced to change default gc settings to make it workable?

Thanks,
Jon

P.S. Sorry about the off topic post, but you guys are just too smart for me to resist.

0
Comment actions Permalink

Well, as far as that lead goes....as best I can tell, the Bloch thing you mention really has to do with processing lots of data and seems it is about distributed processing that doesn't imply a large or very large jvm sessions, but thanks anyway.

0
Comment actions Permalink

You probably mean Joshua blog entry where he was describing how working with large arrays reveals an overflow bug in binary search implementation:
http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html

0
Comment actions Permalink

Yes, I think that was the one.

He said that at Google and other places, they routinely use arrays
containing a billion or more elements.

To me that implies a memory footprint of well over 4GB.

Wouldn't a single array containing 1 billion integers have a footprint
of 4GB because an int is 4 bytes?


Evgeny Goldin wrote:

You probably mean Joshua blog entry where he was describing how working with large arrays reveals an overflow bug in binary search implementation:
http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html

0
Comment actions Permalink

Well, it definitely seems so. The JVM footprint will be even bigger than that ..

0
Comment actions Permalink

True. However, I think most are aware that the JVM on a 64 bit system can potentially go way beyond a 4G max memory size.

The question I have is remains this:
Can the folks here share their direct experience with large JVM memory footprints on 64-bit systems running real world deployments and have you encountered any Garbage Collections challenges with these setups?

Thanks!
Jon

0
Comment actions Permalink

Hi Jon,

We have a few jobs that use ~3GB of heap. These have been running fine on
both 1.5 and 1.6, on 4 x dual core Opteron boxes running Solaris 10 each
with 8GB RAM. Performance has been fine even without tuning the GC (the main
bottleneck for us has been I/O). The jobs themselves run for about 20+ hours
each doing a lot of DB work and data manipulation during that time. During
the jobs' runtime they slowly build up pretty large data structures in memory
(containing predominantly arrays of doubles plus lists of strings) and tend
to hang on to most of that data for the job's lifetime. We've done extensive
profiling and optimising to reduce the memory footprint (and to a lesser
extent, reduce the CPU load), including implementing in-memory compression
of some of the data. Without this optimisation we'd have required somewhere
near 30GB of RAM, which I imagine would have been a more painful requirement...

Chris

What type of experience do folks have with very large memory footprint
jvm sessions (large -Xmx) for servers? Someone here seems to be under
the impression that going over 2G is a universally bad idea due to
inherent garbage collection issues regardless of what version of java
(even 1.6). Is anyone running jvm's successfully with >2G or >4G on
64-bit servers? Were you forced to change default gc settings to make
it workable?

Thanks,
Jon
P.S. Sorry about the off topic post, but you guys are just too smart
for me to resist.



0

Please sign in to leave a comment.