PyCharm Memory Usage

I suppose this post is similar to some others I've read, but I haven't seen a completely satisfying answer yet (and maybe never will :-).

I have my -Xmx and -Xms both set to 1024m, but Linux reports that PyCharm is using over 2.5GB of memory (and possibly still growing). I understand that the JVM needs some additional space other than what is specified by these options, but more than double what I have specified (and 1.5GB total) seems quite excessive.

I do run some large SQL queries and programs that generate a lot of output to the run console, which might be one culprit. However, I was under the impression the memory for those things is still encompassed by the Xmx parameter, since I get warnings about increasing my Xmx if I generate lots of output and set the Xmx value lower.

Any thoughts would be appreciated.

2 comments

Hi,

Instead of trying to explain it myself, I did a simple Google search, and here's some of the top results:

https://plumbr.io/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx

https://stackoverflow.com/a/48874303/6718686

In short - yes, it's possible that JVM is using significantly more memory than you specify by -Xmx flag, especially working with large sets of data. If you think this is excessive however, and it might be an issue, feel free to report a performance issue to our tracker: https://youtrack.jetbrains.com/issues/PY

If you do, please provide a memory snapshot: https://intellij-support.jetbrains.com/hc/en-us/articles/207241235-Reporting-performance-problems

0

I think it was these kinds of pages with the following equation that was causing the confusion and concern.

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss]

-Xmx is set to 1G, as far as I remembered the default MaxPermSize was around 64M (which in Java 7 it roughly was), and the default stack size (-Xss) is around 2M. So I would have expected a couple hundred extra megabytes at most. It would either take a much larger MaxPermSize or a huge number of threads to get the kind of memory usage I am seeing.

For those having a similar issue, it turns out as of Java 8 the PermGen space has been superseded by the Metaspace and the Metaspace does not have a max size by default, so it can grow and grow and grow. You can set a maximum size with -XX:MaxMetaspaceSize=N. Setting it too low is causing problems (PyCharm will freeze and need to be forcefully killed) when running large queries that apparently no longer fit in the Metaspace.

It is still a little confusing, since the Metaspace seems to be reserved for "the representation of class and metadata", so it is not clear why SQL queries would fill it up (although this is well outside my knowledge of the JVM).

0

Please sign in to leave a comment.