IDEA cache size

I'd like to suggest having an option to control IDEA cache size.

I have two moderately sized projects developed for Java 1.5 (no more projects in IDEA, no other JDKs). And the cache (system\caches) is already 850 MB, almost three times (!) bigger than both projects + JDK alltogether (it doesn't even sound as a "cache" to me - which is usually smaller but more critical part of data).

I'd suggest there was an option to limit the cache size. Or, even better, cache storage format could be optimized.
Even plain ZipStream usage for individual plain-text parts of content.dat.storageData would have cut off about half of that size, I guess.
And I don't think it will make anything slower. IMO quick decompression will outperform disk reading speed here.
As long as you do not try to compress the whole file in "solid" mode.  And you can still have content.dat.storageRecordIndex

The main reason for this suggestion is that IDEA cache has grown significantly.
I used to put IDEA system folder onto RAM drive for HDD peace and superb performance.
However, as of not so long ago 1 GB disk is not enough anymore.
I don't want to sacrifice another block of precious memory. And it seems to me the situation can be improved.

Anyway I'm interested in any comments you might have on this suggestion.


P.S. Posting to forum does not work in Opera 10.5x - post text editor does not appear. It'd rather "degrade" nicely and show at least plain textarea.

11 comments

Hello Dmitry,

Supporting the size limit option would require a major re-architecture of
our caches/indexes subsystem, which is currently designed around append-only
patterns and does not have a goal of reducing the space used for indexes.
Zipping the file contents, on the other hand, is rather easy to implement,
but it's not clear what the performance tradeoffs would be. (You can actually
play with this yourself - grab the CE source code, go to the FileAttribute
class and add a ZipInput/OutputStream around the streams used for reading
and writing the data.)

I'd like to suggest having an option to control IDEA cache size.

I have two moderately sized projects developed for Java 1.5 (no more
projects in IDEA, no other JDKs). And the cache (system\caches) is
already 850 MB, almost three times bigger than both projects + JDK
alltogether (it doesn't even sound as a "cache" to me - which is
usually smaller but more critical part of data).

I'd suggest there was an option to limit the cache size. Or, even
better, cache storage format could be optimized.

Even plain ZipStream usage for individual plain-text parts of
content.dat.storageData would have cut off about half of that size, I
guess.

And I don't think it will make anything slower. IMO quick
decompression will outperform disk reading speed here.

As long as you do not try to compress the whole file in "solid" mode.
And you can still have content.dat.storageRecordIndex

The main reason for this suggestion is that IDEA cache has grown
significantly.

I used to put IDEA system folder onto RAM drive for HDD peace and
superb performance.

However, as of not so long ago 1 GB disk is not enough anymore.


I don't want to sacrifice another block of precious memory. And it
seems to me the situation can be improved.

Anyway I'm interested in any comments you might have on the
suggestion.

P.S. Posting to forum does not work in Opera 10.5x - post text editor
does not appear. It'd rather "degrade" nicely and show at least plain
textarea.

---
Original message URL:
http://www.jetbrains.net/devnet/message/5263002#5263002

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"


0

Thanks for the tip! Will try that.

But how can I measure performance with and without those changes in place?
I.e. is there any "random read" performance test for cache or something? Or should I build one myself?

Otherwise there is no point - I cannot integrate it into ideaIU, or  can I?

0

Hello Dmitry,

I don't think we have any existing performance tests for this area specifically.

You won't be able to integrate it into IU yourself, but if the changes turn
out to bring a performance benefit, we'll be happy to integrate your patch
into the main codebase (IC and IU). And IC should be more than enough to
test the performance effects.

Thanks for the tip! Will try that.

But how can I measure performance with and without those changes in
place?

I.e. is there any "random read" performance test for cache or
something? Or should I build one myself?

Otherwise there is no point - I cannot integrate it into ideaIU, or
can I?


--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"


0

Maybe you can suggest then some operations that depend on this cache more than others?
Re-building project? Finding usages?..

0

Hello Dmitry,

Maybe you can suggest then some operations that depend on this cache
more than others? Re-building project? Finding usages?..


The first thing to benchmark is initial index building; it's critical that
it not become slower. Finding usages also depends on indexing performance,
although it's not the main factor. Project rebuild doesn't use indexes at
all.

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"


0

Here are the results of my tests:

I've run initial indexing test twice for each version, deleting folder 'system' in between. And here are the numbers for the first and the second run:

Uncompressed (original) cache:
indexing project1 + JDK - 4:10 and 3:17
indexing project2 (uses the same JDK) - 3:09 and 2:27

content.dat.storageData size = 813 MB

Compressed (modified) cache:
indexing project1 + JDK - 1:57 and 1:51
indexing project2 (uses the same JDK) - 2:39 and 1:43

content.dat.storageData size = 458 MB

Testing platform:
Core 2 Duo @ 2.40 GHz
SATA II HDD @ 7200 rpm
WinXP Pro SP3 (32 bit)
Sun Java 1.6.0_17
-Xms32m -Xmx256m

It should be mentioned though that initial indexing is biased towards less-HDD-usage version (since it uses HDD a lot itself). Besides it is performed not so often. And thus affects "every-day experience" not so much.
So I've also executed some other tests: loading a project, find usages of String, find in files - and ... haven't noticed any substantial difference.

I've attached the patch for FileAttribute.java. But it seems to me the change is better pushed down to ManagingFS.

Hopefully next IDEA build will include this improved cache version.



Attachment(s):
FileAttribute.patch.zip
0

Thanks for your patch, we are giving it a look. This place is a bit
refactored since 9.*

This thread seems to be for Dmitrys only :)



0

Cache compression is implemented in IDEA 10.
It does improve cache building performance and reduce cache size.
Thanks for cooperation.

0

Hurray!
Now I'm even more eagerly (I didn't think it's possible) waiting for IDEA 10 EAP
(do you have any estimates, btw, when it will be available?)

0

Dmitry Avdeev wrote:

Cache compression is implemented in IDEA 10.
It does improve cache building performance and reduce cache size.
Thanks for cooperation.

Excellent.  This shows the real benefit that open source gives us!
N.

0

:(  the Dmitry-only chain was broken...

Seriously though... +1 to Nathan's comment: this collaboration points out the benefit of IntelliJ being open source via the Community Edition. Fantastic! :)

0

Please sign in to leave a comment.