16 core workstation, 16GB RAM, multi module java/scala project, how to speed up compilation

I.e. when I do a clean build, 1-2 cores are used during compilation (I've enabled "compile independent projects in parallel"). Is there something I can do to use more processors and speed up compilation?

12 comments
Comment actions Permalink

You may try to disable Scala compile server (Settings | Scala | Run compile server).

The primary goal of the compile server is speed up incremental compilation, however (currently) it provides only a single instance of Scala compiler at a time (more info).

If you perform a clean build and your project contains many independent modules, disabling the compile server may improve compilation performance by allowing better parallelism.

Additionally, make sure that I/O isn't a bottleneck (are you using SSD?).

0
Comment actions Permalink

Thanks for the suggestion, I gave it a try. Unfortunatelly compilation times are about the same, around 5 mins 30 secs.

(I do have a very fast SSD)

0
Comment actions Permalink

The primal performance constraint in full project compilation is (probably) the Scala compiler itself, you may check an explanation from Martin Odersky.

0
Comment actions Permalink

yes the scala compiler is slow but independent modules could be compiled in parallel by i.e. multiple instances of the intellij scala server in a same fashion as mvn -T 2C

0
Comment actions Permalink

Please try the next Scala plugin nightly build (0.17.192 or 0.18.40), with or without the compile server.

It is very likely that parallel compilation speed will be improved with these builds.

0
Comment actions Permalink

The next nightlies are required - these builds will be available tomorrow.

0
Comment actions Permalink

Ok, thanks will give it a try

0
Comment actions Permalink

ok,seems this improved build times by 2x!

CPU usage goes up to 60% (it uses more than 8 of my 16 cores)

That's a big improvement, thanks

0
Comment actions Permalink

When is this going to be officially released?

Cheers

0
Comment actions Permalink

That's good! By the way, don't forget to increase JVM maximum heap size apropriately (either in Project Settings | Compiler when compile server is disabled, or in Settings | Scala when compile server is enabled), because all the "parallel" compilers are instantiated within a single JVM.

0
Comment actions Permalink

yes, did increase it to 1.5GB


0

Please sign in to leave a comment.