sbt refresh slow -- questions

Hi! Thanks for the report.

We're aware of this problem and working on solution. It will be shipped some time soon.

As for what you can do now to speed up refreshing:

• You can remove `aggregate in update := false` setting. I was mistaken when thought that aggregation could cause slowdown. Researching the problem showed that aggregation causes no harm, so you could enable it.
• updateOptions := ... put in build.sbt enables cached resolution for root project only, which has minor impact on refreshing times. In order to speed up refreshing, you need to enable it for every project. You can do it by putting updateOptions in Global := updateOptions.in(Global).value.withCachedResolution(true) into build.sbt instead.

Unfortunately, that's all you can do about refreshing.

Thanks Nikolay, looking forward to the fix.

Chris

Any update on this fix?  I'm running 14.1.5 and still having performance issues.

There is an experimental solution, but in order to try it you need to upgrade to IDEA 15 / Scala plugin 1.9.2. It is called "cached update", you can turn it on in "Import Project" dialog. It would be great if you try to use this option and then tell me about your experience.

I have a similar project (with many subprojects) and similar problems with very slow update times. (My PC runs a dual core i7 processor with 12G RAM). Here are some test results.

The project uses SBT 0.13.9 and Scala version 2.11. A full reload in the ordinary SBT console takes approx. 10 seconds while a full reload and thereafter a library update check takes around 50 seconds.

In IDEA it takes much longer. I am currently running the latest IDEA 15 RC (143.379.11) with Scala plugin 1.9.4.

A full SBT refresh in IDEA without caching enabled takes around 8 minutes. I have also tried to enable the experimental caching option and it ended up with around 9 minutes - I.e. no visible effect. Another refresh still takes 9 minutes to complete.

I have tried various VM options but nothing seems to affect the refresh time. I can notice that the process launched during import consumes much CPU, but memory consumption is moderate. (With VisualVM I can see that there is a hotspot at the org.jetbrains.sbt.structure.Helper$RichFile.path() method, but I am not sure if this indicates something wrong).

I have also tried the latest Scala plugin from the EAP channel (I believe it was 1.9.4.6), but with this version the refresh time ended up around 15 minutes.

Same for me. Idea 15 EAP makes it even worse.
I just got a new job, where they have a very large Scala project splitted in 20-30 subprojects and refreshing it takes literally an hour..

Thank you for the information. Recently I've came up with another possible solution for this problem, but since it requires a lot of refactoring in Scala plugin and sbt-structure, I won't be able to ship it in 2.0.0 version of Scala plugin. Please, watch for changes in this ticket (https://youtrack.jetbrains.com/issue/SCL-8322) and stay tuned for further version of Scala plugin

It seems that the slow SBT refresh is an issue with many multi-project builds.

IDEA is doing great at incremental Scala recompilation during ordinary coding, but the SBT refresh issue is slowing down development speed significantly, especially in cases where frequent refreshes are needed due to local library Snapshot publishing. A solution (or workaround) to this issue would be much appreciated.

Thanks for the update and the link.

Hi all,

Thanks for th updates.
I am working on a build definition that have 86 projects and it takes 20min to refresh.
So if you want me to experient something on a massive build definition let me know.

Cheers

Hi! Try Scala plugin nightly buld 2.0.240 or later. It improves importing speed greatly for projects like yours.

Thanks for the update. I have testet the latest nightly build and can notice an improvement in speed; it takes around 5 minutes now, much better when compared to 15 minutes earlier.

However, SBT uses 45 seconds for the entire process, so there still seems to be room for improvements.

It seems that intellij is trying to resolve libraries that has already been resolved before. I have a private proxy repo that i am using. I tried switching it off intellij tries to resolve against that repo and time out even when the library is already in my ivy cache.

Having the same issue with latest IntelliJ version:

IntelliJ IDEA 2016.2.4
Build #IU-162.2032.8, built on September 9, 2016
JRE: 1.8.0_112-release-b343 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

Each minor change to root build.sbt file hangs IDE for several minutes.

Are there any workaround so far?

Here is a workaround:

https://github.com/alexarchambault/coursier

It's way much faster in SBT as well as Idea.

Brilliant, thanks a lot Joan G! Didn't know about this plugin.

You're welcome.
I can understand the joy of what you are currently feeling :)

The only thing to bear in mind is that it doesn't support snapshot dependencies.
Also I don't understand why the SBT core team is not just throwing there dependency manager in a garbage destroy machine and just replace it by Coursier.

Still an issue on latest intellij, IntelliJ

IDEA 2016.2.5
Build #IC-162.2228.15, built on October 14, 2016

Still an issue in IDEA 2016.3 with latest Scala and SBT Plugins.  Large multi-module (aggregated) project takes approximately 12 minutes to "refresh" after making any change to our build file.

Tested using http://www.scala-sbt.org/1.0/docs/Cached-Resolution.html and saw no difference in the time (~12 minutes).  It still seems to perform full dependency resolution for every subproject.

Would love to have a fix for this as it makes things very difficult when needing to add projects or new libs.

Still an issue in IDEA Ultimate 2017.2

When I create a new SBT based project, it takes reaaally long in the 'refreshing <project name> SBT project' phase. I'm not happy about it at all. I assume its related to this issue. IDEA Ultimate 2017.2.4

What I found out is that if downloading of source jars is enabled, it takes much longer.

This issue is still valid for SBT 1.0.2 + IntelliJ 2017.2.5 Community.

Still an issue with 2017.3, progresses absolutely nowhere while eating up massive amounts of CPU


I've been looking at that for the past 45 minutes.



%CPU is usually higher.

What is happening and how has it gone unresolved for over two years?

Also having this issue in 2017.3.3

Just to add myself to the ones experiencing very slow updates in IDEA 2018.1 Public Preview with latest Scala plugin!

I am having the same problem, and that is one of the reasons why I am switching back to Eclipse although Scala-IDE has its own fair share of problems. The other reason is that the formatter oftentimes stops working all of a sudden. There are a lot of exceptions in the log. It starts working again if I restart IDEA. I also filed a bug report on that one, and there hasn't been any activity on that issue ever since. Seems like Scala is not a very high-priority language in IDEA.

@Jens Rabe It's not an IDEA, but sbt's issues. It's difficult to fix, though you can mitigate it by turning off automatic project import in intellij and use coursier plugin to fetch dependencies (https://github.com/coursier/coursier). 

@Yuriy Filonov How can this be sbt's issue, if sbt starts and reloads many times faster then Idea reloads?

@Jens Rabe Well, I guess you are right - it's a combination of both :) IDEA's project update mechanism may be a bottleneck, but on top of that dependency resolution strategy being used by sbt by default is not very efficient. Coursier partly resolves this problem. I was able to get away with it and automatic project import turned off. Works fine unless you are changing build.sbt (or other project related files) very often.