Customized Gradle Focus Plugin and IntelliJ Gradle sync
Hello, I work at Faire. We use Gradle for our backend. We use Kotlin. We currently have ~720 projects.
We have configuration cache enabled but misses are still expensive. In addition, when we load the whole project in the IntelliJ IDE then the IDE is slow as well.
So we wanted to try out https://github.com/dropbox/focus. We got it working by adjusting it to our repo.
The focus plugin discovers a project's dependencies through any configuration. This is understandable because otherwise, after focusing, Gradle will fail due to not including all the needed projects.
But this causes us problems. This is because our repo is structured in a way so that most projects have test dependencies on almost all of the other projects. So then the focus plugin brings in almost all the projects when focusing on anything. For best cases, focus plugin is able to reduce the number of projects to ~200 but the most practical cases will bring at least 600 projects. So it does not provide any benefit.
Restructuring the codebase is not an option. This is because most of our tests are integration tests so the tests end up needing a lot of dependencies.
Hence, we wanted to try another idea. The idea is complicated and probably we should not do it but we are out of good ideas currently -_-
Let's say A depends on B. This would be defined in the build file of A in some form. An example can be implementation(project(":B")) . But all A needs is the B.jar that is produced by compiling B. So if B.jar is available then implementation(files("B.jar")) should be enough for A. Hence, we do not need to include B anymore and then both Gradle and IDE overhead will be lower.
We actually implemented the above idea. The interface is a yaml file where users can specify which projects/directories they want to focus on and also specify what projects/directories to exclude. After that the users need to run focusSync task which generates custom build files for the relevant projects and final .focus file which contains what projects to load with what build files. After the users need to run IDE sync.
This actually works. We are able to focus on any project and control how many projects are loaded. We have to include some projects always but it is only 7 and we can even try to reduce it but it is not really worth it because 7 projects overhead is quite low. Usually our users end up with around 20 projects to load. And it improves Gradle and IDE overhead so much. One downside is that focusSync+IDE sync time is felt if you need to change focus frequently but we have some ideas for that as well.
Thank you for reading so far :)
We ran into a tough problem that I need some guidance for. So let's say:
- project A-> project B-> project C ('->' means depends on).
- project C has a class
Foo - project B has the following function:
fun getFoo(): Foo - project A calls
getFoo - Focus on project A and project C. Exclude project B.
In this case, after focusSync + IDE sync, if you open the file in project A that calls getFoo , the IDE will show an error. The error is:
Cannot access class 'Foo'. Check your module classpath for missing or conflicting dependencies
In reality, if you go ahead and compile A or runs tests from A, they all work fine from Gradle's point of view. But the fact that the IDE shows an error is causing friction for our users. I was wondering if I can get the IDE to recognize that Foo is in project C? My guess that the IDE sync assumes that a jar file can't depend on a project which is understandable. But can that behavior be configured or changed?
请先登录再写评论。
Hello, Elshad!
Thank you for reporting this!
According to your description, the issue here appears to be that Gradle, somehow, gives IDEA incomplete/malformed Project Model, while using the correct one itself.
This should be possible to rectify by creating an IDEA Plugin or use of the Gradle's IDEA Plugin, to add the dependency between "Project A" and "Project B".
Also, if you would be able to provide us with a reproducer that does not utilize the Gradle's Focus Plugin we can look further into it.
What we need 3 Projects with minimal code and the same dependency structure (A → B → C) with build scripts modified the same way Focus Plugin modifies them when you focus on Projects A and C and where Gradle builds successfully and IDEA fails to resolve the classes.
Hello Roman,
Thank you for quick response!
I created a sample Gradle project that has the setup I mentioned above. It does not use the focus plugin. Though it has a README.md on how to replicate the issue. I made sure that you will do minimum amount of work. Please let me know if you are able to replicate the issue.
EDIT: I had the wrong link. Now it is correct.
Elshad Mustafayev Thank you very much for providing a reproducer!
After investigating the issue we've determined that this is a Kotlin-specific problem and created KT-66269 for it.
Please keep an eye on it for further updates!
Roman, thank you very much for looking into it and creating the ticket!
This is my first time interacting with IntelliJ support. Do you know how long it usually takes such a ticket to get addressed? If it is too long, is there an opportunity to contribute?
Elshad Mustafayev The issue here is on the Kotlin end, which is not a team I'm a part of, so it's hard for me to tell how exactly the issue will be processed.
Both Kotlin and IDEA are open source, though - please feel free to take a look and if you happen to find any potential clues on this issue make sure to share them in the YouTrack Issue comments!
Number of upvotes on the YouTrack Issue is also a factor that determines the issue's priority.
If you can ask the affected users to upvote the issue, that should help to speed up the resolution too.
I see. Well then thank you Roman! Really appreciate quick help.
I will follow the ticket then.