How to do expensive stuff in Jps Build Process before compiling
I am trying to improve performance of incremental build of Gosu compiler.
I see that a new jps process is started as soon incremental compilation is done which helps the next incremental compilation to be faster.
But inside the GosuBuilder#build method, we have some expensive operation i.e. initializing some stuff and reading some libs which takes about 3 seconds.
Is there a way to do this kind of expensive operation before the build() method is called. i.e. as soon as the new jps build process is started, can we do the initialization of extra stuff there. If yes, what and where can I look into?
-------Update-------
I saw another answer on the forum and found that I can use the BuilderService constructor to initlialize some stuff. Is that right?
But now the problem comes that I need some information from the first compile for ex. context, moduleChunk and compilerDriver.
So what I am thinking is.. Do the first incremental compile the normal way which will be expensive and persist the required stuff somewhere. And when the new jps build process starts, get that persisted stuff and use it inside the GosuBuilderService to do the expensive stuff which will be ready before the next incremental compile.
Does that sound feasible? or is it totally not doable?
If it is doable, is there some intellij built-in mechanism i can use to persist some stuff and read it in the new jps build process? as the data is not serializable as CompileContext and ModuleChunk are not serializable
Thanks
Please sign in to leave a comment.
-------Update------- (updated the description too above with this comment)
I saw another answer on the forum and found that I can use the BuilderService constructor to initlialize some stuff. Is that right?
But now the problem comes that I need some information from the first compile for ex. context, moduleChunk and compilerDriver.
So what I am thinking is.. Do the first incremental compile the normal way which will be expensive and persist the required stuff somewhere. And when the new jps build process starts, get that persisted stuff and use it inside the GosuBuilderService to do the expensive stuff which will be ready before the next incremental compile.
Does that sound feasible? or is it totally not doable?
If it is doable, is there some intellij built-in mechanism i can use to persist some stuff and read it in the new jps build process? as the data is not serializable as CompileContext and ModuleChunk are not serializable
Hi Vicky,
> I see that a new jps process is started as soon incremental compilation is done which helps the next incremental compilation to be faster.
Yes, this is a so-called "process preloading" feature. The started process initializes some data that is unlikely to change between builds and waits for a build request. The initialized data is project file system state and builders configuration. If IDE detects that this data changed, it cancels pre-loaded process so that the next build re-initializes everything from scratch. Such situation does not happen very often, allowing the IDE to benefit from pre-initialized build state.
The keyword here is "the data that is common to all kinds of builds for this project and is unlikely to change between JPS invocations". As builder state is also initialized at at this stage, you are right that it is ok to put your initialization code into your builder's constructor or into the BuilderService.
Your builder can also rely on some customly written caches. The rule is to use org.jetbrains.jps.builders.storage.BuildDataPaths interface to find out the place where to put your data associated with particular project:
DataPaths paths = compileContext.getProjectDescriptor().datamanager.getDataPaths();
DataPaths.getDataStorageRoot() will return you the root directory where all data associated with this project is stored. Here you can create a subdirecotry "<my-plugin-name>" and store everything you need in any format you prefer.
There are also methods to obtain build target-related storage roots. This is convenient to store per-target pieces of data that will be cleared automatically as soon as JPS decides to rebuild particular build target from scratch.