external builder remember sources between differents compilation


Hi all!

This is my problem:

I have written an external builder with category SOURCE_GENERATOR. It generates java source code in gen directory. The java files generated are registered to the outputConsumer using 


Also  I register the files to compile using 

JavaBuilderUtil.registerFilesToCompile(context, toCompile);

Also I have a ClassPortProcessor as done in GroovyBuilder to recompile sources that whatever reason are not compile in this round

Until here all (more or less) is ok. The problem comes here:

P.e.: I compile my custom language files and them generates 75 java files that are compiler perfect.

Then I modify custom language  file, the result is that p.e. 70 java files are generated, but for any reason I unknown the java builder expect to compile the 5 file that does not exist because in this build aren't generated throwing an error like

Error:java: error reading /.../Sample_0.java; /.../Sample_0.java (No such file or directory)

If I try to compile again come to run perfect.

Any idea?? What I'm doing wrong??


Comment actions Permalink

Hi Octavio, 

let's fix the problems step-by-step.

The first change concerns the source generator:

You are right to use outputConsumer.registerOutputFile(generatedJavaSourceFile, Arrays.asList(src1, src2, ....))
where src1 and src2 ... are the sources in your language that you used to generate the "generatedJavaSourceFile".
This call will tell the build system about the association between the file in the language your builder processes and these generated java sources.

You don't need to explicitly call JavaBuilderUtil.registerFilesToCompile(context, toCompile);.
Instead, you should mark "dirty" all generated sources that are supposed to be compiled by java compiler that will be called after your source generator:

FSOperations.markDirty(context, CompilationRound.CURRENT, generatedJavaSourceFile);

This call will make all your generated files visible to java builder that will be called later.
Please also note that the "gen" directory should be registered as a "generatedSources" directory within the build project model.
org.jetbrains.jps.android.AndroidSourceGeneratingBuilder is a good example of source generating builder that generates sources to be consumed by java builder.
GroovyBuilder is a a bit more specific in its tasks, so I would not have used it as a reference.

> that whatever reason are not compile in this round

This sounds suspicious :-) there should be no such non-deterministic behavior.
Let's first make the corrections I mentioned above and check how the situation changes. 

Comment actions Permalink

First thanks a lot for the answer.


I fixed the builder as you proposed and now I seems that all sources are well generating, but I'm having the same problem with the "dirty files memory". I tried the same experiment where between a compilation and another the number of generated classes decrease and I check with debugger using


that a java file that was generated in the last make and now doesn't exist is still marked dirty although the file does not exists.

I don't now if exists any way to get all dirty files and unmark files which does not exists.

Respect of mark directory as "generatedSources" I suposed that you refer to just mark in the module settings or dialog as "Generated sources root" so, it is always done. 

Any idea??

Thanks again!

For more information this is code of the builder class: https://www.dropbox.com/s/cgfnk8w13nfvjh6/TaraBuilder.java?dl=0

Comment actions Permalink

> hat a java file that was generated in the last make and now doesn't exist is still marked dirty although the file does not exists.

Those non-existing files should be marked as "deleted" using FSOperations.markDeleted() call (see AndroidSourceGeneratingBuilder for usage example). As soon as non-existing file is properly registered as "deleted", the build system should handle all the rest.

Previous state of the file system you can discover with the help of BuildDataManager:

BuildDataManager dm = compileContext.getProjectDescriptor().dataManager;

SourceToOutputMapping mapping = dm.getSourceToOutputMap(buildTarget);

for every source in your custom language that your builder should process in the current compilation session, you can obtain the list of corresponding files that you generated in the previous session. If now a different set of java files corresponds to your source, you can calculate a difference and remove all files from the previous compilation that are not relevant anymore. All removed files should be than marked as "deleted" with the aforementioned API call.

Note that if your generated files were properly registered with the OutputConsumer (and were therefore added to the SourceToOutputMapping), then the build system will automatically handle the case when the source in your language is removed: all corresponding generated java files will be removed too and all corresponding internal mappings updated accordingly.

Comment actions Permalink

With that is still not working.

The code that commit the compilation results is results 

private void commit(CompileContext context,
ModuleChunk chunk,
OutputConsumer outputConsumer,
Map<ModuleBuildTarget, String> finalOutputs,
Map<ModuleBuildTarget, List<OutputItem>> compiled) throws IOException {
registerOutputs(context, outputConsumer, compiled);
removeOldSources(context, compiled);

private void registerOutputs(CompileContext context, OutputConsumer outputConsumer, Map<ModuleBuildTarget, List<OutputItem>> compiled) throws IOException {
for (Map.Entry<ModuleBuildTarget, List<OutputItem>> entry : compiled.entrySet())
for (OutputItem outputItem : entry.getValue()) {
final File generatedFile = new File(outputItem.getOutputPath());
outputConsumer.registerOutputFile(entry.getKey(), generatedFile, Collections.singleton(outputItem.getSourcePath()));
FSOperations.markDirty(context, CompilationRound.CURRENT, generatedFile);

private void removeOldSources(CompileContext context, Map<ModuleBuildTarget, List<OutputItem>> compiled) {
for (Map.Entry<ModuleBuildTarget, List<OutputItem>> entry : compiled.entrySet())
try {
BuildDataManager dm = context.getProjectDescriptor().dataManager;
SourceToOutputMapping mapping = dm.getSourceToOutputMap(entry.getKey());
for (String source : mapping.getSources()) {
if (new File(source).exists()) continue;
FSOperations.markDeleted(context, new File(source));
} catch (IOException e) {

is ok?

Please sign in to leave a comment.