Different ClassLoaders

Does your plugin have tools.jar in its local library?

Yes. I added it to classpath. Without it, the code would not compile.

If it's possible to use tools.jar bundled with JDK, then tools.jar could be added to the classpath of JDK.
In that case plugin and IDE would use the same tools.jar.

Thanks, but it didn't help. The problem is different ClassLoaders one system and the other belongs to the plugin (each Idea plugin launches in its ClassLoader).

Yes, you're right. Adding tools.jar to JDK doesn't work. But adding tools.jar to classpath of "Intellij Platform Plugin SDK" should work (at least it worked for me).
Each classloader tries to load a class using its parent classloader, right?
If a class from tools.jar would be loaded using a platform classloader (not the plugin's classloader), then the class could be shared with other plugins/platform.

ClassLoader.getSystemClassLoader() = sun.misc.Launcher$AppClassLoader his parent sun.misc.Launcher$ExtClassLoader his parent is null
"final ProcessingEnvironment procEnv"  - sun.misc.Launcher$AppClassLoader his parent sun.misc.Launcher$ExtClassLoader his parent is null (one and the same object)
"this.procEnv" - JavacProcessingEnvironment load in com.intellij.util.lang.UrlClassLoader his parent null
From these data, I realized that the system is not parent ClassLoader  for ClassLoader plugin.

From these data, I realized that the system is not parent ClassLoader  for ClassLoader plugin.

You're right. Parent classloader of plugin's own classloader isn't a system one.
Did it help to add tools.jar to the classpath of "Intellij Platform Plugin SDK"?

It did not help. Perhaps the reason is that the plugin uses another module (it uses JDK not IDEA SDK).

P.S. And frankly, I don't understand how it would help.

Seems I don't realise how your modules are structured. I thought that your plugin should have depended on "Intellij Platform Plugin SDK" (otherwise extension point declarations and other classes from Intellij Platform couldn't be used).
Could you please describe your module structure?

_

Yes, I can)
I have two modules: one of them is plugin, another module with annotation processor (he uses tools.jar).

Plugin depend on IDEA SDK, another module depend on JDK (IDEA SDK depend on the same JDK).

Plugin depends on annotation processor module, right?

Yes

Hmm, looks like I couldn't reproduce your problem:

private JavacProcessingEnvironment procEnv;

public synchronized void init(final ProcessingEnvironment procEnv) {
     super.init(procEnv);
     this.procEnv = (JavacProcessingEnvironment) procEnv; // here ClassCastException
}

Class variable "final ProcessingEnvironment procEnv"  - AppClassLoader = ClassLoader.getSystemClassLoader()
Class field "this.procEnv" JavacProcessingEnvironment load in another plugin ClassLoader

If I understood correctly, this code is from "annotation processor" module.
What is "procEnv.getClass()" when ClassCastException occurs?
If possilbe, please attach here a small project that reproduces the problem.

If I understood correctly, this code is from "annotation processor" module.

Yes

What is "procEnv.getClass()" when ClassCastException occurs?

procEnv.getClass() = JavacProcessingEnvironment.class

This project reproduce exception.

Thanks a lot! Reproduced.

The problem is that "InCompile.compiler.getClass().getClassLoader()" is "sun.misc.Launcher$AppClassLoader@1216f5a".
So when it needs to create an instance of "JavacProcessingEnvironment" class to pass it to "p.comp.Processor.init(procEnv)", it loads "JavacProcessingEnvironment" using the same classloader (sun.misc.Launcher$AppClassLoader@1216f5a).
But instance of "p.comp.Processor" was loaded using "PluginClassLoader", because it was created in "InCompile.compile" (and class of "InCompile" instance was loaded by "PluginClassLoader"). When "p.comp.Processor" initializes, it loads "JavacProcessingEnvironment" using the same classloader "PluginClassLoader".

Solution#1: replace "ToolProvider.getSystemJavaCompiler()" with "JavacTool.create()".

Solution#2: (I'm not sure it's a good one) add tools.jar as a VM option "-Xbootclasspath/a:/path/to/tools.jar" in the plugin run configuration. Make sure that tools.jar isn't in the classpath of the plugin, because otherwise the plugin will load its own version of JavacProcessingEnvironment class forcibly.
To make code compliable tools.jar can be added to JDK classpath (not "IDEA JDK").

Thank you very much, now I understand how it happened!

Solution#1: replace "ToolProvider.getSystemJavaCompiler()" with "JavacTool.create()".

Thank you very much. This was what I needed. My plugin was working on IntelliJ iDEA but not in Android Studio. This fixed it.

Note: There seems to be 2 `JavacTool`. I used `com.sun.tools.javac.api.JavacTool`