How to remove lib/guava-r09.jar from the classpath a plugin?

Hello chandra,

guava-r09.jar is loaded into the internal IntelliJ IDEA classloader, and
the plugin class loaders inherit it, so there is no way to remove it from
the plugin classpath.

One of our plugin uses internal version com.google.common.collect
, and this plugin is
not compatible with guava-r09.jar

During execution I am getting exceptions like:
java.lang.NoSuchMethodError:
com.google.common.collect.Maps.immutableMap() - because intellij/jvm
is using guava-r09.jar releted classes.

Is there a way to remove lib/guava-r09.jar from the classpath of my
plugin during execution?

I tried removing lib/guava-r09.jar and replacing it with our internal
com.google.common.collect version - that seems to fix the plugin
related issues. But, my concern is that this might break some of the
intellij's functionality which is based on guava-r09.jar. So,for me
this is not an acceptable solution.

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Dmitry,

Here are few scenarios why Intellij 10 is using Guava-r09.jar:

1)  use it in intellij 10's own implementation code

Dmitry,

http://www.jetbrains.com/idea/plugins/plugin_structure.html (plugin class loaders) says that intellij uses different class loader for each plugin, this enables the plugins to use different version of library even if the library is used by intellij. So, Guava-r09.jar related conflict should not arise. correct?

I did quick search at intellij community editon code and found i) use-idea-classloader propery and ii) com.intellij.ide.plugins.PluginManager#createPluginClassLoader interesting. Can I make use of these two properties/method to override existing behaviour of intellij 10 with respect to class loader of plugin?

Thanks,
Chandra

Hello chandra,

IntelliJ IDEA has traditionally exposed the libraries used by the IntelliJ
IDEA code itself to third-party plugin developers. That way the plugin developers
can avoid bundling an extra copy of the libraries with their own code and
reduce the size of the plugin distribution. (We don't carry any libraries
just for the sake of third-party plugin developers; all libraries in idea\lib
directory are used by IDEA own code). We do not plan to change this behavior
or to obfuscate our usage of any libraries.

Actually I don't understand why this is an issue with Guava, unless Google's
internal version of the collections library is radically different from what's
released in the open-source project. The changes in the recent releases of
Guava have been minor and certainly not backwards-incompatible, so I don't
expect any problems for plugin developers when we upgrade to the next release
of Guava. (We do plan to keep updating to new versions of Guava as they are
released.)

Here are few scenarios why Intellij 10 is using Guava-r09.jar:

1)  use it in intellij 10's own implementation code
2)  use Guava types in intellij 10's API
3)  enable plugin writers to get Guava for free, thus keeping plugin
sizes smaller
I think it is mostly 1). If this is the case, then obfuscating Guava
related usage in intellij 10's code seems to be more logical. This
way, intellij 10 libs doesnt interfere with plugin's environment.
Another alternative is to make plugin specific classpath elements
precede the inherited classath elements.

As of now, the only solution for us seems to re-base our plugin usages
of com.google.common using a tool like jarjar (which is quite
fragile). Backporting our plugin's code (and dependent libs) to
guava-r09.jar is not a best solution (here are few reasons: i) we have
to duplicate code and ii) as of now in our code base com.google.common
refers to HEAD version by default - we need to change this to refer to
guava-r09.jar transitively - in all the libs we depend on; some other
users of these libs expect HEAD version of com.google.common)

Apart from this, lets say - if next version of Intellij uses
guava-r10.jar and some of intellij plugins requires guava-r09.jar or
viceversa - in this case you need to either i) obfuscate uses of
com.google.collect in intellij or ii) make plugin specific classpath
elements precede the inherited classpath elements.

Let me know if you need any other information regarding this.

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Dmitry,

Some of the APIs that are available in HEAD version of com.google.common are not available in opensource version yet. Our plugins [and the libraries that our plugins depend on use these additional APIs as well]. So, backporting to guava-r09.jar is tedious.

I tried creating a guava .jar from com.google.common related .class files (from our plugin's deploy binaries - which are built using latest com.google.common sources) and replaced guava-r09.jar with that .jar file. But, this results  in java.lang.LinkageError during plugin load/execution. Here is the  error message:

java.lang.LinkageError: loader  constraint violation: when resolving field "SAME_PACKAGE_FILTER" the  class loader (instance of com/intellij/ide/plugins/cl/PluginClassLoader) of the referring class, com/google/devtools/guide/rules/LaunchTargetMapBuilder, and the class loader (instance of com/intellij/ide/plugins/cl/PluginClassLoader) for the field's resolved type, com/google/common/base/BinaryPredicate, have different Class objects for that type

I guess this is due to two class instances 'com/google/common/base/BinaryPredicate' one part of guava-r09.jar and another part of intellig_platform.jar (which is part of our plugin)

Now, here are few possible solutions:
1. re-base usages of com.google.common in our plugin code to something else [which doesnt conflict with guava-r09.jar]
2. provide an option in intellij to either remove certain .jars from classpath or let plugin classpath elements take precedence over idea classpath elements.

Or, do you suggest any other alternatives?

Thanks.
Chandra

Hello chandra,

The plugin classpath elements do take precedence over the built-in IDEA jars.
See PluginClassLoader class line 57.

There is no way for us to provide an option to remove jars from a plugin
classpath. The jars are loaded in the IDEA's own classloader, which is necessary
because they are used by the IDEA code, and they are inherited by the plugin
classloader, which doesn't have a way to distinguish IDEA's own classes,
classes from libraries which must be inherited, and some other classes. (And
no, we don't plan to introduce any Guava specific hacks into PluginClassLoader.)

Replacing guava-r09.jar in the IntelliJ IDEA lib directory with your own
copy of guava should work just fine if you don't package a second copy of
the com.google.common.* classes into your own plugin jar.

And of course, the problem will disappear as soon as the missing APIs appear
in the open-source version.

Some of the APIs that are available in HEAD version of
com.google.common are not available in opensource version yet. Our
plugins [and the libraries that our plugins depend on use these
additional APIs as well]. So, backporting to guava-r09.jar is tedious.

I tried creating a guava .jar from com.google.common related .class
files (from our plugin's deploy binaries - which are built using
latest com.google.common sources) and replaced guava-r09.jar with that
.jar file. But, this results  in java.lang.LinkageError during plugin
load/execution. Here is the  error message:

java.lang.LinkageError: loader  constraint violation: when resolving
field "SAME_PACKAGE_FILTER" the  class loader (instance of
com/intellij/ide/plugins/cl/PluginClassLoader) of the referring class,
com/google/devtools/guide/rules/LaunchTargetMapBuilder, and the class
loader (instance of com/intellij/ide/plugins/cl/PluginClassLoader) for
the field's resolved type, com/google/common/base/BinaryPredicate,
have different Class objects for that type

I guess this is due to two class instances
'com/google/common/base/BinaryPredicate' one part of guava-r09.jar and
another part of intellig_platform.jar (which is part of our plugin)

Now, here are few possible solutions:

1. re-base usages of com.google.common in our plugin code to something
else

2. provide an option in intellij to either remove certain .jars from
classpath or let plugin classpath elements take precedence over idea
classpath elements.

Or, do you suggest any other alternatives?

Thanks.
Chandra
---
Original message URL:
http://devnet.jetbrains.net/message/5305041#5305041

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Dmitry,

Thanks for the clarification.

Lets say,  we have plugin 'base' and another plugin B which depends on base . Here is dependency graph of our plugins:
com.intellij <- base plugin <- B plugin.

Lets say, if  a class is not found by classloader of B plugin, I expect classloader  of base plugin to load that class, if the class is not found by base plugin classloader then com.intellij classloader should try to load this. Is this  expectation correct? But, what I observed is, if a class is not found by B plugin, then com.intellij classloader is used ahead of base plugin classloader to load that class. Is this expected behaviour?

This base plugin has intellig_platform.jar (which contains internal version of com.google.com). I used debugger to see how plugin classloaders are loading com.google.collect related classes using community edition source code. Here is what I observed:
In case of base plugin - com.google.common.collect.Maps is picked properly from intellig_platform.jar
Where as in case of B plugin, com.google.common.collect.Maps is not part of its classpath. Now, com.intellij.ide.plugins.cl.PluginClassLoader#loadClassFromParents() is invoked. For plugin B its parents are com.intellij.util.lang.UrlClassLoader and PluginClassLoader[base]. As com.intellij classloader is ahead of base plugin classloader - com.google.common.collect.Maps is returned by com.intellij classloader from guava-r09.jar.

com.intellij classloader is at head of parent classloaders of any plugin. Is this by design? If we change order of parents to base plugin classloader followed by com.intellij classloader then class loading seems to work as we need.

UPDATE:
com.intellij.ide.plugins.PluginManager#initializePlugins() inserts com.intellij classloader as dependency at position 0. So, inserting com.intellij classloader at head of parent classloaders seems to be design.

Thanks,
Chandra

Dmitry,

I have filed a feature request to override/change classpath loaders order. http://youtrack.jetbrains.net/issue/IDEA-70535.

I am working on fix for this feature request. Here is my approach:
Add  <depends_order></depends_order> element to plugin.xml which  specifies order of plugin dependencies. Plugins specified in this  depends_order will be at head of parents/depends list of this plugin.  Order of dependencies that are not listed will not be affected.

Here is an example for B plugin:
<depends_order><dependent_plugin_name>base</dependent_plugin_name></depends_order>
This puts base plugin at head of parents/depends of plugin B.

This gives the plugin writers freedom to specify order of class-loaders.

Let me know if you have any concerns with this approach.

Thx,
Chandra

Hey Chandra,

did you declare the dependency of B to base plugin in the plugin.xml of B?
If so, the classloader of base plugin should be consulted before the IDEA classloader if the B classloader doesn't find a class.
That is exactly what you requested, isn't it?

At least http://confluence.jetbrains.net/display/IDEADEV/IntelliJ+IDEA+Plugin+Structure says that it should be like that:

[quote]
By default, the main IDEA class loader loads classes that were not found  in the plugin class loader. However, in the plugin.xml file, one can  use the <depends> element to specify that a plugin depends on one  or more other plugins. In this case, the class loaders of those plugins  will be used for classes not found in the current plugin. This allows a  plugin to reference classes from other plugins.
[/quote]

Regards
Björn