JDI ClassNotLoadedException

Could you provide a sample project to reproduce it? That would help a lot. Thanks!

Sure, I've created one at https://github.com/cursive-ide/cursive-2099. To reproduce:

1. Install Cursive (https://plugins.jetbrains.com/plugin/8090).
2. Clone the repo and open the project.
3. Open src/cursive_2099/core.clj
4. Put a breakpoint on line 5: (+ a b)
5. Debug the -main method using the green gutter run icon.
6. When execution stops at the breakpoint, try to evaluate expressions 'a' or 'b' (just the variable names, without the quotes).
7. The evaluation should hang. Sometimes I can see in the status bar that the evaluation message is flickering, this is because it's retrying repeatedly after receiving the exception.

Let me know if you need any more information.

I was able to reproduce the issue, thanks!

For some reason java.lang.Object is not reported as visible from the DynamicClassLoader used in evaluation, so jdi method args types checks fail. In 1.09 it does not happen. Definitely some changes in 1.10 trigger that.

I can fix the hang by fixing the endless Object class loading, but evaluation would still fail. We need to find the reason why Object is not reported as visible.

Thanks Colin Fleming for sharing this valuable information over here.:-)

Hi Egor, when you say Object is not visible, what do you mean by that? Which method in DynamicClassLoader is not behaving as you would expect? I had a quick look, and I can't see why the recent changes to that class would break anything.

Ok, I've investigated this some more, with some help from the Clojure team. As far as I can tell, DynamicClassLoader didn't change in 1.10 - there was one change to it that was later reverted. However using git bisect, I found the offending commit, it's this one:

https://github.com/clojure/clojure/commit/38705b49fd3dbae11e94c576ef49ff3eb1c47395

There's some details in the commit message, but that commit changed two things: it bumped the bytecode opcode version that Clojure generates from 1.5 to 1.8, and made a change to the generation of the stack map frames, specifically the getCommonSuperClass method which always returns java/lang/Object. Unless there's something about the opcode version that causes JDI to behave differently, I'm suspicious of the getCommonSuperClass change.

Having debugged this further I see what you mean by visibility now. However I'm no closer to understanding why Object isn't visible. Here's a list of the visible classes before and after the change:

0 = {com.sun.tools.jdi.ClassTypeImpl@39806} "class user$eval147$fn__148 (loaded by instance of clojure.lang.DynamicClassLoader(id=2791))"
1 = {com.sun.tools.jdi.ClassTypeImpl@39826} "class java.lang.Object (no class loader)"
2 = {com.sun.tools.jdi.ArrayTypeImpl@39827} "array class java.lang.Object[] (no class loader)"
3 = {com.sun.tools.jdi.ArrayTypeImpl@39828} "array class java.lang.Object[][] (no class loader)"
4 = {com.sun.tools.jdi.ClassTypeImpl@39829} "class clojure.lang.AFunction (loaded by instance of sun.misc.Launcher$AppClassLoader(id=2675))"
5 = {com.sun.tools.jdi.ClassTypeImpl@39830} "class user$eval147 (loaded by instance of clojure.lang.DynamicClassLoader(id=2791))"
6 = {com.sun.tools.jdi.ArrayTypeImpl@39831} "array class boolean[] (no class loader)"
7 = {com.sun.tools.jdi.ArrayTypeImpl@39832} "array class byte[] (no class loader)"
8 = {com.sun.tools.jdi.ArrayTypeImpl@39833} "array class byte[][] (no class loader)"
9 = {com.sun.tools.jdi.ArrayTypeImpl@39834} "array class char[] (no class loader)"
10 = {com.sun.tools.jdi.ArrayTypeImpl@39835} "array class int[] (no class loader)"
11 = {com.sun.tools.jdi.ArrayTypeImpl@39836} "array class short[] (no class loader)"
12 = {com.sun.tools.jdi.ArrayTypeImpl@39837} "array class long[] (no class loader)"
13 = {com.sun.tools.jdi.ArrayTypeImpl@39838} "array class long[][] (no class loader)"
14 = {com.sun.tools.jdi.ArrayTypeImpl@39839} "array class float[] (no class loader)"
15 = {com.sun.tools.jdi.ArrayTypeImpl@39840} "array class double[] (no class loader)"

0 = {com.sun.tools.jdi.ClassTypeImpl@40142} "class user$eval147 (loaded by instance of clojure.lang.DynamicClassLoader(id=2790))"
1 = {com.sun.tools.jdi.ClassTypeImpl@40143} "class clojure.lang.AFunction (loaded by instance of sun.misc.Launcher$AppClassLoader(id=2677))"
2 = {com.sun.tools.jdi.ClassTypeImpl@40133} "class user$eval147$fn__148 (loaded by instance of clojure.lang.DynamicClassLoader(id=2790))"
3 = {com.sun.tools.jdi.ArrayTypeImpl@40144} "array class boolean[] (no class loader)"
4 = {com.sun.tools.jdi.ArrayTypeImpl@40145} "array class byte[] (no class loader)"
5 = {com.sun.tools.jdi.ArrayTypeImpl@40146} "array class byte[][] (no class loader)"
6 = {com.sun.tools.jdi.ArrayTypeImpl@40147} "array class char[] (no class loader)"
7 = {com.sun.tools.jdi.ArrayTypeImpl@40148} "array class int[] (no class loader)"
8 = {com.sun.tools.jdi.ArrayTypeImpl@40149} "array class short[] (no class loader)"
9 = {com.sun.tools.jdi.ArrayTypeImpl@40150} "array class long[] (no class loader)"
10 = {com.sun.tools.jdi.ArrayTypeImpl@40151} "array class long[][] (no class loader)"
11 = {com.sun.tools.jdi.ArrayTypeImpl@40152} "array class float[] (no class loader)"
12 = {com.sun.tools.jdi.ArrayTypeImpl@40153} "array class double[] (no class loader)"

So, many of the base JDK classes are visible, but Object is not. I have no idea why, though.

It seems that only the generated bytecode version matters, if I change it back to 49 and leave the rest as is (COMPUTE_FRAMES etc.) it starts working.

Starting with version 50 it stops working... Will try to investigate further.

My be connected with changed bytecode verification in java 6, but not sure.

I think I moved a little further:

Compile.eval changes the classloader for the generated class, so when we evaluate it in the debugger jdi checks arguments in one (new) classloader but then tries to load the Object class into another (context) classloader. Not sure how it worked before :) But will try to find a way to make it work.

The workaround that seems to be working:

add

context.setClassLoader(invoke.declaringType().classLoader());

into ClojureCodeFragmentFactory right before the main invokeMethod (invoke invocation :)

Fantastic, thank you Egor! I haven't had a chance to try this today but I'll try it tomorrow and let you know.

After testing this today, I can confirm that the workaround fixes the issue - thank you again! I would never have worked that out on my own.

Are there likely to be any side effects of this? Is the EvaluationContext created for each evaluation, or might there be weirdness later if the context is later used with a different classloader?

ClassLoader in EvaluationContext is mostly used for loading not yet loaded classes and searching for classes (for example to invoke a static method). I'm not sure about possible side effects but if you call a method of the class that is loaded in a separate class loader, it seems correct to set it as the evaluation context class loader.

Unfortunatelly I was not able to find out the exact change in java 6 class loading/verification that caused this difference in the behavior :(