EAP 992 - now we can't even import openapi.vfs.VirtualFile!

Dave Kriewall

Created December 09, 2003 19:47

There are a couple of bugs, http://www.intellij.net/forums/thread.jsp?forum=13&thread=45300 and http://www.intellij.net/forums/thread.jsp?forum=13&thread=48597 that describe some serious problems we plugin developers are experiencing, namely, we can't use or include certain classes from openapi.jar or idea.jar. The classes appear corrupted, and this is apparently due to a bug in the obfuscator.

Now, with build 992, we can't even use VirtualFile. Whether I look in openapi.jar or idea.jar for this class (and why is it in both places?), I get the "bad class file, bad signature" error.

This is getting really hard to work around. The open API isn't open if you can't use its classes!

IntelliJ, is there a possibility that

1) a new version of the obfuscator is going to be available, say, later today? :) -OR-

2) I could sign a non-disclosure agreement and get an un-obfuscated copy of idea.jar and openapi.jar? I just want to continue plugin development work.

Thanks!

-Dave

18 comments

ANONYMOUS

Created December 10, 2003 00:04

On Tue, 09 Dec 2003 19:47:02 +0000, Dave Kriewall wrote:

Now, with build 992, we can't even use VirtualFile. Whether I look in
openapi.jar or idea.jar for this class (and why is it in both places?), I
get the "bad class file, bad signature" error.

I noticed several classes are being reported as having generics attached
to them ( such as the Key class used in putUserData() - this will probably
trip us up as well.

--
Discouragement is a dissatisfaction with the past, a distaste for the
present, and a distrust of the future - Maree De Jong, CLCA.

Paul Mclachlan

Created December 12, 2003 02:20

I've posted a workaround here:

http://www.intellij.org/twiki/bin/view/Main/StubbingIdeaJar

Dave Kriewall

Created December 12, 2003 03:20

Thanks, Paul! I'll give it a try.
-Dave

Alexey Efimov

Created December 13, 2003 13:51

Paul,

Thank you for solution! Can you, please, give short description about issue?

Why only VirtualFile have broken signature from generics side, and have VALID signature from non-generics code...

The bcel in %IDEA_HOME%/lib is up-to-date, i use last generics - 2.2. That wrong?

Thanks!

Alexey Efimov

Created December 13, 2003 15:16

I post a little hotfix with description to your topic.
Thanks!

Paul Mclachlan

Created December 14, 2003 00:14

In one of the bug report descriptions a Jetbrain engineer said this was a problem with the obfuscator -- that the obfuscator doesn't work that well with generic'd classes.

That theory fits with the behavior I've seen. My understanding is that generic classes have a bit of extra metadata stored in the .class file. Theory goes that the obfuscator is corrupting that metadata...

The normal javac compiler just ignores this metadata attribute, so it doesn't matter if it's corrupt or not.

But, the EA generics javac tries to use the metadata attribute and, since it contains garbage, it complains and gives up.

The patch strips out that metadata attribute (although I do it an awkward way because I'm too lazy to read about how the metadata is really implemented and just strip that, so I just rebuild the classes from scratch without including the metadata in the first place).

The problem certainly isn't just with VirtualFile - there are a bunch of classes that have this problem. I originally ran into one with PsiTreeUtil. I first noticed it back in build 95x and slowly more and more classes became affected - I'm not sure what's going on to make it get worse like this, unless JetBrains has been adding generics to classes that previously didn't have any (or upgraded a version of a compiler or something).

Hope this helps - I'm happy with the explanation that it's a obfuscator bug... I think with a workaround people won't complain too much, they're using a whole bunch of early-access stuff doing what they're doing... :)

Alexey Efimov

Created December 14, 2003 21:45

Seems that you are right! :)

But it surprise for me, i think before, that generics compiler just convert generics to JDK 1.4 code... I think that metadata is not used, but i was wrong, as it hapend.

In any case, thank you very much for rewriter, it allow me continue my work :)

Dave Kriewall

Created December 16, 2003 21:31

For your convenience I uploaded stub .jar files (openapi.jar and idea.jar) for build 996 at http://www.intellij.org/twiki/bin/view/Main/StubbingIdeaJar.

Dave Kriewall

Created December 23, 2003 19:17

Hi Paul,

There's a problem with the stub generator. It is apparently stripping constants out of classes; e.g. in idea.jar,

is no longer recognized.

IntelliJ folks, this is still a showstopper for my plugin development. Even if I manage to compile with Paul's stubbed .jar files and then revert the classpath to the original idea.jar and openapi.jar files, I get null pointer exceptions whenever I attempt to run a JUnit test, as follows. (I actually get 12 identical NPE's with interleaved output, but I've shortened it to just one here.) I'm suspicious that this is related somehow to the compilation procedure.

(Reader.java:61)
	at java.io.InputStreamReader.(InputStreamReader.java:80)
	at com.intellij.util.JDOMUtil.loadDocument(JDOMUtil.java:110)
	at com.intellij.openapi.components.a.a.a(a.java:167)
	at com.intellij.openapi.components.a.a.a(a.java:175)
	at com.intellij.openapi.components.a.a.e(a.java:165)
	at com.intellij.openapi.application.a.b.s(b.java:233)
	at com.intellij.openapi.application.a.b.(b.java:155)
	at com.intellij.openapi.application.b.b.a(b.java)
	at com.intellij.idea.a.(a.java:7)
	at com.intellij.idea.b.(b.java:3)
	at com.intellij.idea.b.a(b.java:1)
	at com.intellij.testFramework.LightIdeaTestCase.a(LightIdeaTestCase.java:54)
	at com.intellij.testFramework.LightIdeaTestCase.setUp(LightIdeaTestCase.java:70)
	at com.wrq.tabifier.IfStmtAlignmentTest.setUp(IfStmtAlignmentTest.java:33)
	at com.intellij.testFramework.LightIdeaTestCase.d(LightIdeaTestCase.java:31)
	at com.intellij.testFramework.LightIdeaTestCase.access$400(LightIdeaTestCase.java:26)
	at com.intellij.testFramework.LightIdeaTestCase$5.run(LightIdeaTestCase.java)
	at com.intellij.util.LaterInvokator$1.run(LaterInvokator.java)
	at com.intellij.util.LaterInvokator$FlushQueue.run(LaterInvokator.java:9)
	at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:178)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:454)
	at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:100)

java.lang.NullPointerException
	at com.intellij.testFramework.LightIdeaTestCase.b(LightIdeaTestCase.java:37)
	at com.intellij.testFramework.LightIdeaTestCase.a(LightIdeaTestCase.java:4)
	at com.intellij.testFramework.LightIdeaTestCase.setUp(LightIdeaTestCase.java:70)
	at com.wrq.tabifier.IfStmtAlignmentTest.setUp(IfStmtAlignmentTest.java:33)
	at com.intellij.testFramework.LightIdeaTestCase.d(LightIdeaTestCase.java:31)
	at com.intellij.testFramework.LightIdeaTestCase.access$400(LightIdeaTestCase.java:26)
	at com.intellij.testFramework.LightIdeaTestCase$5.run(LightIdeaTestCase.java)
	at com.intellij.util.LaterInvokator$1.run(LaterInvokator.java)
	at com.intellij.util.LaterInvokator$FlushQueue.run(LaterInvokator.java:9)
	at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:178)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:454)
	at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:100)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:100)

Process terminated with exit code -1]]>

May I renew my request to sign an NDA or something so that I can get an un-obfuscated version of the .jar files? I would at least like to determine if the "compile with stubs, run with original jars" approach is the cause of the NPE.

Thanks,
-Dave

Paul Mclachlan

Created December 24, 2003 00:21

Ahh, yes. Sorry about that. I've posted a fixed program on the wiki: http://www.intellij.org/twiki/bin/view/Main/StubbingIdeaJar

Let me know if they let you sign an NDA, that'd make my life a lot easier, too!

:)

Jacques Morel

Created December 24, 2003 06:17

I have no problem since I don't use Generics anymore.

I removed all generics from my plugins right after I filed http://www.intellij.net/forums/thread.jsp?forum=13&thread=45300. This was necessary due to the same problems you have.
Since the gjc generates standard jdk1.3 bytecode, idea.jar and openapi.jar are fine using standard javac. So AS LONG AS you do not enable gjc IN YOUR IDEA projects you will be fine. KlassMaster, IDEA obfuscator, seems to only corrupt generics specific data.

As a treat, as of 1024 you won't have the IDEA editor flag your class that extends/implements a generic type (UserDataHolder, Comparable...) because it would miss your non-generic implementation (almost EVERY SINGLE PSI mock class I have was marked red in the editor !!!)

So this is my advice: Just comment out your "]]>" (so you can remove the comment with a regexp find&replace) and turn your "T" into Object (gjc does that type erasure anyway) and do not use generics anymore. They are just not worth the trouble IMHO.

Jacques

Dave Kriewall

Created December 26, 2003 00:41

Yeah, Jacques, I think I'll have to cave in and strip the generics from my code for now. I understand that the production version of 4.0 will not contain any generics in the openapi.jar and idea.jar files, so we can probably go back to them then.

Thanks for the suggestion,
-Dave

Jacques Morel

Created December 26, 2003 05:15

the production version of 4.0 will not contain any generics in the openapi.jar and idea.jar files

What do you mean? Are they going to remove their usage of generics in their own code?
Or are you refering to the fact that we can use generics compiled code with a non-generics compiler even right now?

Jacques

Dave Kriewall

Created December 26, 2003 15:39

I was told (by EugeneB) that "in the release version of IDEA 4.0 there will be no class files using generics." Sounds like they are going to remove usage of generics in their own code. But they haven't done it yet (as of build 1032) since I still get corrupted class errors. The sooner the better, of course. :) I'm in the process of stripping out my generics usages. I suppose I'll be able to replace them safely with the 4.0 production version.

-Dave

Paul Mclachlan

Created December 26, 2003 15:51

Does my new patch not work for you?

Dave Kriewall

Created December 26, 2003 19:35

Hi Paul,

I am trying to find out why none of my JUnit tests will run (I get an NPE deep in IDEA code). I suspected that maybe it had to do with compiling with the stubs, or not getting the classpath right for execution, or something. So for the moment I am trying to eliminate generics as a factor. (So far, it doesn't seem to help, but I'll try a few more things before totally giving up on plugin development. :))

Anyway, I haven't tried your new patch yet, sorry. Between flu, holidays, and obfuscated generics, I haven't been very productive lately! (Acronym for that would be FHOG; as in, I've been in a FHOG.)

-Dave

Paul Mclachlan

Created December 26, 2003 19:45

Ahh, right. Well, give it a go, I built it just for you.

When I read your original problem, I immediately knew what was wrong with the stub generator - because it wasn't putting constants into the .class files it built. So it would turn

Blah.EQ = 5;

into

Blah.EQ = 0;

(Just because I'd never coded that bit up).

The problem is that javac doesn't (and hasn't since the 1.1 days, I think) put a reference to "Blah.EQ" into your classfiles when it compiles them. Instead, it actually puts the constant "5" in there. And, when you compiled against my stubs, it would be putting in "0" instead, which would cause all sorts of problems, I'm sure.

So, I fixed the stub generator to properly put in initialization values for primitive types. I'm sure it will get you up and running.

Alexey Efimov

Created January 20, 2004 20:19

Compliled all requered classes into one jar.
http://www.intellij.org/twiki/bin/view/Main/StubbingIdeaJar

Please sign in to leave a comment.