Fatal crashes (IDEA X EAP & plugin 0.4.207): incl. data loss!

I'm seeing random JVM crashes (no stacktrace) every other hour or so. MacOS X, latest JVM update, IDEA 98.187, plugin 0.4.207.

There's definitely something weird going on with Macs, their JVM and IDEA.

I started having problems with the UE #95.538 while colleagues with the corresponding CE, als on the Mac, had complete stability. I upgraded to the latest IDEA X EAP and have not had a problem since. When I was having the JVM crashes, I couldn't work more than about 10 or 15 minutes without the JVM dying out from under IDEA.

Unfortunately, I installed Apple's update to the latest JVM somehwere in this timeline, but I don't remember if it was before or after the first crashing started.


Randall Schulz

I guess I should have added my platform details right in my original post.
I was on OS/X Java 1.6.0_20, IDEA IU 98.187 and plugin 0.4.207 at the time.

Meanwhile I upgrade to Java OS/X 1.6.0_22.
However, I did not experience any improvement.
The JVM underneath IDEA continues to die on me a few times a day...
Sometimes I see an error stack trace upon restart, sometimes I don't. However, I think these errors are more related to the inconsistent workspace state left over from the crash rather than the original cause.

Cheers,
Frank

Another crash, this time I copied the relevant bits of the crash report.
Here they are:

Process:         JavaApplicationStub [7503]
Path:            /Applications/IdeaX-IU-98.187.app/Contents/MacOS/idea
Identifier:      com.jetbrains.intellij
Version:         IdeaX (IU-98.187)
Code Type:       X86 (Native)
Parent Process:  launchd [140]

Date/Time:       2010-10-27 15:57:23.905 +0200
OS Version:      Mac OS X 10.6.4 (10F569)
Report Version:  6

Interval Since Last Report:          4529560 sec
Crashes Since Last Report:           40
Per-App Interval Since Last Report:  269976 sec
Per-App Crashes Since Last Report:   8
Anonymous UUID:                      13E74704-F46D-4052-B82E-FB8FBCE6F527

Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000000000008
Crashed Thread:  46  Java: JobScheduler pool 1/4

Application Specific Information:

Java information:
Exception type: Bus Error (0xa) at pc=00000000038d3afe

Java VM: Java HotSpot(TM) Client VM (17.1-b03-307 mixed mode macosx-x86)

Current thread (00000000046a8400):  JavaThread "JobScheduler pool 1/4" [_thread_in_vm, id=-1271390208, stack(00000000b4282000,00000000b4382000)]
Stack: [00000000b4282000,00000000b4382000]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::monitorenter_nofpu Runtime1 stub
J  com.intellij.util.CachedValueBase.getUpToDateOrNull()Ljava/lang/Object;
J  com.intellij.psi.impl.PsiCachedValueImpl.getValue()Ljava/lang/Object;
j  com.intellij.psi.impl.file.PsiPackageImpl.a()Ljava/util/Collection;+37
J  com.intellij.psi.impl.file.PsiPackageImpl.getDirectories(Lcom/intellij/psi/search/GlobalSearchScope;)[Lcom/intellij/psi/PsiDirectory;
J  com.intellij.psi.impl.JavaPsiFacadeImpl$PsiElementFinderImpl.getClasses(Lcom/intellij/psi/PsiPackage;Lcom/intellij/psi/search/GlobalSearchScope;)[Lcom/intellij/psi/PsiClass;
J  com.intellij.psi.impl.JavaPsiFacadeImpl.getClasses(Lcom/intellij/psi/impl/file/PsiPackageImpl;Lcom/intellij/psi/search/GlobalSearchScope;)[Lcom/intellij/psi/PsiClass;
j  com.intellij.psi.impl.file.PsiPackageImpl.getClasses(Lcom/intellij/psi/search/GlobalSearchScope;)[Lcom/intellij/psi/PsiClass;+21
j  com.intellij.psi.impl.file.PsiPackageImpl.processDeclarations(Lcom/intellij/psi/scope/PsiScopeProcessor;Lcom/intellij/psi/ResolveState;Lcom/intellij/psi/PsiElement;Lcom/intellij/psi/PsiElement;)Z+268
J  org.jetbrains.plugins.scala.lang.psi.impl.ScalaFileImpl.processDeclarations(Lcom/intellij/psi/scope/PsiScopeProcessor;Lcom/intellij/psi/ResolveState;Lcom/intellij/psi/PsiElement;Lcom/intellij/psi/PsiElement;)Z
J  org.jetbrains.plugins.scala.lang.psi.implicits.ScImplicitlyConvertible$class.treeWalkUp$1(Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression;Lcom/intellij/psi/PsiElement;Lcom/intellij/psi/PsiElement;Lorg/jetbrains/plugins/scala/lang/psi/implicits/ScImplicitlyConvertible$CollectImplicitsProcessor;)V
j  org.jetbrains.plugins.scala.lang.psi.implicits.ScImplicitlyConvertible$class.buildImplicitMap(Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression;Lscala/Option;)Lscala/collection/Seq;+27
j  org.jetbrains.plugins.scala.lang.psi.implicits.ScImplicitlyConvertible$class.implicitMap(Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression;Lscala/Option;)Lscala/collection/Seq;+11
j  org.jetbrains.plugins.scala.lang.psi.impl.expr.ScAssignStmtImpl.implicitMap(Lscala/Option;)Lscala/collection/Seq;+2
J  org.jetbrains.plugins.scala.lang.psi.api.expr.ScExpression$class.inner$1(Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression;ZZLscala/Option;)Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression$ExpressionTypeResult;
J  org.jetbrains.plugins.scala.lang.psi.api.expr.ScExpression$class.getTypeAfterImplicitConversion(Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression;ZZLscala/Option;)Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression$ExpressionTypeResult;
j  org.jetbrains.plugins.scala.lang.psi.impl.expr.ScAssignStmtImpl.getTypeAfterImplicitConversion(ZZLscala/Option;)Lorg/jetbrains/plugins/scala/lang/psi/api/expr/ScExpression$ExpressionTypeResult;+4
J  org.jetbrains.plugins.scala.lang.psi.types.Compatibility$Expression.getTypeAfterImplicitConversion(ZZLscala/Option;)Lscala/Tuple2;
J  org.jetbrains.plugins.scala.lang.psi.types.Compatibility$.doNoNamed$1(Lorg/jetbrains/plugins/scala/lang/psi/types/Compatibility$Expression;Lscala/collection/Seq;ZZLscala/runtime/ObjectRef;Lscala/runtime/BooleanRef;[Z)Lscala/collection/immutable/List;
J  org.jetbrains.plugins.scala.lang.psi.types.Compatibility$.checkConformanceExt(ZLscala/collection/Seq;Lscala/collection/Seq;ZZ)Lorg/jetbrains/plugins/scala/lang/psi/types/Compatibility$ConformanceExtResult;
J  org.jetbrains.plugins.scala.lang.psi.types.Compatibility$.compatible(Lcom/intellij/psi/PsiNamedElement;Lorg/jetbrains/plugins/scala/lang/psi/types/ScSubstitutor;Lscala/collection/immutable/List;ZLcom/intellij/psi/search/GlobalSearchScope;Z)Lorg/jetbrains/plugins/scala/lang/psi/types/Compatibility$ConformanceExtResult;
J  org.jetbrains.plugins.scala.lang.resolve.processor.MethodResolveProcessor$.org$jetbrains$plugins$scala$lang$resolve$processor$MethodResolveProcessor$$problemsFor(Lorg/jetbrains/plugins/scala/lang/resolve/ScalaResolveResult;ZLorg/jetbrains/plugins/scala/lang/resolve/processor/MethodResolveProcessor;)Lorg/jetbrains/plugins/scala/lang/psi/types/Compatibility$ConformanceExtResult;
J  org.jetbrains.plugins.scala.lang.resolve.processor.MethodResolveProcessor$$anonfun$mapper$1$2.apply(Lorg/jetbrains/plugins/scala/lang/resolve/ScalaResolveResult;)Lorg/jetbrains/plugins/scala/lang/resolve/ScalaResolveResult;
J  org.jetbrains.plugins.scala.lang.resolve.processor.MethodResolveProcessor$$anonfun$mapper$1$2.apply(Ljava/lang/Object;)Ljava/lang/Object;

...

Two things:

1) It's preferable to make such large, formatted text an attachment, not in-line text.
2) You do realize that this is not IDEA crashing, it is the JVM crashing, right? It cannot be blamed on IDEA.

Anyway, it is the same symptom I was experiencing repeatedly with 9.0.4 EAP (#95.538) but which I have yet to see repeated since I upgraded to IDEA X EAP (#98.187).


Randall Schulz

Randall,

thanks for your comment. You are certainly right about #1.
And I do realize that it is the JVM that crashes.
However, since this does happen with certain versions of IDEA and not with others I'm guessing that there is something inside of IDEA that triggers the crashes. Even though the defect is in the JVM implementation there might be a workaround Jetbrains can apply to prevent crashes on the Mac.
I'm sure it is interesting to Jetbrains that their customers are experiencing stability issues with their product.
Ultimately it is unimportant to their business customers whether the root cause of these crashes is on the Apple side or not, which is why I assume that Jetbrains wants to do all it can to alleviate the problem.

It would not be the first time Jetbrains adapts their code in order to workaround JVM implementation problems...

Cheers,
Frank

Yes, there may well be some pattern of activity in IDEA that leads to this crash, but given the disparity of situations in which it occurs and the unpredicatable nature of the crash, it's probably going to be quite difficult to even find, let alone fix, the code or execution pattern that leads to the crash.


Randall Schul

IllegalArgumentException at PersistentEnumerator.valueOf is a known bug in some IDEA EAPs, while the cause of JVM crashes lays between hardware, OS and JVM. I recommend to test RAM for errors for a start.

I dont believe this is just a mac problem.

I have 2 different machines running win7, either of which will have the jvm disappear within 60 minutes when using Idea X on a project that works perfectly using Idea 9.
Have invalidated caches etc.
This is using JDK1.6.0u22

Andreas

Yes, the last EAP seems to have solved the problem (regardless of who to blame - IDEA or the JVM). I haven't worked extensively with the newest version, but had a 5 hour session without crashes yesterday so it seems as if it has been resolved.

So bizarre...

I upgraded to the latest EAP the first thing yesterday and about 10 hours later there was another JVM crash.

Must I install Linux on my Mac? ... I'm not really opposed to doing that, but it will be tedious and I'll probably need more RAM.


Randall Schulz

Whoops, got a new one (attached).

I had some pretty strange JVM crashes for a while. It turned out I upgraded with a broken ram module. Replacing the ram solved the JVM crashes.

Regards,
Wallaby

I certainly haven't ruled out hardware problems, but it is odd that the only application that crashes is the JVM and then only when running IDEA. And the crash always happens in the "Parallel GC Threads" thread.


Randall Schulz

Gentlemen,

I seem to have found a solution to the JVM crashing problem (well, at least on my machine).
About a week ago I changed my config to force IDEA into using the 64bit JVM instead of the default 32bit one.
I have yet to have one crash since then, while during the week before I did have several crashes per day.

Here is how to force IDEA into 64bit mode:
- Close IDEA
- Open the Info.plist in the Contents directory underneath your IDEA app folder
- Move the "<string>x86_64</string>" setting underneath the JVMArchs and LSArchitecturePriority keys from second up to first position.
- Restart IDEA
- Verify 64bit mode by opening Activity Monitor and checking the "Kind" column of the row for the IDEA process

I am currently downloading the new IDEA X EAP build released today. Hopefully it is as stable as the last one was in 64bit mode.

There were two other things I had planned to try hadn't this first move already worked:
- Switch from the client to the server VM
- Disable parallel GC (for most of the crashes happened in a Gang Worker thread used by Parallel GC)

Maybe this works for some of you, if switching to 64bit somehow doesn't help.

Cheers,
Frank

I should have followed up on this earlier...

I tried 64-bit JVM on my Mac to no avail.

I finally quashed the crashes by disabling the profiler libraries that are included in the EAP releases. Unfortunately, at about the same time I stopped using the Mac-specific Look & Feel, which also uses native code libraries.

Either of these could have been corrupting the heap and leading to the crashes I experienced. I'm not in the mood to try an experiment to see which it might be...


Randall Schulz

Well, while I said I was not in the mood to conduct the experiment that would determine whether the native L&F library or the YourKit library was to blame, I upgraded to #93.311 shortly after writing that and forgot to edit the new app's Info.plist file to disable the YourKit profiling agent. After editing away for about an hour or so, ka-BOOM! Same old crash:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00000000e4202c3a
Crashed Thread:  6  Java: Gang worker#2 (Parallel GC Threads)


So I think we have our confirmation (I do, anyway): The culprit is the YourKit profiling agent included with and enabled by default on EAP releases.


Randall Schulz

Thanks, Randall, for sharing this information.
I guess we now have a whole set of possible attack vectors to deal with these annoying crashes...

Cheers,
Frank (still without any crash since the switch to 64bit 10 days ago)

Following Randall's post yesterday I removed the profiling agent from both the 9.0.4 eap & X eap vm options & didn't have any problems yesterday.

This is under Win7 with jdk 1.6.0_22

Andreas