When will Intellij be usable again for Spark Build/Compilation?

I have struggled to obtain a working Spark 1.x build/debugging environment in IJ (latest Ultimate 13.1.3).   From the command line sbt compile | package | assembly are all fine.  But in IJ a few compilation errors are persistent and unable to be resolved.

Finally I noticed in the spark-dev mailing list that the AmpLab folks have recognized this and they do not trust/use the Intellij parser/compiler for Spark 1.X.

Please guys, try to make this work. Spark is maybe the highest profile Scala project. Try to get this  working. I am open to either Maven or sbt.Just let us know which one to use.


Here is the info taken from the spark-dev mailing list archives, from Reynold Xin

IntelliJ parser/analyzer/compiler behaves differently from Scala compiler,
and sometimes lead to inconsistent behavior. This is one of the case.

In general while we use IntelliJ, we don't use it to build stuff. I
personally always build in command line with sbt or Maven.

BTW the current issue is as follows - and it has been corroborated on the spark dev mailing-list by other developers.. But what I am looking for is not just to resolve the following issue but to really have this work end to end.

I followed the directions in the bug - to delete the mesos-18.1.jar But the following errors now happen. Note this error has also been reported in the spark dev mailing list - so the following is just corroborating what others have already noted.

Error:scalac:
while compiling: C:\apps\incubator-spark\sql\core\src\main\scala\org\apache\spark\sql\test\TestSQLContext.scala
during phase: jvm
library version: version 2.10.4
compiler version: version 2.10.4
reconstructed args: -classpath <long classpath . -bootclasspath C:\apps\jdk1.7.0_51\jre\lib\resources.jar;C:\apps\jdk1.7.0_51\jre\lib\rt.jar;C:\apps\jdk1.7.0_51\jre\lib\sunrsasign.jar;C:\apps\jdk1.7.0_51\jre\lib\jsse.jar;C:\apps\jdk1.7.0_51\jre\lib\jce.jar;C:\apps\jdk1.7.0_51\jre\lib\charsets.jar;C:\apps\jdk1.7.0_51\jre\lib\jfr.jar;C:\apps\jdk1.7.0_51\jre\classes;C:\Users\s80035683\.m2\repository\org\scala-lang\scala-library\2.10.4\scala-library-2.10.4.jar -deprecation -feature -unchecked -language:postfixOps
last tree to typer: Literal(Constant(org.apache.spark.sql.catalyst.types.PrimitiveType))
symbol: null
symbol definition: null
tpe: Class(classOf[org.apache.spark.sql.catalyst.types.PrimitiveType])
symbol owners:
context owners: object TestSQLContext -> package test
== Enclosing template or block ==
Template( // val <local TestSQLContext>: <notype> in object TestSQLContext, tree.tpe=org.apache.spark.sql.test.TestSQLContext.type
"org.apache.spark.sql.SQLContext" // parents
ValDef(
private
"_"
<tpt>
<empty>
)
// 2 statements
DefDef( // private def readResolve(): Object in object TestSQLContext
<method> private <synthetic>
"readResolve"
[]
List(Nil)
<tpt> // tree.tpe=Object
test.this."TestSQLContext" // object TestSQLContext in package test, tree.tpe=org.apache.spark.sql.test.TestSQLContext.type
)
DefDef( // def <init>(): org.apache.spark.sql.test.TestSQLContext.type in object TestSQLContext
<method>
"<init>"
[]
List(Nil)
<tpt> // tree.tpe=org.apache.spark.sql.test.TestSQLContext.type
Block( // tree.tpe=Unit
Apply( // def <init>(sparkContext: org.apache.spark.SparkContext): org.apache.spark.sql.SQLContext in class SQLContext, tree.tpe=org.apache.spark.sql.SQLContext
TestSQLContext.super."<init>" // def <init>(sparkContext: org.apache.spark.SparkContext): org.apache.spark.sql.SQLContext in class SQLContext, tree.tpe=(sparkContext: org.apache.spark.SparkContext)org.apache.spark.sql.SQLContext
Apply( // def <init>(master: String,appName: String,conf: org.apache.spark.SparkConf): org.apache.spark.SparkContext in class SparkContext, tree.tpe=org.apache.spark.SparkContext
new org.apache.spark.SparkContext."<init>" // def <init>(master: String,appName: String,conf: org.apache.spark.SparkConf): org.apache.spark.SparkContext in class SparkContext, tree.tpe=(master: String, appName: String, conf: org.apache.spark.SparkConf)org.apache.spark.SparkContext
// 3 arguments
"local"
"TestSQLContext"
Apply( // def <init>(): org.apache.spark.SparkConf in class SparkConf, tree.tpe=org.apache.spark.SparkConf
new org.apache.spark.SparkConf."<init>" // def <init>(): org.apache.spark.SparkConf in class SparkConf, tree.tpe=()org.apache.spark.SparkConf
Nil
)
)
)
()
)
)
)
== Expanded type of tree ==
ConstantType(
value = Constant(org.apache.spark.sql.catalyst.types.PrimitiveType)
)
uncaught exception during compilation: java.lang.AssertionError