Running Scala tests in IntelliJ throws "class not found" errors - works with "sbt test"
I am trying to use IntelliJ as an IDE for developing Scala apps to run under Apache Spark. I set up a new SBT-based Scala project in the latest version of IntelliJ with two simple tests.
When I run these tests with "sbt test" they both pass. When I run them within IntelliJ the second test fails, complaining that it can't find the class "HttpServletResponse" which is, as far as I can tell, pulled in by spark.
Tests:
import org.apache.spark.{SparkContext, SparkConf}
import org.scalatest.FunSuite
class ExampleTests extends FunSuite {
test("works, obviously!") {
assert(1 == 1)
}
test("broken") {
val conf = new SparkConf()
.setAppName("broken test")
.setMaster("local")
.set("spark.default.parallelism", "1")
val sc = new SparkContext(conf)
try{
assert(sc.parallelize("The complete works of Shakespeare".split("\\W")).map(_ + 1).count == 5)
}
finally {
sc.stop()
}
}
}
SBT:name := "NeighbourRelations"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.1.3" % "test"
Running tests with SBT on the command line:$ sbt test
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 [info] Loading global plugins from C:\Users\tay59806\.sbt\0.13\plugins [info] Loading project definition from D:\Dropbox\Progs\Spark\NeighbourRelations\project [info] Set current project to NeighbourRelations (in build file:/D:/Dropbox/Progs/Spark/NeighbourRelations/) [info] Compiling 1 Scala source to D:\Dropbox\Progs\Spark\NeighbourRelations\target\scala-2.10\test-classes... [info] NeighbourRelationshipUsageTest: [info] FunSuiteForSpark: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
...
[info] Run completed in 3 seconds, 544 milliseconds. [info] Total number of tests run: 2 [info] Suites: completed 3, aborted 0 [info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 6 s, completed 29-Dec-2014 12:17:02
Running tests in IntelliJ - second test fails:Testing started at 12:12 ... 14/12/29 12:12:51 INFO spark.SecurityManager: Changing view acls to: tay59806,
... A needed class was not found. This could be due to an error in your runpath. Missing class: javax/servlet/http/HttpServletResponse java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse at org.apache.spark.HttpServer.org$apache$spark$HttpServer$$doStart(HttpServer.scala:73) at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60) at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442) at org.apache.spark.HttpServer.start(HttpServer.scala:60) at org.apache.spark.HttpFileServer.initialize(HttpFileServer.scala:45) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:243) at org.apache.spark.SparkContext.<init>(SparkContext.scala:203) at ExampleTests$$anonfun$2.apply$mcV$sp(ExampleTests.scala:16) at ExampleTests$$anonfun$2.apply(ExampleTests.scala:10) at ExampleTests$$anonfun$2.apply(ExampleTests.scala:10) at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22) at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:158) at org.scalatest.Suite$class.withFixture(Suite.scala:1121) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1559) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:155) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:167) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:167) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:167) at org.scalatest.FunSuite.runTest(FunSuite.scala:1559) at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:200) at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:200) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401) at scala.collection.immutable.List.foreach(List.scala:318) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483) at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:200) at org.scalatest.FunSuite.runTests(FunSuite.scala:1559) at org.scalatest.Suite$class.run(Suite.scala:1423) at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1559) at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:204) at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:204) at org.scalatest.SuperEngine.runImpl(Engine.scala:545) at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:204) at org.scalatest.FunSuite.run(FunSuite.scala:1559) at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557) at scala.collection.immutable.List.foreach(List.scala:318) at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043) at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722) at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043) at org.scalatest.tools.Runner$.run(Runner.scala:883) at org.scalatest.tools.Runner.run(Runner.scala) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:137) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: java.lang.ClassNotFoundException: javax.servlet.http.HttpServletResponse at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 63 more
I can only assume that for some reason, IntelliJ is building things slightly differently to SBT. This means it can't find the appropriate libraries. I am new to IntelliJ and Scala, so all help appreciated - however simple!!
Thanks!
Please sign in to leave a comment.
There seems to be some differences involving org.eclipse.jetty
sbt classpath using - show test:full-classpath
.ivy2\cache\org.eclipse.jetty\jetty-plus\jars\jetty-plus-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.transaction\orbits\javax.transaction-1.1.1.v201105210645.jar)
.ivy2\cache\org.eclipse.jetty\jetty-webapp\jars\jetty-webapp-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-xml\jars\jetty-xml-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-util\jars\jetty-util-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-servlet\jars\jetty-servlet-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-security\jars\jetty-security-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-server\jars\jetty-server-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar)
.ivy2\cache\org.eclipse.jetty\jetty-continuation\jars\jetty-continuation-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-http\jars\jetty-http-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-io\jars\jetty-io-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-jndi\jars\jetty-jndi-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.mail.glassfish\orbits\javax.mail.glassfish-1.4.1.v201005082020.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.activation\orbits\javax.activation-1.1.0.v201105071233.jar)
Intellij passed classpath to runner ( look at the command line in the test runner folded at the top )
.ivy2\cache\org.eclipse.jetty\jetty-continuation\jars\jetty-continuation-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-http\jars\jetty-http-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-io\jars\jetty-io-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-jndi\jars\jetty-jndi-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-plus\jars\jetty-plus-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-security\jars\jetty-security-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-server\jars\jetty-server-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-servlet\jars\jetty-servlet-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-util\jars\jetty-util-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-webapp\jars\jetty-webapp-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-xml\jars\jetty-xml-8.1.14.v20131031.jar
javax.servlet-3.0.0.v201112011016.jar is the one with that missing class I think but it seems to be a bit fubar. Pretty new to this stuff and there is a lot of noise in the classpath to sift through.
Might be worth you creating an issue in the tracker with full info copied into files then uploaded for both methods. More likely to get attention as there is possibly a greater underlining issue with project analysis)
---
Just confirmed manually adding .ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar to the dependencies under the module with all the others makes it work.
HttpServletResponse is findable by the ide on navigate to the class (ctrl-n or whatever) when set up right.
Thanks very much for the response. I can confirm that adding the javax.servlet dependency does indeed fix the problem... though having to do it seems wrong somehow.
Hi! Thanks for the report.
I've investigated a bit further and find out that the source of this problem may be an incorrect resolution of dependencies when "download sources" is enabled. As a workaroud I suggest that you try to reimport project with "download sources" feature disabled. If you haven't used this option in a first place or if it's not helping please provide me with details on Youtrack.
I have 14.1.4 Ultimate and the problem still exists. Is there a YouTrack for this "Download Sources" issue?
How do we disable that option in an existing project? I can not find any setting in the Project Settings dialogs.
What version of Scala plugin do you use? This particular issue was fixed in 1.5.2. CNF could be caused as well by this issue which is not fixed yet.
I installed latest 14.1.4 maybe two weeks ago - and the scala plugin is from that time. The problem is with the spark build. I don't kwow if they have both 2.11.2 and 2.11.5 together or not. The tests do run from command line.