Running Scala tests in IntelliJ throws "class not found" errors - works with "sbt test"

I am trying to use IntelliJ as an IDE for developing Scala apps to run under Apache Spark.  I set up a new SBT-based Scala project in the latest version of IntelliJ with two simple tests.

When I run these tests with "sbt test" they both pass.  When I run them within IntelliJ the second test fails, complaining that it can't find the class "HttpServletResponse" which is, as far as I can tell, pulled in by spark.

Tests:

 
import org.apache.spark.{SparkContext, SparkConf}
import org.scalatest.FunSuite

class ExampleTests extends FunSuite {

  test("works, obviously!") {
    assert(1 == 1)
  }

  test("broken") {
    val conf = new SparkConf()
      .setAppName("broken test")
      .setMaster("local")
      .set("spark.default.parallelism", "1")

    val sc = new SparkContext(conf)
    try{
      assert(sc.parallelize("The complete works of Shakespeare".split("\\W")).map(_ + 1).count == 5)
    }
    finally {
      sc.stop()
    }
  }
}


SBT:

 
name := "NeighbourRelations"

version
:= "1.0"

scalaVersion
:= "2.10.4"

libraryDependencies
+= "org.apache.spark" %% "spark-core" % "1.2.0"

libraryDependencies
+= "org.scalatest" % "scalatest_2.10" % "2.1.3" % "test"


Running tests with SBT on the command line:

$ sbt test

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 [info] Loading global plugins from C:\Users\tay59806\.sbt\0.13\plugins [info] Loading project definition from D:\Dropbox\Progs\Spark\NeighbourRelations\project [info] Set current project to NeighbourRelations (in build file:/D:/Dropbox/Progs/Spark/NeighbourRelations/) [info] Compiling 1 Scala source to D:\Dropbox\Progs\Spark\NeighbourRelations\target\scala-2.10\test-classes... [info] NeighbourRelationshipUsageTest: [info] FunSuiteForSpark: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

...

[info] Run completed in 3 seconds, 544 milliseconds. [info] Total number of tests run: 2 [info] Suites: completed 3, aborted 0 [info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 6 s, completed 29-Dec-2014 12:17:02


Running tests in IntelliJ - second test fails:

Testing started at 12:12 ... 14/12/29 12:12:51 INFO spark.SecurityManager: Changing view acls to: tay59806,
... A needed class was not found. This could be due to an error in your runpath. Missing class: javax/servlet/http/HttpServletResponse java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse      at org.apache.spark.HttpServer.org$apache$spark$HttpServer$$doStart(HttpServer.scala:73)      at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)      at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)      at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446)      at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)      at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442)      at org.apache.spark.HttpServer.start(HttpServer.scala:60)      at org.apache.spark.HttpFileServer.initialize(HttpFileServer.scala:45)      at org.apache.spark.SparkEnv$.create(SparkEnv.scala:243)      at org.apache.spark.SparkContext.<init>(SparkContext.scala:203)      at ExampleTests$$anonfun$2.apply$mcV$sp(ExampleTests.scala:16)      at ExampleTests$$anonfun$2.apply(ExampleTests.scala:10)      at ExampleTests$$anonfun$2.apply(ExampleTests.scala:10)      at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)      at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)      at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)      at org.scalatest.Transformer.apply(Transformer.scala:22)      at org.scalatest.Transformer.apply(Transformer.scala:20)      at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:158)      at org.scalatest.Suite$class.withFixture(Suite.scala:1121)      at org.scalatest.FunSuite.withFixture(FunSuite.scala:1559)      at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:155)      at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:167)      at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:167)      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)      at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:167)      at org.scalatest.FunSuite.runTest(FunSuite.scala:1559)      at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:200)      at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:200)      at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)      at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)      at scala.collection.immutable.List.foreach(List.scala:318)      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)      at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)      at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:200)      at org.scalatest.FunSuite.runTests(FunSuite.scala:1559)      at org.scalatest.Suite$class.run(Suite.scala:1423)      at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1559)      at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:204)      at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:204)      at org.scalatest.SuperEngine.runImpl(Engine.scala:545)      at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:204)      at org.scalatest.FunSuite.run(FunSuite.scala:1559)      at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)      at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)      at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)      at scala.collection.immutable.List.foreach(List.scala:318)      at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)      at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)      at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)      at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)      at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)      at org.scalatest.tools.Runner$.run(Runner.scala:883)      at org.scalatest.tools.Runner.run(Runner.scala)      at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:137)      at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)      at java.lang.reflect.Method.invoke(Method.java:483)      at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: java.lang.ClassNotFoundException: javax.servlet.http.HttpServletResponse      at java.net.URLClassLoader$1.run(URLClassLoader.java:372)      at java.net.URLClassLoader$1.run(URLClassLoader.java:361)      at java.security.AccessController.doPrivileged(Native Method)      at java.net.URLClassLoader.findClass(URLClassLoader.java:360)      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)      ... 63 more


I can only assume that for some reason, IntelliJ is building things slightly differently to SBT.  This means it can't find the appropriate libraries.  I am new to IntelliJ and Scala, so all help appreciated - however simple!!

Thanks!

0
6 comments

There seems to be some differences involving org.eclipse.jetty


sbt classpath using -  show test:full-classpath

.ivy2\cache\org.eclipse.jetty\jetty-plus\jars\jetty-plus-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.transaction\orbits\javax.transaction-1.1.1.v201105210645.jar)
.ivy2\cache\org.eclipse.jetty\jetty-webapp\jars\jetty-webapp-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-xml\jars\jetty-xml-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-util\jars\jetty-util-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-servlet\jars\jetty-servlet-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-security\jars\jetty-security-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-server\jars\jetty-server-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar)
.ivy2\cache\org.eclipse.jetty\jetty-continuation\jars\jetty-continuation-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-http\jars\jetty-http-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-io\jars\jetty-io-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty\jetty-jndi\jars\jetty-jndi-8.1.14.v20131031.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.mail.glassfish\orbits\javax.mail.glassfish-1.4.1.v201005082020.jar)
.ivy2\cache\org.eclipse.jetty.orbit\javax.activation\orbits\javax.activation-1.1.0.v201105071233.jar)


Intellij passed classpath to runner ( look at the command line in the test runner folded at the top )
.ivy2\cache\org.eclipse.jetty\jetty-continuation\jars\jetty-continuation-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-http\jars\jetty-http-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-io\jars\jetty-io-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-jndi\jars\jetty-jndi-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-plus\jars\jetty-plus-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-security\jars\jetty-security-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-server\jars\jetty-server-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-servlet\jars\jetty-servlet-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-util\jars\jetty-util-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-webapp\jars\jetty-webapp-8.1.14.v20131031.jar
.ivy2\cache\org.eclipse.jetty\jetty-xml\jars\jetty-xml-8.1.14.v20131031.jar


javax.servlet-3.0.0.v201112011016.jar is the one with that missing class I think but it seems to be a bit fubar. Pretty new to this stuff and there is a lot of noise in the classpath to sift through.

Might be worth you creating an issue in the tracker with full info copied into files then uploaded for both methods. More likely to get attention as there is possibly a greater underlining issue with project analysis)

---
Just confirmed manually adding  .ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar to the dependencies under the module with all the others makes it work.

HttpServletResponse is findable by the ide on navigate to the class  (ctrl-n or whatever) when set up right.

0
Avatar
Permanently deleted user


Thanks very much for the response.  I can confirm that adding the javax.servlet dependency does indeed fix the problem... though having to do it seems wrong somehow.

 
name := "NeighbourRelations"

version
:= "1.0"

scalaVersion
:= "2.10.4"

libraryDependencies
+= "org.apache.spark" %% "spark-core" % "1.2.0"

libraryDependencies
+= "javax.servlet" % "javax.servlet-api" % "3.1.0" % "test"

libraryDependencies
+= "org.scalatest" % "scalatest_2.10" % "2.1.3" % "test"


Having got that working, I have now run in to a very similar error.  This time I aam using cucumber to do some BDD.  My project is much more complex so it's harder to paste it here.  However, just like last time, I have tests which pass when I run "sbt test" but fail when I run them within IntelliJ.  Essentially I am running the same test as above but wrapped in BDD test cases.

Running in IntelliJ

java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;      at akka.actor.ActorCell$.<init>(ActorCell.scala:336)      at akka.actor.ActorCell$.<clinit>(ActorCell.scala)      at akka.actor.RootActorPath.$div(ActorPath.scala:159)      at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:464)      at akka.remote.RemoteActorRefProvider.<init>(RemoteActorRefProvider.scala:124)      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)      at java.lang.reflect.Constructor.newInstance(Constructor.java:408)      at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)      at scala.util.Try$.apply(Try.scala:191)      at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)      at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)      at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)      at scala.util.Success.flatMap(Try.scala:230)      at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)      at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:584)      at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:577)      at akka.actor.ActorSystem$.apply(ActorSystem.scala:141)      at akka.actor.ActorSystem$.apply(ActorSystem.scala:118)      at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121)      at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54)      at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)      at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1676)      at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)      at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1667)      at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56)      at org.apache.spark.SparkEnv$.create(SparkEnv.scala:206)      at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)      at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)      at stepdefs.CheeseBurger$$anonfun$7.apply(CheeseBurger.scala:45)      at stepdefs.CheeseBurger$$anonfun$7.apply(CheeseBurger.scala:39)      at cucumber.api.scala.ScalaDsl$StepBody$$anonfun$apply$2.applyOrElse(ScalaDsl.scala:97)      at cucumber.api.scala.ScalaDsl$StepBody$$anonfun$apply$2.applyOrElse(ScalaDsl.scala:95)      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)      at cucumber.runtime.scala.ScalaStepDefinition.execute(ScalaStepDefinition.scala:71)      at cucumber.runtime.StepDefinitionMatch.runStep(StepDefinitionMatch.java:37)      at cucumber.runtime.Runtime.runStep(Runtime.java:298)      at cucumber.runtime.model.StepContainer.runStep(StepContainer.java:44)      at cucumber.runtime.model.StepContainer.runSteps(StepContainer.java:39)      at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:48)      at cucumber.runtime.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:91)      at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:63)      at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:18)      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)      at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:70)      at cucumber.api.junit.Cucumber.runChild(Cucumber.java:89)      at cucumber.api.junit.Cucumber.runChild(Cucumber.java:40)      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)      at cucumber.api.junit.Cucumber.run(Cucumber.java:94)      at org.junit.runner.JUnitCore.run(JUnitCore.java:160)      at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)      at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)      at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)      at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)      at ✽.When I count the words in "The Complete Works of Shakespeare"(myfirst.feature:10)


Running with "sbt test"

D:\Dropbox\Progs\Spark\BeeDeeDee>sbt test Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 [info] Loading global plugins from C:\Users\tay59806\.sbt\0.13\plugins [info] Loading project definition from D:\Dropbox\Progs\Spark\BeeDeeDee\project [info] Set current project to BeeDeeDee (in build file:/D:/Dropbox/Progs/Spark/BeeDeeDee/) Feature: Trying to make the cucumber plugin work in IntelliJ   Scenario: Spark should be awesome                               # myfirst.feature:9     When I count the words in "The Complete Works of Shakespeare" # CheeseBurger.scala:39     Then there are '5' words                                      # CheeseBurger.scala:5415/01/09 09:31:40 INFO RemoteActorRefP rovider$RemotingTerminator: Remoting shut down. 1 Scenarios (1 passed) 2 Steps (2 passed) 0m3.205s [info] ScalaTest [info] Run completed in 3 seconds, 885 milliseconds. [info] Total number of tests run: 0 [info] Suites: completed 0, aborted 0 [info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0 [info] No tests were executed. [info] Passed: Total 3, Failed 0, Errors 0, Passed 3 [success] Total time: 4 s, completed 09-Jan-2015 09:31:41


Again this seems to be some sort of library incompatibility/version error.  IntelliJ is clearly not building the same thing in the same way as sbt!

Before I submit an issue, which I will do, do people think this is another example of the same error above?

Thanks,

Dan

0
Avatar
Permanently deleted user

Hi! Thanks for the report.

I've investigated a bit further and find out that the source of this problem may be an incorrect resolution of dependencies when "download sources" is enabled. As a workaroud I suggest that you try to reimport project with "download sources" feature disabled. If you haven't used this option in a first place or if it's not helping please provide me with details on Youtrack.

0

I have 14.1.4 Ultimate and the problem still exists.  Is there a YouTrack for this "Download Sources" issue?

How do we disable that option in an existing project?  I can not find any setting in the Project Settings dialogs.

0
Avatar
Permanently deleted user

What version of Scala plugin do you use? This particular issue was fixed in 1.5.2. CNF could be caused as well by this issue which is not fixed yet.

0

I installed latest 14.1.4 maybe two weeks ago - and the scala plugin is from that time.  The problem is with the spark build. I don't kwow if they have both 2.11.2 and 2.11.5 together or not.  The tests do run from command line.

0

Please sign in to leave a comment.