How to setup Intellij 14 Scala Worksheet to run Spark
I'm trying to create a SparkContext in an Intellij 14 Scala Worksheet.
here are my dependencies
name := "LearnSpark"
version := "1.0"
scalaVersion := "2.11.7"
// for working with Spark API
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.0"
Here is the code i run in the worksheet
import org.apache.spark.{SparkContext, SparkConf}
val conf = new SparkConf().setMaster("local").setAppName("spark-play")
val sc = new SparkContext(conf)
error
15/08/24 14:01:59 ERROR SparkContext: Error initializing SparkContext.
java.lang.ClassNotFoundException: rg.apache.spark.rpc.akka.AkkaRpcEnvFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
When I run Spark as standalone app it works fine. For example
Can someone provide some guidance on where I should look to resolve this exception?
Thanks.
Please sign in to leave a comment.
This is a forum for discussing the development of IntelliJ IDEA plugins. Please ask your question in the Scala forum instead.
Thanks Dmitry,
I have posted question on Scala forum
Do i remove the question from here?