IDEA for Spark/Scala development

Hi everyne!

i plan to start development with Spark/Scala.
i installed all scala plugins and it all now perfectly works.

But i don't know how to start wotjing woth spark (how to import libraries or stuff like that...)

any advice are appresiated!

3 comments
Comment actions Permalink

Hi,

Use SBT for it. IntelliJ IDEA can open any SBT project.
You also can start project in IntelliJ IDEA from Activator template (new project -> Scala -> Activator -> choose Spark related template).

Best regards,
Alexander Podkhalyuzin.

0
Comment actions Permalink

This response does not go far enough.  First, on Spark of December 2016, maven rather than SBT is used.  On the standard download, neither maven nor sbt are even included anymore, so you have to install them separately. Next, the imports for sql and other components, and even the import for spark itself cannot be found at all by Intellij, even though the programs compile and run on the command line.  Please correct the spark documentation (there really is not much, even on the web (which is all very old), and help out the users.  Start a new clean install and see if JB can get scala to work in spark on a Mac, and then explain how the libraries can get loaded / imported, please.

 

0
Comment actions Permalink

Hi Alex,

 

Check this blog: http://blog.miz.space/tutorial/2016/08/30/how-to-integrate-spark-intellij-idea-and-scala-install-setup-ubuntu-windows-mac/ 

This blog post is a step-by-step tutorial. There you will find information on how to create Spark Scala project and import all necessary libraries.

Apart from that, you can just check out the template project from GitHub: https://github.com/mizvol/SparkTest 

 

Cheers,

Volodymyr

0

Please sign in to leave a comment.