I can create a spark scala app, but now how do I run it?

I am trying to build a spark scala application with IntelliJ. I installed the scala plugin, and made a new project going to the Activation option and chose the Hello Spark App. It downloaded lots of stuff. I can build, but when I run, it says it can't find the jar file. I am new to building scala - I know you use the sbt with a project/src/main/scala directory structure (i see that in this starter project) and a build file - I see that also.

Basically, I am a python programmer, and I would like to compare scala spark to python - but I don't know anything about building scala spark apps. Ideally, I'd like to produce these small test scala/spark jar files - and set up a remote run configuration on the aws emr cluster that we keep running so I could run through ssh.

 

0
2 comments

Most likely there is an issue in the template configuration. Have you tried a workaround described at https://github.com/jaceklaskowski/spark-activator/issues/4 ?

0

My problem might have been more basic - I did not have sbt installed on my macbook - I have installed it and built a spark scala package outside of IntelliJ from the command line. I still got weird errors with that hello spark program in intellij -- anyways, I think this is good enough for me right now

0

Please sign in to leave a comment.