Nicolas A Perez
1 min readAug 25, 2016

--

Well, now we have completely new problem to solve. The thing is that when you submit apps to Spark, you need to include your app’s dependencies. Look at the Spark docs for more info: http://spark.apache.org/docs/latest/submitting-applications.html

The way around this is that you instead of having a .jar with your application and separated dependencies, you create a “fat” .jar that includes everything. In order to do this, I normally use SBT Native Packager. Please, read its documentation for more info on how to install it. Once you have it ready, then you need to package your application using this plugin instead of using sbt package. The .jar that is generated from this process has all the dependencies you need so then you are ready to submit it to Spark using the spark-submit command. In this case, you don’t need to specify external dependencies since them are already into the .jar you have previously created.

Hope this works for you, it has for me for long time now.

--

--