How to create maven project for Spark and Scala in Scala IDE

Spark is a very good framework to play with big data. You can quickly download and install spark on your computer or laptop and you can start coding in spark shell but it is a bit tedious to create project in Eclipse , IntelliJ IDEA or Scala IDE.

So, I am going to tell you that how to create your first maven project in Scala IDE where you can code in spark and scala. Please follow below steps to create your first project.

1. Download Scala IDE from link — http://scala-ide.org/download/sdk.html
2. Extract it and open Scala IDE like you open eclipse
3. Once the Scala IDE is opened then start creating first maven project. To create first project ,please follow below steps carefully.
4. Click on File -> New -> Project ->Maven -> Maven Project ->next
5. Check option — Create a simple project(skip archetype selection) and click on next
6. Provide group Id(ex-com.spark.scala) and artifact Id(ex- demo) and click on finish.
7. Expand your project(Demo) and open pom.xml and add dependency from link —( https://github.com/Manojd664/Spark-ML/blob/master/pom.xml ) after line — <version>0.0.1-SNAPSHOT</version> in pom.xml, now save the pom.xml
8. After saving pom.xml , It will build the project. After build completion, you will see some errors in problems panel. To remove these errors — Right click on errors -> click on Quick Fix -> click on finish
9. Now, Right click on project(Demo) -> Click on configure -> click on add scala nature
10. Again you will some list of errors in problems panel. To remove these erros — Right click on project(Demo) -> select “scala compiler” -> check “Use project settings” ->From scala installation ,select scala version -> click on apply and close
11. Right click on project(Demo) -> Build Path ->Configure Build Path ->From top of panel , select “libraries” ->Select “Scala library container” ->click on remove -> Apply and close
12. Expand project — Right click on src/main/java and rename it to src/main/scala
13. Right click on src/main/scala and create package with package name= group Id.artifact Id(ex-com.spark.scala.demo)
14. Right click on package(com.spark.scala.demo) and create a scala object and start with coding by creating spark session.
15. You can see code for your first project from link — https://github.com/Manojd664/Spark-Scala/blob/master/src/main/scala/com/spark/scala/learnscala/wordcount/WordCount.scala

I hope this will help you. Please feel free to comment suggestion to improve it.