<![CDATA[Stories by Volodymyr Miz on Medium]]> https://medium.com/@mizvladimir?source=rss-3502d10e47ee------2 https://cdn-images-1.medium.com/fit/c/150/150/1*npi3ybhaFRNWi49uirsAwQ.jpeg Stories by Volodymyr Miz on Medium https://medium.com/@mizvladimir?source=rss-3502d10e47ee------2 Medium Thu, 28 May 2026 03:13:43 GMT <![CDATA[How to integrate Apache Spark, Intellij Idea and Scala]]> https://medium.com/@mizvladimir/how-to-integrate-apache-spark-intellij-idea-and-scala-bf5edef8c4c?source=rss-3502d10e47ee------2 https://medium.com/p/bf5edef8c4c Tue, 30 Aug 2016 00:00:00 GMT 2017-04-21T10:00:09.273Z Jupyter Notebook is really useful when you want to present some code, let someone reproduce your results or just learn how to use new tools and libraries. I use Jupyter almost every day and, as many others, I started learning Spark and developed my first data analysis pipelines using interactive notebooks and Python API. Then I realized that I want more and running notebooks locally is not enough for me, so I signed up for Databricks Community Ediditon subscription. Databricks allows to forget about the problems related to setting up and maintaining the environment.

Everyone who is learning and using Spark eventually realize that Python API is not as powerful and flexible as the core language of the framework — Scala. This language allows to start feeling the full power of Spark comprising Analytics, Streaming and Graph processing tools. However, Spark is just yet another framework for large scale data analytics. Yes, it is convenient and powerful, but it has a limited number of algorithms and sometimes you need to implement your own custom algorithm. And that is the moment when you need an IDE.

You can find an example project in my Git repository. It allows you to get started with Spark Scala development in Intellij Idea. Alternatively, you can follow step-by-step instructions on my blog and create this project from scratch by yourself.

]]>