Start/Deploy Apache Spark application programmatically using spark launcher

Sometimes we need to start our spark application from the another scala/java application. So we can use SparkLauncher. we have an example in which we make spark application and run it with another…

Easiest Way To Insert Scala Collection into PostgreSQL using Slick

Few days ago, I had a scenario, in which I had to insert scala collection into postgreSQL using Slick. My postgreSQL table has some columns with data types such as Arrays, hstore etc..
I tried to do this using…

Headless Browser testing in Scala using Selenium Webdriver

Headless browser term is used which have no GUI(graphical user interface). Headless provide the same environment like a web browser to a web page for the testing.These are the program which are run on our machine without showing…

Demystifying Asynchronous Actions in Spark

What if we want to execute 2 actions concurrently on different RDD’s, Spark actions are always synchronous. Like if we perform two actions one after other they always execute in sequentially like one after other.

How to Implement Cross Cutting Concern using AOP in Scala

In this blog, I have explained that how easily we can implement Cross Cutting Concern such as logging using Aspect Oriented Programming in Scala. These concerns have nothing to do with core business logic. Aspect Oriented…

Shufflling and repartitioning of RDD’s in apache spark

To write the optimize spark application you should carefully use transformation and actions, if you use wrong transformation and action will make your application slow. So when you are writing application some points should be remember…

Setup a Apache Spark cluster in your single standalone machine

If we want to make a cluster in standalone machine we need to setup some configuration.

We will be using the launch scripts that are provided by Spark, but first of all there are a couple of…