Day 6: Realtime Tweets Analysis using Spark Streaming with Scala

Harini Janakiraman
May 13, 2016 · 2 min read


Project (1 Hour): Create a twitter app and use its API to Stream realtime twitter feed using Spark Streaming with scala.

All the code for this project can be found on my github

Step 1: Download and Setup Spark and Scala IDE

Ensure you have JDK already setup, verify it using the below command, if not go ahead download and setup your JAVA_HOME environment variable.

$ java -version
java version “1.8.0_91”
$ echo JAVA_HOME

Download Spark from:

Run a test scala code from the downloaded directory using Spark Shell.

scala> sc.parallelize(1 to 1000).count()
res1: Long = 1000

You can also test using the example python code.

./bin/run-example SparkPi

Finally install scala IDE built on top of eclipse from:

Step 2: Create the project with Twitter App credentials

Create a twitter app using and then fill in the following in a text file.


Setup a scala project in IDE and create the following scala code that prints out live tweets as they stream using Spark Streaming.

Building and running the above should continuous stream tweets to your console. English doesn’t seem to be the popular language at this hour!

Image for post
Image for post

Day 6 of #100DaysOfCode DONE

If you enjoyed this, please click 👏 so that others can enjoy it as well. Follow me on Twitter @HariniLabs to get the latest updates or just to say Hi :)

PS: I curate a bi-weekly #WomenInTech newsletter for a dose of inspiration from the world of tech and yes men can signup too! Get it here :)

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store