Connecting Python Bolt with Apache Storm Topology

Apache Storm is an open source distributed real-time computation system. I have used Strom for one of my projects to analyze real time data feed. During the developing period, I wanted to connect some machine learning code part written in python to the storm topology. With the Apache Storm Multi-Language Protocol, Storm can work with Ruby, Python, JavaScript and Perl. In following steps I will show you how to connect a simple python bolt to a storm topology.

Step 1

First import the python dependencies for the Storm project. If you have used maven it would be very easy as follows.

<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>multilang-python</artifactId>
<version>1.1.0</version>
</dependency>

Make sure it is same as the your storm-core version. Otherwise, it can cause errors.

Step 2

Download storm.py file from https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py

(If you want to connect Ruby, JavaScript or Perl Bolt, then use these files instead of storm.py - Ruby file, JavaScript file, Perl file)

Step 3

Write your python file as follows. For the following “import” command you should use the downloaded file in step 2.

import storm

class SplitBoltPython(storm.BasicBolt):

def initialize(self, conf, context):
self._conf = conf;
self._context = context;

def process(self, tuple):
word = tuple.values[0]
# do your processing here
storm.emit([word]) # return list object

SplitBoltPython().run()

save your python file and use the file location in step 3.

Step 4

extends your Bolt with “ShellBolt” and implements with “IRichBolt”. Then In the constructor of that class, call your python file as follows.

public class SplitBolt extends ShellBolt implements IRichBolt {    public SplitBolt(){
super("python", "{absolute path}/splitBoltPython.py");
}

Step 5

Create a “SplitBolt”object as follows.

SplitBolt splitBolt = new SplitBolt();
Map env = new HashMap();
env.put("PYTHONPATH", "/{absolute path}/");
SplitBolt.setEnv(env);

Then connect this bolt with the topology as a normal Bolt.

That’s all. With these steps you can simply connect your python code with your storm topology.……

)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade