Connecting Python Bolt with Apache Storm Topology

Apache Storm is an open source distributed real-time computation system. I have used Strom for one of my projects to analyze real time data feed. During the developing period, I wanted to connect some machine learning code part written in python to the storm topology. With the Apache Storm Multi-Language Protocol, Storm can work with Ruby, Python, JavaScript and Perl. In following steps I will show you how to connect a simple python bolt to a storm topology.
Step 1
First import the python dependencies for the Storm project. If you have used maven it would be very easy as follows.
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>multilang-python</artifactId>
<version>1.1.0</version>
</dependency>
Make sure it is same as the your storm-core version. Otherwise, it can cause errors.
Step 2
Download storm.py file from https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py
(If you want to connect Ruby, JavaScript or Perl Bolt, then use these files instead of storm.py - Ruby file, JavaScript file, Perl file)
Step 3
Write your python file as follows. For the following “import” command you should use the downloaded file in step 2.
import storm
class SplitBoltPython(storm.BasicBolt):
def initialize(self, conf, context):
self._conf = conf;
self._context = context;
def process(self, tuple):
word = tuple.values[0]
# do your processing here
storm.emit([word]) # return list object
SplitBoltPython().run()save your python file and use the file location in step 3.
Step 4
extends your Bolt with “ShellBolt” and implements with “IRichBolt”. Then In the constructor of that class, call your python file as follows.
public class SplitBolt extends ShellBolt implements IRichBolt { public SplitBolt(){
super("python", "{absolute path}/splitBoltPython.py");
}
Step 5
Create a “SplitBolt”object as follows.
SplitBolt splitBolt = new SplitBolt();
Map env = new HashMap();
env.put("PYTHONPATH", "/{absolute path}/");
SplitBolt.setEnv(env);Then connect this bolt with the topology as a normal Bolt.
That’s all. With these steps you can simply connect your python code with your storm topology.……
