When Erlang befriend Python
Python and Erlang are two programming languages, which I cannot get enough of. Having said that my first crush is Pascal and first love is C although that story is for some other time. It is interesting how they compliment each other in so many ways making this combination amazing for a developer.
I have written numerous tools (including a custom shell) in python and even went to the extent of writing a telecom service (although that was primarily used for internal consumption) in it along with deep integration with library in C++. The primary motivation for using python, was quick prototyping which undeniably makes it one of the best alternative in the present programming language landscape. There are so many freely available libraries and frameworks built on top of python and it is getting better each day.
Erlang on the other hand is the dark horse. It is a powerhouse of concurrency and one of the best alternative for building services which are in desperate need of soft real-time behaviour. Not many engineers realize this, but most system require responsiveness a lot more than raw latency (except for in cases where it does matter). Its amazingly intuitive (after a while looking at the syntax) semantics for building responsive, fault-tolerant, distributed systems is addictive. Read my story of how it helped redBus build a massively scalable fault-tolerant, soft real-time transaction platform.
Erlang Node in Python 3 — Pyrlang
Pyrlang is an awesome project which can be used to build Erlang node in Python 3.5+. This allows an easy integration of Erlang (and other BEAM programming languages) with Python 3. I am eagerly waiting for the Native C++ ETF implementation to come through which will make the information interchange between Erlang and Python optimal.
There are so many useful libraries like numpy, pandas, matplotlib and many others which are just waiting to be exploited from Erlang. How about marrying the concurrency of Erlang with compute of Python?
This is nice but then you do have to maintain a separate node for running python and if you are lazy like me then it is an unnecessary work. So, how about changing it? Say hello to BeamParticle, which brings in many more advantages other than just connecting both the programming languages.
Python within Erlang — BeamParticle
BeamParticle is an interesting project which brings Ideas to life very rapidly. This is a very rapid and dynamic development environment, which enables developers to break the programming language barriers and build stuff in 6 different programming languages. More details are in the blog.
BeamParticle — Making it easy to build complex systems without the complexities of making different programming languages work together (both for development and deployment).
It is very important that the interface between Python and Erlang be rock solid, which allows application development massively scalable and resilient. There are numerous ways to build services in Python and Erlang which work together; namely — Custom tcp, udp, via Queue, HTTP(s) REST, SOAP, gRPC, Protobuf, Thrift, and many more. Although, it is lucrative to pick any one there are number of important aspects which must not be ignored.
- Request / Response based transactions, which must support unique references and timeouts.
- Acknowledge that programmers will make mistakes and some of them may even write code which blocks for a very long period. Such cases must be handled and heals the system when the situation arise.
- Flexible architecture which supports many different nodes (whether Erlang or Python) to work together.
Both of the above requirements are met with BeamParticle (powered by Pyrlang) way of connecting the two. Additionally, the logic whether Erlang or Python is stored in a central database (in this case BeamParticle’s main storage), while the Python interpreters just caches the logic. This allows the system to scale massively with negligible boot-up time. The design decisions (more on that later) taken by the BeamParticle project ensures that all the requisite aspects (see bullet list a little above) are met. I have experimented with running hundreds of Python interpreters under BeamParticle on a quad-core Intel i5 machine which is able to run all of them easily with each interpreter consuming around 0.1–0.2% CPU when idle.
Lets quickly build an interesting application to extract text from website where Python has numerous alternatives, so lets see how we can use it within Erlang via BeamParticle.
The Erlang code which retrieves a given website and then invokes the above python function is as follows:
Note that the above Erlang function invokes “py_text_readability” which is written in Python as a regular function. The prefix “py_” has nothing to do with the programming language although the header “#!python” is important when writing function in Python programming language. BeamParticle inspects and understands cross functional call and appropriately invokes the python function on any of the available python interpreters and give back the result like Magic.
Final Notes
It is amazing how easy it is to develop and deploy multi-lingual systems with open source components, which opens up the possibilities of leveraging the best of many worlds (that is libraries in programming languages) to build awesome systems. This post is just an introduction into the subject and many of the details are intentionally left out to keep this story short.
References
- Neeraj Sharma, BeamParticle — A Polyglot Dynamic Programming Engine. [Online]. Available: https://medium.com/@nsharma9/beamparticle-a-polyglot-dynamic-programming-engine-1250109af1cd
- [Online]. Available: https://github.com/esl/Pyrlang
- [Online]. Available: https://github.com/neeraj9/SignalBlocks
- [Online]. Available: https://github.com/buriy/python-readability#python-readability
