BeamParticle — A Polyglot Dynamic Programming Engine
BeamParticle is an open source project built on top of the Erlang virtual machine (BEAM), which allows dynamic (re)programming in multiple programming languages. This project tries to take some simple decisions thereby making the life of developer easy in realizing dynamic code patching and reprogrammability. It is very easy to setup this software on any of the modern GNU/Linux distributions, although Debian packages are available for Ubuntu Xenial. The project supports six different programming languages; namely: Erlang, Elixir, Java, Python, Efene, and PHP.
Note: BEAM stands for Bogdan/Björn’s Erlang Abstract Machine.
The Motivation
There are numerous reasons why a (re)programmable system is the need of the hour. I agree that the thought of instantaneous code to production gives jitters when regular software process itself is sometimes insufficient in avoiding issues (bugs).
How about building read APIs dynamically? There is no mutation possible. How about building simplistic APIs over complex data stored in stores like Cassandra, MongoDB, MySQL or other HTTP APIs. We don’t have to be correct the first time when all it does is read. Although, try not to DDOS other services with massive parallelism which this can provide (say thanks to Erlang BEAM).
“With great power comes even greater responsibility.”
Irrespective of how you would look at this project, be aware that ultimately it is just a software. Additionally, it is too early to predict what turn this project might take in the coming future. That is what makes the journey exciting!
Key Features
You write functions, which are automatically available as HTTP POST API (as /post/function_name), which takes JSON input and must give back JSON response.
Although, the above is largely true the system is capable of much more while the internal details are intentionally left out (for now) to keep things simple. Some of the key features of BeamParticle is as follows:
- Supports 6 programming languages: Erlang, Elixir, Java-8, Python-3, Efene and PHP-5(subset).
- Can run as a cluster (via Erlang distribution protocol)
- Functions are dynamic and saving them instantly deploys them as HTTP POST API
- Functions written in Erlang, Elixir, Efene and PHP can call other functions natively (with type conversions).
- System can create Jobs (or crons) which can be persisted to disk as well, so that they exist even after system restart.
- System supports creation of pools, which are very useful for maintaining persistent connections or for any other use cases.
- Easily compile and deploy logic of code (functions) (across cluster when one exists) and archives them into a local Key-Value store (powered by Google LevelDB).
- HTTP(s) support with both HTTP 1.1 and HTTP 2.0 protocol.
- Alternative custom (minimal HTTP 1.1 with optional keep-alive) interface provided for /post/function_name interface for very low latency calls (sort of a fast-path) matching performance guarantees of some of the fast web servers .
The dynamic functions will be available as HTTP POST API as shown below. There are two different ports where the API would be available, wherein one is a complete HTTP REST implementation while the other is a minimal (custom) very fast version of the same. The regular interface (HTTPS on port 8282) would be substantially slower though complete compared to the fast version (HTTP on port 8484).
The curl commands highlighted above are now shown with the popular GUI based test tool Postman below. The first example uses the regular https interface.
The alternative port (8484) where the custom HTTP/1.1 server is running can also be used (without ssl and authentication) as shown below.
Now that we know what will be the end result, lets get on with installation and other details.
Build and Installation
The installation steps sufficiently covered in the project website (http://beamparticle.org/about.html), but lets just cover a little bit before we dive in. Please note that it is assumed that you have a sufficiently modern Debian (or Ubuntu) system to play with.
sudo apt-get install -y wget build-essentials libsctp1 libwxgtk3.0 debhelper libcap2-bin git
It is a good idea to install python3 and openjdk8 as well, since you can write python and java dynamic functions as well.
sudo apt-get install -y python3 openjdk-8-jdk
Download and install latest Erlang (https://www.erlang-solutions.com/resources/download.html)
wget https://packages.erlang-solutions.com/erlang/esl-erlang/FLAVOUR_1_general/esl-erlang_20.1-1~ubuntu~xenial_amd64.deb
sudo dpkg -i esl-erlang_20.1–1~ubuntu~xenial_amd64.deb
Clone (download) and build BeamParticle
git clone https://github.com/beamparticle/beamparticle
cd beamparticle
make deb-package
You will now have a debian package in packaging/debian/beamparticle-0.1.2_x86_64.deb, which can be installed as follows:
sudo dpkg -i packaging/debian/beamparticle-0.1.2_x86_64.deb
If you want to run python and java examples then you need to enable pynode and javanode in /opt/beamparticle/releases/0.1.2/sys.config (see below). Notice that {enable, true} is set for both python and java.
{pynode, [
{enable, true},
{num_workers, 1},…
{javanode, [
{enable, true},
{num_workers, 1},…
It is now time to run the software which has a systemd service as follows:
sudo service beamparticle start
Beam Me Up Scotty
You can now open your web browser and point to http://localhost:8282 and use username as “root” and password as “root” (without the double quotes). The authentication is very very* basic and password is as plain-text, but then there is a lot to be worked out before security is a concern. It was very convenient to add a dialog box for username and password so it was done as such, but it needs to change; see github-issue-6.
The WebSocket interface is provided to train the system, while Google LevelDB stores the dynamic functions. The function compilation happens for the first time, while subsequent invocation uses the cached version to gain performance.
At present there is no restriction on function execution, so anyone with access can efficiently run any function available within this framework. Having said that there is HTTP Basic authentication both at WebSocket and HTTP(S) REST interface for simple checks. However, once authenticated all the functions are available for execution.
The site would simply have (1) text bar, (2) “send” button and (3) “+” button. The text bar is used for specifying commands (we will come to that a little later), while the “send” button is used to send those to the backend server. The purpose of “+” button (light blue just beside the “send” button) is to toggle display of an inline source code editor.
Some Help Please
All the commands start with a dot (“.”) at the beginning and a general description of the available commands is displayed with “.help”. Note that the interface is javascript and html, so javascript is required for this to work in your browser. Additionally, any request and its response is displayed below the command bar in reverse order (latest on the top).
Function names have two parts; namely (1) Name and (2) Arity (number of arguments to that function). You must refer to a function by combining its name with the arity (separated by “/”). The system will have no memory (that is no functions) for the first time, so lets import them from an open source repository via the “.atomics fun restore” command (see below).
Import BeamParticle Atomics
The following step of importing beamparticle-atomics is OPTIONAL and can be skipped as well.
Due to a known bug in BeamParticle v0.1.2 we have to import atomics v0.1.0.
Commands: List
Now using the “.ls” command shall display the imported functions.
Commands: Open
In chrome browser you can just click on any of the functions listed, while for the rest use the “.open” command to view the contents of weather_for_city/2 function as shown below.
Commands: Save
Saving a dynamic function (or API) is as simple as using the “.save” command followed by just the name of the function.
Note: Do not include the function arity while saving.
Syntax Highlight and Other IDE Help
You can use the drop down to choose the programming language, which at present is just for syntax highlighting, code folding, etc. Choosing a language in the drop down have no impact on how it will be executed (we will come to that later).
APIs in Java 8
The system rely on user to specify the programming language, hence a need for having a header “#!java” at the top (as shown below) for writing dynamic logic in Java 8 programming language. Notice that the header is immediately followed by comments (in case of java only “//” must be used). These shall be used when invoking the “.help” command on a particular function. Additionally, keep the import at the top because at present BeamParticle has a very basic Java parser to extract import statements at the top which start without a space and terminates with “;” (semicolon).
There is a strict requirement before any function can be used for receiving HTTP POST API. In case of Java 8 it must have a “main” and “handleEvent” methods with a specific signatures (see the code snippet below). Notice that the “main” method invokes “handleEvent” internally, hence the user is expected to modify “handleEvent” alone and keep “main” unmodified. The “postBody” contains json, which must be deserialized, while the response must be a serialized json. There is a mechanism wherein the user can install additional jars, but it is not seamless as for Python at present (more on that in subsequent posts).
APIs in Python 3
The same principles apply to Python (must be Python 3 code) as was for Java. The user is expected to modify “handle_event” function to do some useful processing of the incoming request. Notice the “#!python” header, which determines that this source code is Python. BeamParticle uses the system default python 3 interpreter, so any python package installed can be directly imported here. This is very cool, because using new packages is as simple as installing on the machine running BeamParticle.
APIs in Erlang (OTP 20+)
The Erlang version is a bit different because it uses native data types where no conversion is required (unlike Java and Python). The “#!erlang” header is optional (only for Erlang), but it is a good practice to use it for consistency. You would notice that some of the imported functions (earlier) do not have this header, where Erlang is assumed. Unlike Python there is no easy way to install additional Erlang libraries. If you have detailed understanding of Erlang build system then you could hack around and make it work but for the larger audience this feature isn’t there. At present BeamParticle come equipped with a lot of dependencies; namely clients for redis, mongodb, cassandra, mysql, postgresql (to name a few, while the complete list is here).
APIs in Elixir 5.x
The Elixir version is similar to Erlang, wherein the anonymous function takes two arguments and follows the same principles. The header in this case is “#!elixir”.
APIs in Efene
Efene is an interesting programming language and worth checking out. Efene runs directly on Erlang BEAM, so supporting that was not much of a challenge. The language is very small and requires a lot less typing when compared to the other programming languages mentioned above. Notice the header “#!efene”, which must be used for Efene programming language.
Final Notes
This project is a humble attempt to explore the idea of allowing code to production rapidly. There are a lot of challenges, which makes it fun to explore and try to push the boundaries.
I have left out a lot of the details to keep this post simple and an allow me more time in other posts to justify the content. It will be great to see some feedback / comments or even better a pull request / issues on the beamparticle github project.
Note: The system is use in limited capacity within redBus production infrastructure, but for services which are largely support flow and have alternate fallback in place. Having said that the system has not gone down since its inception, which is around 2 months since the time of this writing.
References
[1] http://www.beamparticle.org/
[2] https://github.com/beamparticle/beamparticle
[5] https://docs.oracle.com/javase/8/docs/api/java/lang/package-summary.html
[8] http://php.net/
