Mixing Python with Elixir II

Asynchronous communication

Important: Read Part I of this post before continuing.

In Part I, we looked at how to call Python functions from Elixir using Erlport!

ErlPort is an Elixir library which makes it easier to connect Erlang to a number of other programming languages using the Erlang port protocol.

However, as you may have noticed, the calling process had to wait for Python to finish processing and return the results. This works for operations that are instant and doesn’t take much time. Otherwise, it’s not much fun when you are running an intensive Python function. Because your Elixir program is gonna hang waiting for the Python code to finish.

For operations that take a while to complete we need a way to call the Python function and get notified in our elixir code when the function is complete — similar to how we write elixir program with processes talking to each asynchronously. This is useful, especially in cases when you want an Elixir process to control/supervise several Python operations. We can run the Python functions concurrently which makes life a little better.

In this post, we’ll explore how to achieve this — Asynchronous communication between Python and Elixir!

We can reason about the challenge the same way we do in communicating between Elixir processes — by using cast — which allows us to communicate between processes asynchronously.

Previously, we used erlport’s :python.call/4 mostly when calling functions from Python. This is synchronous and the elixir program had to wait for the python function to finish and send response.

In this post, we will use :python.cast/2

Understanding :python.cast/2

:python.cast/2 works the same as GenServer.cast/2. You give it process id (or registered process name) and the message to send to that process. Unlike :python.call variant, :python.cast is used to send message instead of calling a Python function. Same way your Elixir processes are sent messages. Erlport provides a mechanism for us to receive messages sent to the python instance.

Erlport Python Module

Erlport offers a python library that helps with working with elixir data types and processes. You can install erlport using pip.

pip install erlport #don't run this yet!

However, the Erlport Elixir repo already comes with the Python library which are automatically loaded at runtime. So you don’t have to install the Erlport Python library if you are using Erlport through Elixir.

Below is how data is mapped between Erlang/Elixir and Python

Erlang/Elixir data types mapped to Python data types (http://erlport.org/docs/python.html)

Python data types mapped to Erlang/Elixir data types (http://erlport.org/docs/python.html)

Handling Message Sent from Elixir to Python

For Elixir processes you have a receive block in which all messages are handled. Or the handle_call, handle_cast and handle_info functions if you are using GenServer. Erlport provides a similar mechanism for handling messages sent to the Python process. set_message_handler is the function. It’s in the erlport.erlang module. It takes a Python function with one argument-message i.e the incoming message. Erlport will then call the function passed to set_message_handler whenever a message is received. It’s like a callback function.

Sending Message from Python to Elixir

Since an Elixir process isn’t aware of who sent a cast message, we have to find a way to know which process to send the result to once the python function completes. One way is to register the process id, of the Elixir process that you want the result sent to.

The Erlport python module provides cast function for sending a message to Elixir process. With this we can send asynchronously message back to the elixir process when the function is done!

Putting it all together

Open Terminal and create a new Elixir project using mix

$ mix new elixir_python

Add dependency

Add erlport to your dependencies mix.exs

defp deps do
    [     
      {:erlport, "~> 0.9"},
    ]
  end

Then install project dependencies.

$ cd elixir_python
$ mix deps.get

Create priv/python directory where you’ll keep our python modules

$ mkdir -p priv/python

Create an elixir module wrapper for Erlport related functions.

#lib/python.ex
defmodule ElixirPython.Python do
    @doc """
      Start python instance with custom modules dir priv/python 
     """
    def start() do 
        path = [
             :code.priv_dir(:elixir_python), 
             "python"
        ]|> Path.join()
        
        {:ok, pid} = :python.start([
           {:python_path, to_charlist(path)}
        ])
        pid
    end
    def call(pid, m, f, a \\ []) do 
       :python.call(pid, m, f, a)
    end
    
    def cast(pid, message) do 
       :python.cast(pid, message)
    end
   
    def stop(pid) do 
       :python.stop(pid)
    end
end

In order for Elixir process to receive message asynchronously, we will create a simple GenServer module

#lib/python_server.ex
defmodule ElixirPython.PythonServer do 
   use GenServer
   alias ElixirPython.Python
   def start_link() do 
      GenServer.start_link(__MODULE__, [])
   end
   def init(args) do
      #start the python session and keep pid in state
      python_session = Python.start()
      #register this process as the message handler
      Python.call(python_session, :test, :register_handler, [self()])
      {:ok, python_session}
   end
   def cast_count(count) do 
      {:ok, pid} = start_link()
      GenServer.cast(pid, {:count, count})
   end

   def call_count(count) do
      {:ok, pid} = start_link()
      # :infinity timeout only for demo purposes
      GenServer.call(pid, {:count, count}, :infinity)
   end

   def handle_call({:count, count}, from, session) do 
      result = Python.call(session, :test, :long_counter, [count])
      {:reply, result, session}
   end
   def handle_cast({:count, count}, session) do 
     Python.cast(session, count)
     {:noreply, session}
   end
   def handle_info({:python, message}, session) do         
      IO.puts("Received message from python: #{inspect message}")

      #stop elixir process
      {:stop, :normal,  session}
   end
   def terminate(_reason, session) do 
     Python.stop(session)
     :ok
   end
end

Now lets create the python module priv/python/test.py

#priv/python/test.py
import time
import sys
#import erlport modules and functions
from erlport.erlang import set_message_handler, cast
from erlport.erlterms import Atom

message_handler = None #reference to the elixir process to send result to

def cast_message(pid, message):
    cast(pid, message)

def register_handler(pid):
    #save message handler pid
    global message_handler
    message_handler = pid

def handle_message(count):
    try:
        print "Received message from Elixir"
        print count
        result = long_counter(count)
        if message_handler:
           #build a tuple to atom {:python, result}
           cast_message(message_handler, (Atom('python'), result))
    except Exception, e:
      # you can send error to elixir process here too
      # print e
      pass
def long_counter(count=100):
    #simluate a time consuming python function
    i = 0
    data = []
    while i < count:
        time.sleep(1) #sleep for 1 sec
        data.append(i+1)
        i = i + 1
    return data

set_message_handler(handle_message) #set handle_message to receive all messages sent to this python instance

Compile!

$ mix compile

Run!

$ iex -S mix
iex(1)> ElixirPython.PythonServer.call_count(4)
[1, 2, 3, 4] #printed after waiting 4sec
iex(2)> ElixirPython.PythonServer.call_count(10)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10] #printed after waiting 10sec!

Also notice that call_count causes iex to hang till we get the back result from python

Now let’s call same python function asynchronously!

iex(3)> ElixirPython.PythonServer.cast_count(4)
:ok
Received message from Elixir
4
Received message from python: [1, 2, 3, 4] # printed after 4sec, no wait
iex(4)> ElixirPython.PythonServer.cast_count(10)
:ok
Received message from Elixir
10
iex(5)>
nil
Received message from python: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] #printed after 10sec without blocking

Notice how we can continue to work in iex and get the result once our python function completes. Try callingcast_count with different numbers — large and small.

That is it! Now you can communicate between Elixir and Python asynchronously.

Happy coding!