Adding a Language to Dispatch

As of a couple weeks ago, you can run functions in ANY programming language on Dispatch. To start, you need a base image for that language.

Build your FUNctions!

Dispatch already supports JavaScript, Python 3, PowerShell, Java and probably (by the time you’re reading this) others. I’m a big fan of Clojure, so I’ve created a base image for it to use as an example.

Functions, Images, Base Images

In Dispatch, functions are built on top of images, which themselves are built on top of base images. All of those are Docker images:

• Base image provides the language runtime,
• Image provides library dependencies,
• Function wraps the function (source code) into a runtime API server, ready to execute incoming invocation requests

We are building the base image for Clojure, which will contain templates for building images and functions. An overview of this process:

1. Start with an empty Dockerfile
2. Choose the “FROM” image
3. Add language runtime and necessary system packages
4. Create image template
5. Create function template
6. Implement Function Runtime API
7. Write README

Choose the “FROM” image

First, we need to choose the starting point, the “FROM” image. There are many great choices here (scratch, ubuntu, alpine, busybox, debian, vmware/photon2, etc.) depending on what your preferences for libc, package manager, package stability, image size, license and what not. Dispatch uses Photon OS everywhere else, so we’ll use it here as well:

We like Photon OS :)

Add language runtime

Usually you can install it with your package manager, e.g. for Node.js on vmware/photon2 that would look like this:

tdnf install -y nodejs-8.3.0-1.ph2

Our case is a bit different, because Clojure is a JVM language and I’d like to use the container friendly Java 10 runtime, which Photon OS doesn’t provide. Also, turns out, vanilla JDK 10 doesn’t want to “Just Work” on Photon OS, so we use Zulu (an excellent drop-in replacement):

Now we’ll fetch and install Leiningen and Clojure. Leiningen (or lein) is a popular Clojure build and package manager based on Maven.

Now we have a solid Clojure runtime environment complete with package management tools in our base image.

Create image template

A Dispatch base image is supposed to have a directory to build images from. This directory should be specified in the base image metadata label io.dispatchframework.imageTemplate (the default value is /image-template):

LABEL io.dispatchframework.imageTemplate=/image-template

Typically, this directory only contains a Dockerfile:

See the notes below on the Bash-Fu

This Dockerfile accepts 3 build arguments:

• BASE_IMAGE — the base image, to build the image on top of
• SYSTEM_PACKAGES_FILE — lists required system packages in plain text format
• PACKAGES_FILE — library dependency manifest

In the above Dockerfile, we read system packages from the file specified with SYSTEM_PACKAGES_FILE, make sure the list is not empty and has unique entries, and then install them with tdnf (Tiny DNF), the PhotonOS package manager.

Clojure uses deps.edn as its library manifest, so we just copy the provided file as deps.edn, and run clojure -Stree which fetches all the deps and lists them as a tree.

Create function template

A Dispatch image is supposed to have a directory to build functions from. This directory should be specified in the image metadata label io.dispatchframework.functionTemplate (the default value is /function-template):

LABEL io.dispatchframework.functionTemplate=/function-template

Since images are built on top of base images, we’re going to place that directory in our base image, and it will propagate to all derived images.

Typically, this directory only contains a Dockerfile:

This Dockerfile accepts 2 build arguments:

• IMAGE — image, to build functions on top of
• FUNCTION_SRC — function source code is copied into this file at build time

We need to obtain the function and wrap it into the function server — an implementation of Function Runtime API.

We can run Clojure source files like scripts using clojure CLI tool. We add a couple lines to:

1. Require our func-server.main namespace (the Function Runtime API implementation),
2. Grab the user-supplied function as function (by convention) and pass it as an argument to run our API server.

Implement Function Runtime API

The Function Runtime API server needs to wrap the user-supplied function and provide these endpoints:

GET /healthz  - healthcheck
POST /*       - run function

You can check out implementations for more languages on GitHub: https://github.com/dispatchframework

Below is what needs to be done to implement it.

Obtain the function from source code

A function source code generally has enough information to fully define the function. For example, in JavaScript the function is assigned to module.exports — that’s all.

In Clojure it’s only slightly harder than in JavaScript. We just need a convention as to what function should be used as the entry point. This convention is up to the base image’s author, and needs to be clearly documented (the base image’s README is a good place).

In our case, as mentioned above, we will use a function named function, for example:

Once we have the function, we construct a web app from it and launch it with an HTTP server:

Implement the API endpoints

Implementing this simple API in Clojure is straightforward: for each endpoint you need to provide functions mapping requests to responses. See (handle f) below:

Constructing the web app from a function — that’s what (app f) does — is basically taking the function and threading it through some transformations: API handling, JSON request parsing and response encoding.

Capture any thrown error and writes to stdout/stderr

In the above web app, (core/wrap-func f) does all the important work — capturing error and logs:

Again, threading of a function through some higher order functions. Here’s the full source — for the brave and true!

Write README

You definitely want your users to know how to get started with your new language pack. So, make sure to write up quick instructions in the README!