Next Generation Distributed Systems On The BEAM
So I was hired by Wayfair to write a new distributed systems language. Then my new boss set himself on fire and I became a general manager and then I got laid off, soooooo…
Wayfair originally hired me because of stuff I had written up about eventually consistent SQL with CRDTs.
Seeing as big companies have a habit of claiming your thoughts are theirs I took good care to write down my thoughts before starting. This post is from an email I sent to my putative boss before I started at Wayfair.
Also, gies-a-job, Erlang/Elixir/R&D peeps ;-)
In this post I will talk about how we could build a next-gen language on the BEAM platform, in this follow-up article I talk generally about next-gen languages targeting LLVM.
Things have moved on since I wrote this — in particular the rise of a typed language on the BEAM, Gleam and I have written about the challenge of eventual-typing in distributed systems.
The most interesting things about the BEAM platform is the process by which the ecosystem developed.
Erlang itself was built against a tight set of requirements — a range of systems were built in it.
The original team then reviewed the applications built and took the design patterns that emerged and systematised them into OTP — a set of libraries, programmable in part via callbacks, that provide the basis for the modern ecosystem.
An operating system is just a set of libraries that run on a single computer and give your unwritten software a set of properties; the ability to use:
- persistent storage
- some concurrency primitives
- networks
- GUIs
- etc, etc
You can think of Erlang/OTP as a set of libraries that run on two or more computers, which give your unwritten an additional set of properties; the ability to:
- fail over when one computer goes away
- scale across a cluster
- etc, etc
You can think of this as a Cluster System.
Things have moved on since OTP was created — the question is if you were to repeat the process of identifying design patterns built on top of Erlang/OTP what sort of behaviours and structures would you want to bake in to the platform?
The correct way to do this is a proper structured survey of BEAM shops with a well designed questionnaire — but it is possible to sketch out a vision of what that world might look like.
The key elements are mainstays of distributed systems as they are built:
- causality
- consensus
- some sort of hashing/routing/distribution algo
- distributed Finite State Machines
- separation of the control plane from the data plane
- streaming/back-pressure and overload protection
- shared config
- cluster management/cluster join/leave for various cluster types
So a NewTP would aim to give your unwritten software the ability to:
- have pre-defined causal properties
- have pre-defined consensus
- use predefined hashing/routing/distribution mechanisms
- be composed of understandable, testable and documented distributed FSMs
- have plane separation between data/control
- be overload resistant
- be part of a well-behaved and operationally-manageable cluster
- etc, etc
The challenge, as always, is how to eat the elephant. Notwithstanding that the right place to start is a with a properly structured survey, we can have a stab at that as well.
An Architecture Of Distributed Systems
Finite State Machines are simple and well understood things:
- this FSM has three states A, B and C
- A is ready for work, the start and end state
- B is attempting to do work
- C is cleaning up after work
- the following transitions are allowed A -> B, B -> B, B -> C, C -> B, C -> A
Traditionally we think of an FSM as being uni-polar — but that is just a degenerate case, more often we build multi-polar FSMs:
In this world the state B of the first FSM actual contains the remote FSM — B wishes that some thing remote happens and I is the clean initial state on the remote node that it contacts to make it so.
J tries to do it, and K and L report back success or failure. (This is a happy-path diagram only).
There is on top of that a set of failure states associated with partition — did the remote machine get the message? Did the ack just time out? and all the other normal dist-sys problems.
In this world we start seeing a separation of concerns:
Many of the things we want out NewTP to have can be implemented as standard libraries — we can cleanly separate the plane of data from control at this point as well.
The plane of control needs additional things tho:
There are additional problems that we need to systematise the handling of. For reliability you need multi-polar FSMs, not bi-polar ones, and multi-polarity means you must handle consensus and causality:
This functionality needs to be baked into our FSMs.
In addition the clusters so-built must be upgradeable incrementally — by moving individual nodes from version N to version N+1 including full and partial roll-back.
A Possible Way Forward
Needless to say, it must be re-iterated that the correct way to go about this is to do a survey of major BEAM shops and get confirmation about exactly what design patterns are in use and represent major blocks of commonality that could be systematised.
That notwithstanding, here are my thoughts.
The big shift from plain Erlang to Erlang/OTP came via the addition of two things to the language:
- a new set of standard libraries
- major software components that implemented new functionality and that were implemented as behaviours and customisable via callbacks.
I think there is a room for an auxiliary language on the BEAM for writing distributed FSMs. It is an auxiliary language only because the core business logic would be implemented in callbacks which would be written in Erlang, Elixir or LFE. You might consider it a specification language (supported by libraries) that helps the end user design and specify distributed, updatedable, robust multi-polar FSMs with standard routing/hashing and plane of data implementations available at write time.
The task is pretty large:
Lots of the necessary library parts already exist — as other libraries — but systematizing them is not a small task. The issue of how to implement causality and eventual consistency mechanisms across dFSMs alone is a big task.
It is worth stepping through the toolchain.
Clearly the core output of the compiler is the behaviours that implement the distributed FSMs.
However, the dFSMs have plumbing built in — so there is an opportunity to make them sort of self-mocking where the plumbing is replaced by Erlang messages — and can be made to iterate over the failure modes (partition, remote node down, remote going down during comms, remote node coming back, etc, etc) — in otherwords you should be able to compile the code into testing dFSMs and run detailed extensive tests of many types (common test, fuzz testing, model testing, failure testing, upgrade/downgrade testing, etc, etc) on a single machine.
The toolchain/language will also need to be able to have versioned interfaces with upgrade/downgrade built into the contracts.
The language should be typed using the dialyzer typing — and this means that the compiler should spit out generators and (possibly) models for things like Quick Check model testing or Fuzz Testing.
There will be a need for config output to, to be able to bring the cluster up — things that both ends of a dFSM pair can talk to each other.
One of the key outputs of the toolchain should be state transition documents that reflect the actual state of the FSM — that is the diagrams should be first class objects that are in synch with the code and then enable detailed design review *prior* to the actual detailed implementation of the code.
Implementation Notes
There is an anti-pattern of YASLs — Yet Another Shitty Lisp — and there are a range of suggested compiler targets for BEAM languages — like Core Erlang.
My proposal would be to use a hypothetical LISP called Liffey as a target — it would be a LISP composed of Erlang terms that would have a 1-to-1 correspondence with LFE so that the AST generated by the compiler could just be to_string()ed to LFE source code and then would be pushed though the LFE compiler.
The goal, obviously, would be do partial, iterative implementations — the old design-big-build-small way.
If you have read this far you should follow me on Twitter
