First drops of Elixir

As part of my research for my book on reactive machine learning, I’ve been doing a bunch of research into things like supervision, concurrency models, and message-passing. Even though I’m using Scala for the code in the book, a lot of this research has pointed me towards techniques pioneered in Erlang and its closely related toolkit, OTP.

So, I decided to try to understand what makes Erlang so powerful by building things using the language. However, Erlang’s syntax and overall language design have some very strange quirks that didn’t make me optimistic about the project.

But then I discovered the Elixir programming language, which addressed pretty much all of my concerns with using Erlang. It is a young language, built on top of Erlang. At first glance, it seems to solve most of my concerns about spending time hacking around in Erlang.

Here’s a quick list of enticing Elixir features:

• Massive concurrency capabilities from running on top of the Erlang VM
• Homoiconic syntax that looks more like Ruby than LISP
• Metaprogramming via hygienic macros
• Modules for code organization
• All of the typical functional programming niceties like immutability, higher order functions, pattern matching, etc.

Beyond all of those technical concerns, I’ll admit that was interested in checking out a younger programming language community, to get a peek into the social aspects of a young, rapidly evolving programming language. In this post, I’m going to try to record some of my initial impressions of Elixir and relate them back to my experiences with Clojure and Scala.

Elixir Camp

As part of my experiment in programming language anthropology, I checked out Elixir Camp, a community organized conference. Actually, that’s not quite right. The event was actually the Erlang/Elixir/Phoenix/Nerves Camp, with the activities being sometimes single tracked and sometimes multi-tracked. Usually, everyone was in the same room, focused on the same topic. It was a pretty different vibe than you find at a several thousand person Java conference or something.

I don’t have detailed notes on the whole conference but here are a few highlights that struck out to me as a total Elixir n00b:

Evolution

Johnny Winn had a neat example implementation of evolution that did a great job of showing how to use supervision in your application design. He hasn’t posted the code on his GitHub account yet, but I hope he does soon. I found it to be a very good example of how well the approach to designing system components in Elixir can match to the semantics of the application.

In particular, it reminded me of some of the work that some old colleagues of mine have done on MOSES, which is an attempt to evolve artificial intelligence using some ideas taken from real evolution. MOSES has a really complex C++ implementation that I’ve never really understood, honestly. I’d be really interested to see an attempt to implement some of the ideas from MOSES in Elixir.

Changing the World

Dave Thomas was the keynote speaker, and he came with an ambitious agenda: trying to motivate us to change the world. After framing the concept of disruptive innovations, he enumerated a set of innovations that he saw coming together in an exciting way: open source, mesh networks, low-power protocols, distributed fault-tolerant algorithms, tiny and cheap processors, Phoenix, and Nerves. For his full vision, you’d really need to watch the talk, once it’s posted.

The part of it that I want to highlight was his insight that Elixir’s focus on communications and embedded devices instead of just more web apps was unique and powerful. Moreover, the possibilities of enabling more peer-to-peer architectures has some interesting political implications as well. It reminded me of the usage of FireChat during the Hong Kong democracy protests. You can also see this in the various databases that can tolerate extreme forms of partition tolerance, many of which are implemented in Erlang. I talk more about how partial knowledge in partition-tolerant databases works, using the specific example of Couchbase, in chapter 3 of my book.

Presence

Beyond the interesting possibilities posed by peer-to-peer architectures, Dave also talked about an interesting concept of identity and presence, which was continued in the talk by Sonny Scroggin in his overview of channels and presence. The sophistication of the communications functionality provided by Phoenix really blow me away. Despite some very surface-level similarities, Phoenix is really something quite different from Rails. Comparing it to what’s available in Scalaland, it combines functionality found in Play, Akka-HTTP/Spray, and beyond into something that feels a lot more general.

In particular, I was very impressed to see Phoenix use conflict-free replicated data types (CRDTs). Several folks are working on a CRDTs in Scala, but it’s really exciting to see a concept so powerful plugged into the language standard web/messaging framework. Powerful CS concepts like CRDTs clearly have a place in enabling anything like the bold vision that Dave talked about, but developers are only going to use them if they’re well integrated into productive frameworks like Phoenix. My early impression is that Phoenix and Elixir are going to do a lot to bring distributed systems techniques to a much wider audience.

Heritage

One of the most striking things about the Elixir community to me is their attitude towards Erlang and the decades of work that came before. It’s really clear that Elixir developers understand and appreciate what is great about Erlang: immutability, pattern matching, supervision, processes, applications, transparent distribution, and on and on.

Clojure and Java

The Elixir community’s reverence for Erlang is really striking after having spent some time writing Clojure and hanging out with that community. I often encountered people who had the attitude of:

Everything about Java is terrible. Clojure is great because it changes everything that was terrible about Java.

I never agreed with this stance. I find Java’s static type system deeply imperfect but still very useful. When I used to write Clojure and Java for a living, I often felt like the right solution for any problem was a mixed Clojure/Java project that allowed me to get some sort of guarantee about my code via compilation.

This sort of fundamental distrust for the semantics of your runtime has real consequences. In particular, Clojure’s radically different semantics mean that it can be much slower than Java, due to things like the reflection necessary to implement a dynamic type system on top of infrastructure built for a statically typed language.

Contrast this situation with Elixir. By design, Elixir shares a lot of semantics with its host language, Erlang. Elixir programs can call Erlang code with no overhead and vice versa. Perhaps more importantly, they share a type system, success typing. This allows Elixir to reuse Dialyzer, the awesome static analysis tool developed for Erlang.

Make no mistake: Elixir is waaay more than just syntactic sugar for Erlang. It’s a thoroughly conceived approach for how to build the tools that developers need in 2016. From tools like IEx and Mix to the rapidly evolving approaches to collections and dataflow, examples of exciting innovation are all around. But Elixir devs seem to (rightfully) exhibit a lot of reverence for the accomplishments of Erlang and the strengths of its design. Erlang and Elixir devs both view the enormous success of WhatsApp’s usage of Erlang as part of the proof of the value of the technologies in this ecosystem.

Scala and Java

In several ways, Elixir reminds me a lot more of Scala than it does of Clojure (even though there’s clearly a lot of sharing between the Clojure and Elixir communities). Despite being a functional programming language, Scala doesn’t reject the object-oriented features of Java. Instead, it tries to take them further and make them both more powerful and easier to use than they are in Java.

Of course, Scala also introduces a new type system that can do some pretty crazy stuff beyond what is possible in Java. But even Scala’s insanely sophisticated type system isn’t a complete departure from what came before. If anything, the Scala type system could be viewed as a rationalization and extension of the Java type system.

For better or for worse, Scala has always provided developers new to the language a path from imperative style of Java to the functional style of Scala. This extends all the way to allowing for the use of mutability, which ends up coming in handy for interoperation with Java code that doesn’t hide the differences between the two languages.

The downside of this deference to Java is that sometimes new Scala developers will write what is often called “Java in Scala” by overusing some of these Java-friendly features and failing to understand some of the bigger ideas of Scala. Hardcore Haskellers might view this as watering down the principles of functional programming, but I’ve always viewed this as a necessary, pragmatic approach that rightfully recognizes that not all of OO was bad. From what I can tell, this pragmatic approach has gotten more developers writing functional code full time than any preceding effort, and that is a real achievement.

Moreover, those new Scala developers can repurpose a lot of their useful knowledge around things like multithreading on the JVM, tuning garbage collection, artifact packaging, dependency resolution, etc. It’s a huge benefit to the Scala community for people with deep JVM skills coming over to learn Scala and contribute.

Opportunity

I see a similar opportunity for Elixir. With its friendly (yet macro-ready) syntax, Elixir invites Rubyists and other web developers to jump right in to a new language that promises to solve some of the challenges with concurrency that have plagued web app frameworks like Rails on MRI. The foundations of the Erlang VM are at least as relevant now as when they were originally invented. All sorts of applications are becoming effectively distributed systems. This is a large part of the impetus for the Reactive Manifesto that has been taken up by the Scala community. These lines of work are intertwined, most obviously in the relationship between Akka and the Erlang actor model but also more generally in trying to make parallel, concurrent, and distributed programming easier.

Don’t mistake my enthusiasm for Elixir: I’m not abandoning the good ship Scala anytime soon. I think that there’s no better language than Scala out there for the sort of large scale data processing that we have to do in machine learning. I love seeing a gigantic community and set of resources grow up around Scala and Spark.

But it’s also really exciting to see different approaches to the same problems developing interdependently. People are working on CRDTs in both Scala and Elixir (and in other languages), so now we distributed systems engineers get to compare and contrast implementations that have rich static type systems using OO hierarchies vs. implementations that use pattern matching and language-native supervision.

Technical communities help each other grow and learn. Elixir clearly takes some great ideas from Erlang, Ruby, and Clojure just as Scala did from Java and Haskell. This is part of the beauty of programming languages and computer science as a whole:

We can build on what we love about past designs, while still developing fundamentally new approaches.

Getting to participate in the exciting process of recombinant innovation is just one of the many benefits of being a software developer in 2016. I’m personally excited to see where Elixir is headed and what I and the rest of the Scala community and beyond can learn from those efforts. It’s bound to be interesting stuff that we all are better off for having explored.

So for now, that’s my Elixir status.