Programming languages and innovation
Or the story of three attempts at introducing new programming languages in the workplace and their cross-pollination effects
I have always been attracted by programming languages, especially functional ones. This goes back as far as 1990, when I learned Miranda (a precursor to Haskell) in a programming course at the University of Montréal. I then learned Scheme while doing my Ph.D. and was amazed by its simplicity, its coherent design. The dreadful parentheses (Scheme is a member of Lisp family of languages, well-known for its use in old AI system, but also for being full of parentheses) were never a problem to me. Past a certain point, you don’t see them anymore. In fact, I always liked them and well-written Scheme programs can be really aesthetically pleasing to the eye. I still believe that. But at that time, I thought syntax was not that important for the adoption of a programming language.
Fast forward a few years and a couple of kids…
Prior to working at Nu Echo, I worked for some months at two companies doing professional Scheme/Lisp programming. The first was developing a 3D solid modeling technology, with Scheme as its scripting language. The second had a really sophisticated Lisp development environment. Not the best implementation IMO, but the IDE was just… Wow! Programming was so enjoyable! That was an eye-opener!
Then I joined Nu Echo in 2003, a one year old company developing speech recognition applications for call centers in large organizations. Yeah, large organizations… At that time, that meant Java! (And it still does, in fact.) I knew Java, of course. It was not my preferred language, but at least the Eclipse IDE alleviated some of its verbosity. I can’t imagine doing serious Java programming without a decent IDE like Eclipse or IntelliJ IDEA.
Scheme and the dreadful parentheses
So here I was, doing some Java programming. But you can’t take out functional/Scheme programming from someone so easily! And I had drunk the Paul Graham Kool-Aid… You can easily figure out what happened next!
I tried hard to convince others (management included) that Scheme would be a good fit for the company. I discussed the virtues of the language at will, made some prototypes for DSLs in Kawa Scheme, a Scheme system that compiles to JVM bytecode and integrates quite well with Java libraries.
But things were a little more complicated than I first imagined. A language alone is nothing. You need good editor support (Emacs was simply not an option), linters, unit testing frameworks, and the like. Since Eclipse was the de facto IDE at Nu Echo, I decided to develop a Scheme plugin (SchemeScript) to get at least that part right.
Then, having some experience writing Eclipse plugins, I started the development of a complete IDE for ABNF speech recognition grammars for VoiceXML applications, named NuGram IDE.
Scheme was really a great choice to write language processors, thanks to some macros implementing pattern-matching and parsers. (A basic, free edition of NuGram IDE is still available, with several downloads a week.)
However, the language itself never gained much traction internally, which had the consequence of concentrating the expertise and know-how in the hands of just a few people (which is never a good thing!). Scheme’s alien syntax was certainly in large part responsible for that.
But the most surprising and long lasting impact of Scheme on the company was more the concepts it brought to light. I had countless discussions with Jean-Philippe Gariépy about the use of continuations, and especially about continuation-based web servers (a concept invented by researcher Christian Queinnec back in 2000), where web applications are written in a more sequential form, hiding most of the back and forth between the client and the server. It turns out that this concept inspired our Rivr lightweight framework for developing VoiceXML-based applications. It does not rely on full continuations (it’s still Java!), but it exposes a similar interface. One writes a dialogue as a sequential program, like:
To do that, you need 2 threads, one for the application and the other for the part that receives requests from a client and sends responses back to it.
The neat thing about this approach is that it makes it very easy to write unit tests of whole dialogues, simulating all events a voice application has to handle. Just replace the part that interacts with the web container, and voilà! You can now use JUnit to write unit tests!
Sometimes, ideas come from the programming concepts themselves!
Second attempt: Erlang
In 2009, we started working on a hosted service for robust parsing and semantic interpretation of text-based inputs. You can think of it as a precursor to NLP engines for chatbots, but without the machine learning part. We thought that project would be an ideal candidate for experimenting with the Erlang programming language and OTP (Open Telecom Platform). The language was developed by Ericsson starting in the late ’80s to help build more scalable and more robust telecom applications. One distinguishing feature of this language is that it heavily relies on low-level, really cheap threads called processes, which communicate through message passing. Each process is programmed in a functional style.
I can’t say the experiment was a great success, as we never proved that we could get a more robust and scalable platform because of Erlang. The reason is simple, we never got much traction on the platform. It is still up and running, but it’s never been used much.
And internally, although more people got curious about our use of Erlang, I still think the syntax (which is highly similar to Prolog) was an issue. And also the lack of a good editor.
But again, we felt the impact of Erlang in an unpredictable way. A few years later, Nu Echo won a project to built a sophisticated, clustered solution for a large bank in Canada. It consisted in a web-based configuration console and a runtime environment executing scripts developed using the console. Since this was to be a fully redundant environment, the whole configuration had to be replicated on all nodes in the cluster.
The system was designed using some principles and architecture inherited from Erlang. All the configuration elements were organized in a tree-like structure. Each and every change to the configuration was applied to the tree in a functional way, i.e. the tree is replicated, with the exception of one node, which contained the change. (In fact, since both trees are almost identical except for one node, most of the nodes are shared instead of having two distinct copies.) When all validations are done on the new tree, a simple swap of pointers can occur, the application now using the newly created tree. This is really similar to how you would structure such a system in Erlang: you have a process that receives a message, alters its state in a functional way, then waits for the next message.
This idea was really dead simple. But it mattered a lot in terms of simplifying the design and ensuring robustness. If a problem occurs during the creation of the second tree (a bug, no more memory, etc.), an exception is raised and the previous version remains in effect. No corruption of data structures! In fact, if we exclude some data replication problems we had in production due to cluster split brains, we never had a single bug in that part of the system in 6 years. That tells a lot!
And then, Elixir!
I always liked Erlang. In fact, I like Erlang more than Scheme now. And the reason is simple. You need more than just the language, you need the platform. Like Java and the JVM, and the tools that come with it. With Erlang, you have a nice language, but more importantly a platform to start developing highly concurrent, distributed applications. And you have all the tools to monitor your programs at runtime (the Observer application lets you monitor any node in the cluster, inspect internal tables, explore your services, etc.).
And then came Elixir. A language designed by people from the Ruby/Rails community, with a strong focus on a simple yet attractive syntax, and tools.
After studying the language on my own, I suggested we use it in our Omnichannel Innovations Lab. The goal was to be able to make fast proof-of-concepts (and chatbots PoC, as a matter of fact), even if it’s disposable code. Dynamic languages are notoriously good for that.
Another aspect of the language that was very attractive for the kind of systems we build is pattern-matching. A lot of the code we write is to interface to existing HTTP-based APIs, typically consuming and producing JSON structures. Using Elixir, Phoenix, and some other libraries, making HTTP requests, parsing JSON and destructuring documents to extract content is a piece of cake. Just try that in Java!
I think Elixir got the attention from more people internally than Scheme and Erlang ever had. We were able to hire interns based on our choice of the language, and all those who tried the language just love it! The tooling is great, always improving, and editor support too (in Atom, Sublime, VSCode, at least). Our CI server supports it. What else can we expect?
Syntax does matter, after all!
I think the conclusion here is two-fold:
- The syntax of programming languages matters much more than I was expecting ten years ago. It has to be familiar enough. Unless you need some features really bad. But I’m certainly not the first to realize that.
- It is always worthwhile to learn other programming paradigms and languages. They may not be relevant or useful to you today, but you don’t know when the concepts they introduce will be of use. (Well, that’s true of general knowledge anyway.)
Always be on the lookout to learn new things! You never know!
