This interview is part of a series of interviews with EMPEX speakers. Today we hear from veteran EMPEX speaker Pete Gamache, CTO at Appcues, who is giving a talk entitled “Inclusion Starts with Docs” at EMPEX LA on February 10, 2018.
What have you been up to the last couple years?
Running Elixir in production! Which I’m happy to say. We’ve been in production with Elixir for almost two full years at this point. Two years ago right now I was building our API service; it wasn’t yet in production. And at this point Elixir is one of the more boring parts of our stack. Which I can’t say anything better. Not like it’s boring to program in, more like I don’t get woken up in the middle of the night because I chose Elixir.
That’s a powerful endorsement. Languages don’t often sell themselves on being boring, do they?
You can be of two minds there. Elixir has some of the advantages of both. I really like being part of a growing ecosystem — and that’s something that hasn’t changed since the last time we did this interview. The community still feels roughly the same, maybe bigger, but still has a very welcoming feeling to it. I still find that members of the Elixir community approach problems a little bit more maturely than I see in some other communities.
Problems like what?
Just general approaches to writing software, and honestly I think it spills into people’s general attitude. There are more people in other communities, and what that means is there are more people who will happily jump down your throat for being a noob if you’re making any sort of mistake. It doesn’t have to be a noob mistake. The feeling I get from the Elixir community is there are a lot of people who have made mistakes themselves and realize that this is a completely natural part of what we do. And the name of the game is managing that and not mocking people when they do make a mistake.
Immutability is a nice example of this. You can take two approaches to the concept of wanting to do things concurrently. You can say “I’m smart enough to keep all of this straight in my head” or you can say “maybe I am, maybe I’m not, but if everything’s immutable then there’s a whole class of problems I don’t even have to worry about.” That to me is a mature approach to the problem. It’s not the only mature approach to the problem. I guess what I’m talking about in terms of maturity is acknowledging that problems do occur and figuring out ways to avoid those problems is a lot more important than mocking people for it.
Realizing that there’s a certain maturity in saying I’m not perfect, I can’t do everything, so how do I set up the conditions to avoid those kind of mistakes?
Yeah, and Elixir takes just one approach to it. I spent December writing a bunch of Rust. I try to do Advent of Code every year and I decided this year to look at Rust because I’ve never been forced to deal with the concepts of ownership and borrowing head-on in a language. Those concepts are how Rust tries to make it difficult for a programmer to shoot themselves in the foot. C is the example of an arrogant programming language for arrogant programmers who don’t think they make mistakes. That’s not even an oversimplification, that’s just a dis.
Have you looked at the JSON parser that Michał Muskała just wrote?
No. I know there is one.
I haven’t even used it in production yet but I was looking through the code. It seems to be pretty good code actually. It’s built around a byte-matching macro that compiles down to jump tables — I believe that’s the secret sauce to how fast it is. If I’m wrong, hopefully someone will gently correct me when this interview comes out.
In any case, there was an open GitHub issue for a pretty printer. In order to put a pretty printer in without affecting the speed of the main loop, he wanted one to be completely outboard. He put a Help Wanted tag on the issue so that got me to thinking on my bicycle ride home that it wouldn’t be that difficult to write a JSON pretty printer — or a JSON minimizer, they’re really two different sides of the same problem — and so I figured I’d try to write a fast version in Elixir. So I wrote a slow version in Elixir. And the whole time I thought “this would be so fast in Rust.” So I wrote a Rust version of it as somewhere between a yak shave and a prototype. Once I was more or less done with the Rust version I had a much better idea of what I wanted the Elixir version to look like. Anyway I just pushed that PR a day ago. We’ll see if it lands.
You’ve been working with Elixir for a while. What new things have come up for you recently?
There were some random segfaults that I was getting in BEAM 20.0 that I stopped getting in 20.1. That was nice. I’ve found that it’s relatively difficult to find a high performance HTTP client that does pooling that will not trip over its own feet after a while. This is a problem that is driving me up the wall because you’d figure you’d be able to just pull Hackney the shelf. You wouldn’t associate Elixir or Erlang with a bad story around making repeated TCP connections to the same host. But that said, we’ve debugged a lot of problems around our connection to other HTTP services as part of our server pool, so that’s kept life a little interesting.
Is that because the libraries aren’t managing their own resources properly?
Essentially, the different HTTP clients all have different problems and will all go off into the weeds in different ways. There’s a patched version of Hackney we’ve been working with. Unfortunately we see a performance hit there. We’re still trying to find the perfect solution.
What else? Elixir also wears well. At this point I’ve been programming in Elixir constantly for over two years now and there really aren’t parts of the language that irritate me more with time. The tooling continues ot improve. Elixir 1.6.0 is coming out with the formatter. I think we’ll have a very similar situation to the Go community, which is: nobody loves go fmt style but everyone loves go fmt. It’s just a way of ending a bunch of really stupid discussions. I’m in favor of that.
One question that’s been bouncing around the community is the idea of what a mature Elixir app looks like. What’s it like to work on a mature Elixir project?
I definitely roll my eyes at where my coding style was two years ago. There are plenty of non-broken parts that could be a lot better and would be a lot better if I wrote them today. A lot of it was written prior to Elixir 1.3 so the with keyword hadn’t appeared yet. After the with keyword emerged I really settled on using {:ok, value} and {:error, value} tuples as return values, Erlang-style. It felt like there was some uncertainty around the community two years ago whether that was going to be a dominant idiom. And I think the with keyword pushed that over the edge. You essentially get the Either monad as part of the language and you don’ t have to think too hard about it.
We’re right now in the process of starting to carve our monolith into multiple services. We’ve waited a long time to do that just because we didn’t have a lot of headcount. We’re essentially two programmers working on the API. Up until last September it was mostly me, so that was two years I was working on that project largely solo. A lot of ops decisions I just pushed into the future. So we have this monolith and it’s deployed in a few different contexts. It’s exactly the same API, but depending on where it’s deployed, different endpoints will get hit and it’ll perform different functions. So we’re in the process of cleaning up the way we do that. We have our first dedicated ops engineer. We’re excited to bridge the gap between how our Elixir projects are set up and how our other projects are set up. We’re looking for some commonality in configuration and secrets management.
The deployment is always going to look a little different because we’re trying to play to the Elixir/Erlang platform’s strengths. But at this point all our server pools are autoscaling. When I gave the talk at EMPEX 2016 we were not autoscaling at that point. Our servers were pets and not cattle. They’re cattle now. We didn’t have to do much work to get there. We’re using a slightly hacked version of eDeliver that will query AWS for servers with a given tag that we’re looking for and that’s how we know what hosts we can deploy to.
Your original talk at EMPEX had the most views of all our talks. Deployment’s a very popular topic. It sounds like your attitude towards it has shifted.
Not really. I’d say my attitude has matured as the size of my team has allowed it to. Essentially we’re still using roughly the simplest thing that could possibly work. We do hot upgrades all the time, although I think we’re going to end up restructuring our platform so we’re not as reliant on doing hot upgrades for zero-downtime deploys. It’s a really nice option to have but there is a slight performance hit when you’re performing an upgrade. You’re pausing and unpausing the execution of code while you’re doing the upgrade so there’s always a little bit of a hit and we take care to spread that across the cluster so we don’t do that on all of our machines at once.
But our attitude is still fairly minimal- we’re trying not to fight the Erlang platform. We’re going to veer off a little from the way it does things. Like this configuration and secrets management, that doesn’t really fit in with the mindset that your configs should be baked into your release. We’re moving to the opposite of that.
What are your thoughts on the short-term direction of Elixir?
Improved support for releases is very interesting to me. I think the reason my video has the most watches is because nobody has come forward and said with authority “this is the right way to do deploys on Elixir” and if strong deployment tools became a part of the distribution, that represents an authoritative statement that this is the way we intend you to do it.
Do you think that there’s One Way to deploy?
I don’t necessarily think that there’s One Way to deploy, but I also think of Elixir differentiating itself in terms of programmer comforts and developer experience. You can write all the same stuff in Erlang, and you can release Erlang projects, and the developer experience is roundly worse. In Elixir you get a lot of really nice stuff for free.
I’ll be talking about the documentation system in LA. Coming from almost any other language, it’s amazing to have that stuff built in. You don’t have to Google anything and figure out which one of these five independent projects is where mindshare is. When it’s a part of the core platform and it’s really good, everyone’s going to use it. So I guess there doesn’t need to be one way to deploy, but I’d like to see more deployment functionality as part of the core distribution and perhaps a few very sane default making it very easy to do release-based deployment.
How did pick the subject for your talk?
I am a huge fan of the doc system. I was pondering about the kinds of problems that you see in much larger communities. I’m not going to name any names, but the sort of personality and identity style conflicts where you end up with a bunch of people butting heads and technical discussion is really not a part of that at all. In order to get your project to the point where you have random people on the internet butting heads over it- you need to draw people in. You need to make people care about what you’ve done. And the easiest way to do that is write really kickass documentation.
Elixir has the best documentation system of any language that I’ve ever used. I want to teach people about it. It’s such a multiplier of a small project to easily and concisely say “this is what the project does, here’s how you use it, here’s some code samples and they’re part of our test suite so you know they work, here’s how you can get to our source code.” Elixir has these features that other languages don’t have. Even a lot of prominent projects only have subsets of these features that they’re using in the Elixir doc system. I want to give people an introduction to that.
I think people often look at documentation as something to do after they’ve done the real work.
I like to do things in the opposite direction. Once I’ve thought about a problem enough that I’m ready to put my hands on a keyboard, often the first thing I do is write my sample code. I try to dream up the most pleasant way to interact with the software just totally greenfield. And then it’s a simple matter of making that sample code work. I can’t consider documentation a second-class citizen.
I have this argument with people about code comments — code comments are smell and code should be self-documenting, etc — you don’t have to document what code does. The details are spelled out fully in the code itself. The detail of why you’re doing that, that’s what only a human can fill in. My background also comes from some API design, and an API without docs is not an API. You haven’t said anything. You might have a URL you can hit with data and get some different data back, but if there’s no documentation on how to use it, what you have built is not real yet.
It’s like a lock with no key.
Exactly, and Elixir spoils the programmer with the doc system. You’re just writing inline comments in your code that look completely natural as you’re writing them. Then you snap your fingers and both your code and your project are both hosted online, your docs look exactly the same as Elixir’s itself. That’s a really nice. It’s not exactly dogfooding, but it is a little bit. This is a strong enough system to document the entire standard library.
Are you excited to be back at EMPEX?
Oh my god, you know how I love EMPEX, so yes! Very excited. I’m interested to see how the vibe in LA will be more, you know, L-A. I haven’t spent a lot of time there so that’s going to be nice. But the fundamentals are the same: one-day, one-track, skewed toward technical talks, relatively small group, pianist on stage. What more could you ask?
The pianist is in the back for this one.
Oh, well, you know. East Coast-West Coast.
What advice do you have for people coming to Elixir in early 2018?
You’re right on time. Just jump in. It is one of the friendliest and most professional communities that I’ve seen around a programming language. Help is never very far away. You see José answering noob questions on the IRC channel. That’s how welcoming this community is. Dive in!
Pete is CTO at Appcues, where Elixir has been a boring part of the platform for two years and counting (and that’s high praise). He enjoys pattern-matching, immutable data structures, and long walks on the beach.
EMPEX LA will be held on Saturday, February 10, 2018 in downtown Los Angeles, California. Catch Pete and our other great speakers by picking up your ticket today.
interview conducted, transcribed, and lightly edited by Desmond Bowe.
