EMPEX Speaker: John Kelly

John Kelly, a Senior Backend Developer at Bleacher Report, speaks at EMPEX on May 20th. The title of his presentation is “Monitoring Production Elixir Applications.” John sat with me for a bit to talk about how he came to Elixir and what he’s doing with Elixir in production.

Troy Denkinger: You’re are an autodidact programmer with some interesting languages under your belt like Clojure and Go in addition to Elixir. How did you get started coding?

John Kelly: I had an interest in programming from back in my high school days. I taught myself Java then and took one introduction to programming class. I was on my own from there and got particularly interested in web programming — HTML, CSS — then went on to PHP. After awhile I heard about this new framework called Ruby on Rails, so I took a look at it and saw how much easier it was to do web programming compared to PHP. I went headfirst into Rails and got my first official programming job as a full stack Rails developer. After getting a really great understanding of Ruby I started to explore other languages.

TD: That’s around the time you started looking at Clojure?

JK: I heard about this concept called “functional programming,” it seemed like such a different paradigm, and I was intrigued by that. I had heard about Elixir too, but at that time I decided to go forward exploring Clojure. I learned a lot about functional programming through Clojure. I bought the SICP Lisp book, watched some lectures I found on YouTube by the SICP authors from the 80s, and bootstrapped my way through Clojure. And I just kept building on top of that. I went through book after book, meanwhile I was still doing Ruby professionally. I have a friend who was also working on Clojure in his free time who decided to start looking at Elixir, but I didn’t have time.

That friend, Ben Marx, ended up at Bleacher Report working in Elixir. He told me he loved it and that they were doing interesting stuff. So I decided to join Bleacher Report and have been doing Elixir since.

TD: You didn’t really dig into Elixir until you were at Bleacher Report?

JK: Yes, exactly. Ben gave me advice on books to get to prep for the Bleacher Report job. My first true production Elixir code was at Bleacher Report. That was pretty cool.

TD: What hurdles did you find in your way moving from Clojure to Elixir?

JK: The core concepts are the same. Just different syntax to do similar operations. Obviously the OTP stuff doesn’t map, and I found that to be the hardest thing to learn with Elixir and Erlang. That functionality was new to me and the hardest part about making the transition. The actor model was new to me. In Clojure they have different concurrency models than Elixir, so that was one new concept I had to learn.

TD: What got you over that OTP hump?

JK: OTP took some time to grasp because I was working in Phoenix and Phoenix does a great job of abstracting away GenServers, supervisors, and supervision tree stuff. And I didn’t have a project right in front of me where I could work on those things.

I had an “Aha!” moment when I saw a slide online that broke down what a GenServer is: a generic server that’s blocking on I/O, recursing itself, but also waiting for messages to come from a process. That made sense to me.

TD: At Bleacher Report you’re working with Elixir but also Neo4j and Apache Kafka. Can you talk about how you decided on that stack?

JK: This is a new project where we’re looking at a lot of write heavy operations. We use RabbitMQ for much of our queuing now. Kafka is outside the ecosystem a bit, so my job is to evaluate it and make a recommendation on using it. And this project also has some graph implications — I work on the team that works with user information, so I’m also evaluating Neo4j as a potential persistence layer for storing user data. So far it’s been pretty nice using Elixir with them.

TD: San Francisco has an active Elixir community, I understand, do you participate in that?

JK: Ben [Marx], my coworker, co-organizes the San Francisco Elixir/Erlang meetup. He’s always looking for speakers, so I’ve done some presenting of the stuff I’ve been learning. We have a pretty good turnout, and we’re starting to see new companies want to host the meetup as the community grows.

TD: At EMPEX you’re going to be talking about production monitoring. Of all the technology areas you’re working in, how did you decide on that topic?

JK: I feel monitoring is an overlooked topic. When I joined Bleacher Report, one of the areas I saw I could make an immediate impact was in our monitoring infrastructure. I took it on as an extra responsibility to try to up our game. At the time we were using Datadog for some metrics but we didn’t have infrastructure in place for monitoring our apps.

We didn’t have centralized logging. Most log monitoring was done by logging into the server, and we had many Elixir servers running. You had to hop onto a server and hope your request ended up there. So I introduced centralized logging with the ELK [Elasticsearch/Logstash/Kibana] stack. We saw some huge benefits when we got ramped up. We were able to see how our apps were performing and debugging became easier. These days we use our ELK stack for debugging individual requests, and we have Datadog for statsd aggregation.

I want to get across in the talk one of the challenges in monitoring is not implementing it — it’s straight forward to implement high-quality production logging. The big challenge is seeing what is the value in monitoring. It changes how you handle errors and deal with performance issues. That’s the theme I want to get across.

TD: Thanks for speaking at EMPEX, John. See you in May.

JK: I’m looking forward to it.

Thanks to John Kelly for taking the time to talk with me, and don’t forget to get your tickets to the Empire City Elixir Conference being held on May 20th, 2017 in Manhattan. See you there.