EMPEX Speaker: James Lavin
James will be speaking at EMPEX on May 19th about “Lessons Learned Building a Medication Service at Teladoc and How Ecto Saved the Day.” James has a very interesting history, so I hope you enjoy this interview with him.
Troy Denkinger: Thanks so much, James, for agreeing to speak with me. I really appreciate it.
James Lavin: And I appreciate the opportunity to be part of this conference.
TD: Can you tell us a bit about what your role at Teledoc is?
JL: I’m a senior software engineer. I have been there a little over three years. Originally I joined as a Rails developer, and about a year and three-quarters ago I was given an Elixir project. I’ve been doing mostly Elixir since then. Although I’m about to transition to something different, and I’m not sure what it’ll be. I’m not sure it’ll be 100% Elixir, but we’ll find out. Hopefully it’s something in that area because I really enjoy Elixir. We do have a number of Elixir apps that need maintenance and probably new features, so over time I’m sure I’ll be working on those.
TD: At Teledoc you built a medication service that is the subject of your talk. Can you describe what that is, and is it in production?
JL: Yes, that’s in production. It does a couple of things.
First, if you opt-in to give your physician the ability to see your medication history, we’ll reach out to a third party company to get your history and supplement that with additional information from our drugs database and other data we have. We use that medication history to inform the physician during a consultation. If they’re considering prescribing new medication, they’ll be able to see if there’s a drug interaction or a drug allergy. So it gives us more insight into your history.
The second piece is knowledge about your insurance plan or plans, if you have them. Again, we reach out to a third party company to get information about which insurance plans you may be on. Then we have built a massive database of how each insurance plan treats each drug. So it’s a drugs by insurance plan database — it’s massive. Once we have your insurance plan, we can look it up in this database. If a physician is considering prescribing Xanax, or whatever it might be, we can quickly see various pieces of information such as whether the insurance plan will cover it, what your copay might be, whether there are alternative drugs that the insurance plan may prefer or have a lower copay on, and various other things.
This is a big issue not only because people like to save money but something like 20% of medication prescriptions in this country are never filled. A big part of the reason is that people just can’t afford the drug. So if we can lower the price, that will save everybody money and encourage them to take the medications they should be taking.
TD: The title of your talk mentions that Ecto saved the day for this application. How is that?
JL: This project originally started out as a functional prototype, and I was tasked with seeing if it was even possible. I finished the functional prototype and it worked, and people were excited about it and they said it’s time to productize this.
At that point the database, analytics, and reporting people took a look. I’d built it in Postgres, but we’re mostly a MySQL shop. They have a lot of standards for the way they name fields and things like that, so they had to dramatically overhaul the database to match their reporting needs. That posed a dilemma because I was relying on the data format from the database in the application, everything relied on that. So I thought I might have to rewrite everything.
I panicked. But then I realized that I didn’t have to rewrite everything. I could use Ecto as an anti-corruption layer to separate all the data that’s coming in from the code and tests I’d written. So if I could just translate from my original data structures to the new structures using Ecto I could feed the old data into my code and everything would continue working. All the certification tests that were passing would continue passing. I could keep all the changes right on that boundary between the database and the code.
It worked beautifully.
TD: Are there other features or libraries within the Erlang/Elixir ecosystem you are using in this application?
JL: I’m going to talk about a number of things in the talk. One that’s interesting is Fuse which is a simple Erlang application that allows you to use the Circuit Breaker pattern. So say a third party is down, the database is down or very slow, whatever it might be, you can specify strategies such as “if i’ve had two failures in the last minute, stop sending requests for the next five minutes” and so on. That has various advantages: you can send an immediate failure response back to the requester and you can avoid overloading the server that is struggling so it can recover from its issues.
That’s an Erlang thing, and one of the points I’m going to make is that because José Valim so thoughtfully made Elixir completely compatible with Erlang, there’s a wealth of libraries we can take advantage of. And the syntax isn’t that dramatically different. It’s quite feasible to use Erlang code within your Elixir apps. We’ve done that in a number of cases. It’s a wonderful option.
Another one is ElixirRetry. Because of the “let it fail” philosophy Erlang and Elixir embrace, you don’t try to specify all the different ways your software can fail, you just try to write for the happy case. Once we put our application into production I found that it was virtually flawless, but every day we’d have
a couple random failures — out of many thousands of requests
— that we couldn’t process. But whenever I tried to replicate those errors I got the right answer. So, related to “let it fail” is how about we retry it once? With ElixirRetry, you wrap your function in retry logic where you can specify how frequently you want to retry. In our case, I’m just giving it a mulligan — you can have one retry. So, we’ll see soon how that worked, but that’s the basic concept. Maybe by adding in ElixirRetry we can get to flawless.
TD: I’d like to ask about your bio a little. So what were you doing for those 14 years where you were not programming? And I’m definitely interested in the New England Patriots book.
JL: I started out as a classics major at Harvard, but I decided that’s not very practical and useful to society. So I switched and eventually landed on government and that’s what my degree is in. But while I was doing that I realized you can’t understand government without understanding the economy, so I decided to go to graduate school in economics. I went to the London School of Economics and did a Master’s there. Then I went to Stanford Graduate School of Business. I planned to do a PhD in political economics, but for various reasons I decided to transfer to the economics department and got my PhD in economics.
While I was doing that, I met my future wife who is from China. She was studying for a physics PhD at Stanford, and I said “Oh my gosh, political economy of China! That would be the best!” — so I started spending more time on my Chinese than on my economics, and I got a Master’s Degree in Chinese on the side.
When I finished my PhD, I had two fellowships to do advanced Chinese in China. I was deciding which program to go to and then the U.S. blew up the Chinese embassy in Yugoslavia. I was watching TV as students were hurling Molotov cocktails at the U.S. embassy in Beijing. And I was like “I don’t think I’m going to China.”
I didn’t know what to do, but one of my professors at Stanford, Paul Milgrom, told me he was working with a startup in Palo Alto, and they could use somebody like me. I did that, and that’s how I got into technology again after about 15 or 20 years. I hadn’t touched it because I’d been scared off. After learning COBOL, my dad said “now you need to learn assembly” and tried to learn assembly. I thought I was too stupid to do programming. [laughs]
TD: [laughs] That’s quite a backstory. At what point did you write Management Secrets of the New England Patriots?
JL: As part of my PhD, I had studied what I called high performance work organizations. The Patriots, who I’d cheered for as a child had never been very good. Then Bill Belichick took over and they started playing really well without any stars. I was reading about it regularly and realized this is a high performance work organization. He has built exactly what I studied. So I thought I should write about it. I wrote a couple books about that.
TD: In your abstract you mention the Teledoc application is an umbrella app. What has been your experience with that?
JL: I am going to talk about that specifically. I love it. There are a few issues with it, one is the robustness of the deployments, and testing it can be a little tricky. The Elixir community is aware of the issues and is working hard to improve the situation. It’s not as simple as just deploying a Phoenix app. I have found the flexibility to be wonderful. I like the fact that everything is in one git repo, yet I can separate things that should be separate. I will talk specifically about how we structure this. I know not everyone loves umbrellas, but I do. There’s a talk by Georgina McFadyen who spoke in London last year specifically about umbrella projects, and she ended up calling them “majestic microservices” and I agree with that.
TD: You have a 30 minute time slot, right?
JL: Yeah, I’m going to have to talk fast.
I really appreciate James taking the time to talk with me. Please don’t forget to get your ticket to the EMPEX conference to be held on May 19th in Manhattan. Say hello if you see me!
