NLP Newsletter: Special Edition! Interview with Erik Dunteman on MLOps and Building ML Platform

Learn what it takes to be a solo founder at an ML startup.

Hi everyone!

It’s been a while! We are back with the regular release cycle of the NLP Newsletter. In this special edition of the NLP Newsletter, I am sharing my conversation with the founder of Booste, Erik Dunteman, on his experience with MLOps and providing pre-trained NLP models via API.

In the next issue, we go back to the normal sharing of machine learning and NLP resources, educational resources, projects, research papers, and tools. Hope you find this issue useful! Please leave some feedback sharing how you think we can improve. Enjoy!

Over the past few years, software engineering interviews have matured and follow more standardized procedures that help applicants prepare better for the interview process. However, the same cannot be said about machine learning interviews. This is understandable especially as the field is still experiencing growth and maturity in all aspects that range from standardized research, industry best practices, and including the standardization of the machine learning interview process.

Part of the reason that machine learning interviews confuse applicants is because there is no consistency and clarity in what specific responsibilities define the different ML-based roles. For instance, inexperienced applicants may be confused or unclear about what are the requirements to become an ML engineer or ML researcher. This is the case because different companies have different expectations of candidates, which makes it very difficult for applications to prepare ahead of interviews. Some standardization and consensus are definitely needed.

In an effort to help applicants better understand machine learning roles, ML interviews, and overall get insights that can better prepare them to pursue a career in machine learning or data science, I will be sharing the results of a series of interviews that I am having with folks building AI and machine learning tools or AI-powered solutions/products. I will be publishing these conversations raw and unfiltered and provide an opportunity for others to get a real glimpse of expectations and what life could potentially be were they to pursue a career in machine learning. This initiative is to provide more clarity on what the different ML-based roles look like in these companies, ranging from ML engineers to ML managers, all the way to founders.

With these interviews, we aim to add some more clarity around topics that confuse people, especially on the expectations surrounding machine learning related roles, careers, challenges, and opportunities. My hope is that through these interviews the reader can better understand how companies use machine learning to solve problems or use those insights to better prepare for machine learning interviews and help people build a prosperous machine learning career. In addition, this is a great opportunity for industry players to connect with applicants and the wider ML community.

Here is my first interview with Erik Dunteman, a solo founder at Booste.

Introduction

Elvis: Briefly introduce yourself (name, experience, education, role)

Erik: My name is Erik Dunteman, I’m a software engineer and founder focused on developer tools and infrastructure.

I have a swiss army knife background — I studied mechanical engineering in undergrad and did internships across agricultural and chemical engineering before teaching myself how to code. My initial inspiration for getting into software was learning about deep learning.

I’ve been full time on angel funded software startups for two years since graduating undergrad, starting with bioinformatics and moving to cloud infra, developer tools, and now to MLOps.

Elvis: Who do you work for?

Erik: I’m a solo founder at Booste https://booste.io

Elvis: What problems are you solving using machine learning and how does it solve the problem?

Erik: My work is more on the Ops side, providing pretrained NLP models via API with simple one-line SDKs to call the model.

As of this week, I’m starting to do custom retraining on GPT2 models, to allow teams to use custom text generation models without worrying about infrastructure and API maintenance.

Elvis: How is the machine learning team structured?

Erik: I’m a team of one

Elvis: Explain with a few words “A day in the life” in the context of your role or the machine learning team?

Erik: On ML-focused days, I’m researching a model or technique that my customers request. For example, today I am working on integrating Uber’s PPLM into my API (https://github.com/uber-research/PPLM/blob/master/run_pplm.py)

I’ll clone the repos, network in my inference servers (if relevant) and run through the demo scripts until I get a grasp for how the data is being processed before and after inference. I’ll then build an API endpoint around this logic. I am generally not retraining.

Product/Technology

Elvis: What is the main service/product your company provides?

Erik: Machine learning inference as an API.

Elvis: Is there an open-source product/effort you are working on? If so, explain what it does.

Erik: The client SDKs are open source: https://github.com/booste-corporation/nodejs_client

Elvis: What is the technology stack (e.g., tools, libraries, etc.) the machine learning (ML) team mostly works with these days?

Erik: TensorFlow Serving as the inference engine, Tensorflow for matrix operations in pre/post processing, ONNX for cross-framework model conversion.

Elvis: What’s a challenging machine learning related task your team has encountered?

Erik: Economically serving the model is a struggle. Especially with large models like GPT2, it’s extremely memory intensive and difficult to justify without a large volume of calls.

Elvis: Are you working with specific cutting-edge techniques like Transformers, GANs, etc.?

Erik: Our work is all tech released within the last four years. We’ve done YoloV3, Transformers (GPT2), and are now working on PPLM which is a single layer inference that tacks onto a transformers model to help guide output to contextually appropriate words.

Elvis: Do you have or are you building your own ML platform/infrastructure? If so, please share how it helps the team?

Erik: Our whole offering is the no-code training platform (to be released) and the API serving infrastructure (currently live with pretrained models). Our customers include even ML-heavy teams who would rather focus on business logic than on model-serving and infra.

Elvis: Are you working with machine learning models in production? What technologies are useful and lessons

Erik: I run everything in Kubernetes (specifically a user-friendly Kubernetes service called https://zeet.co) and would never try to run production without Kubernetes.

RedisAI was my first production tech, and I was very excited by the idea of it. https://oss.redislabs.com/redisai/

It was economically reasonable for multi-model inference systems, since you can load all of the models at once into the server (from TF, torch, or onnx model files) , and they’d be hot and ready in-memory, ready to go. It handled parallel inference jobs really well, and I thought I’d struck gold by finding this relatively new serving tech. Unfortunately, there’s a nasty memory leak somewhere in RedisAI, so my system would crash and restart every hour due to memory overflow.

I’ve moved to Tensorflow Serving. It works well, but I unfortunately need to run each model on a dedicated inference machine, so I lose some economies of scale that the unified RedisAI server gave me.

Elvis: How does the ML team stay informed with the latest machine learning developments/research?

Erik: Word-of-mouth.

Elvis: How does machine learning contribute to the overall business objective?

Erik: It allows us to offer more complex inference options to our users. IE: learning BERT and getting it into production means we can now offer a BERT API as a new feature.

Elvis: Do you often work with data? If so, what are the best practices for working with data (e.g., sanity, cleaning, feature store, etc.)?

Erik: I’ll clean data once a month without using many prebuilt tools, since the focus recently has been just on pretrained NLP models and the Ops side of things.

Opportunities

Elvis: Are you hiring for machine learning related roles? If so, for what roles specifically are you hiring?

Erik: I’m a solo founder and happy to chat with potential cofounders who are more experienced in the training/research side of ML, to complement my infrastructure experience.

Elvis: What level of experience are you looking for in applicants?

Erik: 1 year of ML research experience is ideal, though since it’s a cofounder role: grit, energy, and willingness to learn on the fly is more important than credentials.

Elvis: What incentives/opportunities/activities do the company provide for employees in terms of career growth and learning?

Erik: Learn how to be a founder, ship faster than you ever have before, own a cofounders share of a venture-scaleable company.

Elvis: Are there non-technical AI roles at your company? If so, what are these?

Erik: None

Elvis: How can candidates reach out to you if they are interested in these roles?

Erik: DM me at @erikdoingthings on twitter

If you enjoyed this type of interview, let me know your thoughts here or on Twitter. If you are interested in being interviewed, please connect with me either through email (ellfae@gmail.com) or Twitter.