How to build more effective machine learning teams

“Alone we’re ok, but together we’re a genius.”

Pair programming is “the deliberate practice of staffing every workstation with two software engineers focused on writing software together.”

I’ve spoken at hundreds of events, tech talks, and company off-sites. The #1 question people ask me is why I am so vehemently in favour of pair programming. My answer: intensity, sustainable pace, fierce learning, higher quality, no information silos, co-workers actually helping each other. I ran the largest office at Pivotal with about 125 pairs (250 engineers) so I should know. :)

Those advantages are why we’re following the same intense learning paradigm at Helpful. We want to extend the benefits of pair programming to the field of Machine Intelligence (MI).

Pairing is the fastest way to learn

Pair programming is the best way to rapidly create high quality products. FAST. Adding Machine Intelligence into your products shouldn’t slow you down.

Pairing with data scientists gives engineers the magical ability to learn about machine learning research quickly. In return, the engineers teach data scientists how to apply their research and write production ready code.

Everyone gets to see the entire system instead of just the part that is relevant to their expertise. Both individuals will understand how the product works end-to-end and will learn to build quickly. And most importantly, engineers and data scientists help each other and work through problems faster.

Instead of having to schedule brainstorming meetings, two engineers pairing together can knowledge share continuously. Pair programming is innately social and hyper focused. Engineers can now learn from each other and solve problems together for 8 + hours a day. They can keep each other on track and away from distractions, like the good old Facebook and cat videos.

The result is better quality code. The number of lines of code isn’t a metric. Actually, less code is better! With two engineers working through code, talking out loud and helping each other out you’ll see code quality increase drastically.

Data science is too academic

Experienced software engineers and data scientists with PhD’s in Machine Learning come from two completely different worlds. They innovate and tackle issues in different ways. Data scientists are often interested in doing more research and less practical work. They develop new ML theories and don’t typically ship production code daily.

In contrast many engineers have an interest in ML but no academic training. I would argue that engineers don’t need a PhD to apply Machine Intelligence in a useful way. Especially not when they can work with a PhD side by side to accelerate their learning. After all, what’s a better way to learn then to get help from an expert?

For us, pairing means removing the barriers between theory and practicality. We don’t think data scientists should be doing pure research in a production environment. We work on research that can be implemented into our product right away. All the ML we ship becomes valuable to our customers immediately.

Silo’d teams never win

In the old days (and even now), companies would have a separate research team. That team would work alone, coming up with amazing ideas that they would maybe implement 10 years later. Eventually some of the ideas would trickle down to the actual product. In the majority of cases, none of the research sees the light of day. Many research labs work like this.

This flawed method creates a disconnect between the research phase and the implementation phase of product development. The data scientist isn’t getting enough feedback. And the engineer has too many constraints to put research into practice. When you take research and “throw it over the wall,” the product never turns out the way it was intended to, so it has to be revised.

With the help of pair programming, we’re immediately applying the latest academic thinking and theorizing in a very agile way. Some start-ups have their MI experts work alone, away from the rest of the company functions. They never “get out of the building” and test their ideas with real customers. We are doing the exact opposite. It helps us test the latest MI theories in the real world quickly.

At Helpful, our entire team of engineers and data scientists pair all the time. We asked our engineers that never paired before how their experience has been so far at Helpful. They echoed 1) results and 2) supercharged learning. One of the engineers added, “I’m focused the entire time I’m at work. We get used to solving problems in pairs. And that carries throughout the entire team, creating a super helpful dynamic between all of the engineers.”

In Michael Lewis’ new book The Undoing Project, he interviews Daniel Kahneman about his Nobel Prize winning collaboration with Amos Tversky. He recounts his relationship and how they paired and helped each other on so many important ideas in what came to be known as behavioral economics.

Kahneman’s answer was simple: “Alone we’re ok, but together we’re a genius”.

If you’re curious about what we’re building at Helpful, check us out here!