Building a factory for machine learning
“What does your company do?” I get asked when I tell people I work at Algorithmia.
I’m going to need to answer this question a lot more because we just raised a $25m Series B and so I’m writing this to help all my friends understand how we’re changing the game for machine learning.
Almost no data scientists have the skills to do each of these steps, and so a whole bunch of specialists need to work together to get a single model deployed and scaled. This drags out the whole process and is often measured in months and quarters instead of minutes and hours.
Customers come to Algorithmia’s AI Layer first because it empowers data scientists to deploy and iterate their own models at speed. They fall in love when they realize how it can transform their entire organization’s machine learning life cycle.
We’re building a machine learning factory…
- Divides a task into distinct and measurable steps
- Automates where ever possible
- Allows for quality control and consistency
- Enables massive scaling
How factories multiplied productivity by 40,000X
Adam Smith famously illustrated the multiplying power of a factory by looking at how simple pins were manufactured.
When pins were made by artisans, it was expected that an expert pin maker could produce 10 or 20 pins per day. Once a factory methodology was applied to the workflow a factory was able to put out nearly 5,000 pins per day per worker (and they didn’t have to be an expert).
Having a factory mindset increased productivity per worker by 250–500x. Over time as each step of pin production was automated and optimized a modern factory could easily produce 800,000 pins per worker per day. This is a 40,000X improvement in productivity.
In the coming years we will get to 40,000X productivity in machine learning by automating and abstracting the manual tasks and infrastructure.
How the AI Layer multiplies machine learning productivity
When machine learning models are deployed manually it’s very similar to when artisans were making pins by hand. It’s very complicated, takes a wide range of skills that most people don’t have, and every. single. step. is. manual.
Data Scientists should be able to deploy and iterate their own models, and this is what we’re making possible.
Machine Learning is still its infancy — and building a machine learning factory is the next step in unlocking exponential growth.
If you have any questions:
Here’s my Twitter
Here’s a Hacker News thread