AI’s Big Trade Secret
Why Algorithms are Worthless
Artificial Intelligence is on the rise, but (for a change) I’m not talking about the great AI panic of 2015. I’m talking about everyday AI that’s build into vacuum cleaner robots, Siri, web analytics, and your email. Everybody is fascinated by the technology, and it seems like AI is making huge breakthroughs.
I’m sitting at the source of cutting edge AI development (pun intended), and as in any other discipline, 95% of the work has very little to do with coming up with amazing new algorithms to outsmart humans. But more importantly, 95% of how well these AI systems perform also has nothing to do with smarter algorithms.
To explain why, here’s a little story.
Back in 2005, I took an applied machine learning class in college. We had a little competition to classify smileys drawn on surveys into various categories like “happy”, “sad”, “angry”, and “confused”. The better exemplars looked like that:
The more challenging smileys were wearing hats, eye-patches, tongues sticking out, ears, nose piercings, Marge Simpson wigs, and Braveheart war paints. I am not kidding.
Each smiley was 200⨉200 pixels large, which meant a solid 40,000 input dimensions. The various teams in the class immediately brought out the heavy artillery: support vector machines, recurrent neural networks, subspace ensemble classifiers… Their performance was well below satisfactory.
Disappointed with the absence of a quick win, our team tried something which we would now call “feature engineering”. We wrote a little script that would start at the bottom edge of the image and draw a line up until it hit the first black pixel. Then we compared the length of three of those lines, and also the ratio of black pixels in the upper left quadrants to compared to the upper right quadrant:
In the end, we ended up with only three values. We built a maximally dumb decision tree with these three values and outperformed all other teams by a large margin.
However, what I now call “feature engineering” and do every day was called “cheating” back in the class and got us disqualified, which lead me to this tendentious observation:
The dirty secret is that this kind of feature engineering — using your human intuition, domain knowledge, and reckless shortcuts to reduce 40,000 input dimensions to three— is exactly what makes most of AI applications work. The other thing, of course, is having enough, good, well-curated data.
This is precisely the reason Facebook and Google are giving away all of their machine learning and AI infrastructure for free. Algorithms, on their own, are worthless.
The real value is in what you feed these algorithms, and companies keeping a tight lid on both their data and their feature engineering. The real job of many AI engineers is using their experience massaging data into something more digestible for the algorithms.
If you had a similar feature engineering win, please share your story!
