Why AI Now?

My friend Brad Power has been studying IBM Watson, and he writes “Do you have some ideas, or know of someone I could talk to, with some ideas on why AI seems to be coming of age now? … I’m starting skeptical since Index spawned Applied Expert Systems, which didn’t get traction in the 1980s.”

I think I have the experience to explain what is happening. In 1992, I took a year off to study “the economics of innovation”, where innovation is considered as the production of “design”, a real good that has very nonlinear payoffs, infinite durability, and a weird “non-rivalry” property where anyone can use it. I got the general impression that most technologies take about 20 years to go from invention to use. For example, lasers were invented in 1960 but started being designed for widespread use (in CD players) around 1980. You see this a lot if you look. I don’t know what the rate limiting factor is, but there is a theory (which I don’t believe but can’t dismiss) that a new generation of humans has to grow up to use a new technology.

I also took that year to experiment with a pure form of innovation — evolution. I designed a “genetic programming” system that evolved computer code. This was a hot new research topic at the time. I ended up filling a house in New Hampshire with single board computers (cloud computing was not available) and going to a lot of A-life conferences. So, I was part of the first wave, the 20 years too early wave, of this technology.

It’s definitely the right time for what they used to call AI, now called “machine learning” or “deep learning” or even reduced to “data science”. Over the last four years, things have changed and computers are doing things that they couldn’t do before. The classic examples are the self-driving car, and the long-despaired-of untrained speech recognition. Now, people are attempting these things and succeeding commercially, where before it was only a research project There are some reasons why it’s happening now:

• Mystical reasons having to do with the 20 year gestation period
• Moore’s law. If you look at old predictions based on Moore’s law, they pretty much nail the transition at 2010. You can consider my house full of feeble 486 computers to see how important this is.
• And…. the big one … Big data. We suddenly have a LOT more data to play with. It’s lying around all over the Internet in terabytes of logs, and spouting out of the Internet of things. The new AI isn’t like “expert systems” with rules and inferences. It’s data analysis and statistics … with really big data. We have a gusher of data and we are figuring out how to use it in useful ways.

To further emphasize the point about big data being the deciding factor, or even the only factor, consider that the core of Watson is not really an inference engine like the old expert systems. The core of Watson is open source software — UIMA and Hadoop — that allows it to read text at huge scale. Here is an article about it — Apache Innovation Bolsters IBM’s “Smartest Machine on Earth”

Systems that can consume large amounts of information and figure out something from it will do well in an environment where we have increasing amounts of data. They will do well even if some of the data is wrong or ambiguous in the way that human text is often wrong or ambiguous. Now we have lots of data. The article mentions 200 million pages for Watson back in 2011. The data and the analysis is imprecise. Watson converts human text, which is wrong and ambiguous, to rules and inferences using some of the simple parsers in UIMA, which is a sloppy process. However, in bulk it works.