The biggest challenge of AI is not people or technology

Yes, that’s correct. Well, I think so and let me explain.

I think the biggest challenge of AI right now is making it an industry or an engineering field. All the other challenges were there for a while if you really think about it. There can be slow downs and booms in fundamental research, talent can be in short supply, still we will make progress. But I think we are yet to define AI as an industry. And we need to do it right away!

But why..? We all can see how AI is changing the world; from photos we take to how we move around. It is becoming more common and ubiquitous everyday. From a technical point of view this means the systems we use in our day-to-day life have much more AI underneath.

AI as a field has existed for more than six decades. But most of its lifetime as a science not as an industry. With the emergence of deep learning and its recent successes, big players (Google et al.) saw the potential and started using AI in their products and services. Now it’s quite a race among them. But do we yet have a field called “AI engineering”? You may say yes because there are few MOOCs and many job openings. Of course it’s good but not enough to define a new field.

If you think about it, software engineering as field is decades old and it has evolved as an engineering field for a while. It has been taught at schools, hundreds of books have been written, frameworks and paradigms have been introduced. AI needs to reach that maturity soon. But still why do I think it is the biggest challenge? Let me explain a little bit more.

Software engineering or software development as a field has evolved under the assumption that everything is deterministic. True that your code stops working randomly and infrastructure fails (we think) for no reason. But those are not intended to be like that. We write code and build infrastructure to work in same way, all the time. But AI, specially more commonly used machine learning or deep learning, is mostly probabilistic. We don’t build things to work in the same way 100% of the time. That’s also the beauty of AI. This might not be new to you, if you at least know the basics of machine learning. And nothing should bother you much as long as you do your research implementation or ML hobby project. But think about building production systems that involve complete software development life cycle with that probabilistic behavior in mind.

We started thinking about all this as soon as we started working with clients. From requirement analysis to system testing, everything changes when you’re working with probabilities underneath. Instead of (ideally) 100% up time, now we’re talking about 70% of the time in one state and 30% in another. Don’t forget that these are probabilities, 60%, 40% is also acceptable. In reality it gets more complicated than this.

“All this is fine but why today? We have some time right?” you may ask. Ideally it should have been yesterday! As I said at the beginning, we see AI products everywhere but everyone is building AI products in their own ways. We don’t have common frameworks (I’m not talking about tools here) or engineering paradigms for AI product development yet. We can be missing compatibility, interoperability, security, scalability, resilience … you name it. I know what you think, I should be out of my mind to think that they don’t maintain these at Google or Tesla. Yes, they do, but it is very likely that those are their own standards and frameworks. It is highly unlikely that these different companies follow same paradigms of AI product development even at a high level as we have been doing in software engineering. And that is why we need to define a new engineering field for AI. We need to talk more, we need to write text books, we need to develop courses for schools.

I have some good news too! Things are not bad as you think. I can remember few related sessions at SOCML 2017, maybe because the sessions were suggested by attendees. Unfortunately I had to skip “ML in Production” for something more interesting (!) but (in my defence) I was at “Software Frameworks for Machine Learning”. Hopefully we’ll be able to see more discussions and events around AI as an engineering field in coming months and years. I think AI incubated in 2010s and it will reach its maturity in 2020s. But there’s a lot to be done!

I’ll be writing more about our AI experiences at ConscientAI in upcoming posts. I would love to to hear back if you think this is useful. Do share your thoughts / feedback below.