Natural Language Processing (NLP) and A.I.

by Zacharias Voulgaris

What is the relationship between these two fields and how is it expected to evolve in the years to come?

Natural Language Processing, or NLP for short, is a very popular Data Science methodology that has gained a lot of traction over the past few years as more and more companies have realized that it’s easy to get access to text data and use it to derive valuable insights. Twitter, for example, offers a rich data stream which when processed properly, can yield important insights on topics or brands, using just NLP as a paid resource. However, NLP wouldn’t have gone far if it weren’t for A.I., since the latter allows it to go beyond the rudimentary statistical models that NLP has in its toolkit.

Just to clarify NLP is so much more than just running a Bayesian classification system, or a regression model on text data that’s been encoded into binary features. NLP also involves topic discovery, text similarity, and even summarization of a document, among other things. All these tasks would be extremely difficult, if not impossible, if it weren’t for A.I. So, NLP is at least partly dependent on A.I., at least for applications that are really worth a data scientist’s time. Of course A.I. is not there to displace Stats, but rather complement this more formal approach. Think of it as the fox that works side-by-side with the hedgehog against a common enemy, rather than two animals fighting each other for dominance.

What about the other side of the relationship? Does A.I. need NLP in any way?

Well, the short answer is “it depends” since A.I. is very application-specific. So, for any A.I. system that involves communicating with a human as a main part of its agenda it is important for it to be able to use natural language as much as possible. So, NLP is not only useful but also necessary. That’s something we observe a lot nowadays with chatbots, for example, A.I. systems geared at emulating human communication through a web API, in order to convey useful information or facilitate a certain action.

Personal assistants like Cortana depend on NLP to connect with their users. However, A.I. systems in many strictly operational scenarios, such as autonomous vehicles, don’t necessarily need NLP since communicating with the users is not the primary function. This is bound to change in the future as it would be easier to market a vehicle that you can talk with, particularly in case of an unexpected situation, such as an engine problem.

Naturally, the relationship of NLP and A.I. is as much essential as it is conditional. Still, as general A.I. is getting closer and closer, we should expect NLP being an inherent part of A.I. since such a system should be able to excel in pretty much every task that a human can undertake (as well as some tasks beyond our abilities, such as handling big data). So, instead of seeing NLP and A.I. as static entities (hedgehog-like approach), we ought to view them as co-evolving ones (fox-like approach) that at the present moment they have a co-dependent relationship. Still, a A.I. becomes more and more advanced, it is not implausible to expect NLP to become just another module of an A.I. system, much like the linguistic center is part of the human brain.

What does all this mean for data scientists?

Clearly, there is no point ignoring either one of these fields, even if our specialty lies in some other part of data science. So, at the very least we ought to be informed about what’s happening in NLP and how A.I. influences data science. We don’t need to write our own algorithms on these fields, but we should be able to tackle an NLP problem, preferably using some A.I. method, or develop an A.I. system that makes use of NLP in the back-end. As the relationship between NLP and A.I. becomes stronger, it’s bound to become something of a requirement in the near future.

About the author:

Zacharias Voulgaris has a PhD in Machine Learning (University of London) and have worked at Georgia Tech as a Research Fellow and as a Data Scientist in both Elavon and G2. I have authored two books, Data Scientist: The Definitive Guide to Becoming a Data Scientist (2014), and Julia for Data Science (2016).

He also teaches at Ivy Data Science’s Artificial Intelligence Bootcamp where he provides practical training on Machine Learning and Deep Learning.