Hey Alexa, What is NLP?

You all must be familiar with Alexa playing your favorite playlist, making you aware of forthcoming turbulence, suggesting the best route back home, and turning your lights off. Have you ever wondered what seemed a futuristic possibility to many has finally entered the mainstream? What is the technology behind this revolution? Natural Language Processing(NLP) is the answer to all your questions.

Natural Language Processing (NLP) is a field of artificial intelligence that narrows the interaction between humans and computers. It has its origin from Machine Translation which came into existence during the second world war.

What is Machine Translation?

Machine Translation (MT) is an automated language translation process where our computer software translates a Source Code (foreign language) to the Machine Code (target language).

Although it sounds straightforward, there are a lot of complex processes which go into making the most basic of translations possible. The different types of MT systems that are used today are Rules-Based Machine Translation (RBMT), Statistical-Based Machine Translation (SBMT), and Neural Machine Translation(NMT).

Source:https://miro.medium.com/max/2362/1*9N7caHoZiHsO5dUWAS6zDQ.png

How Does Natural Language Processing Work?

Data-Pre-Processing

Before NLP tools can make sense of human language, data scientists will need to perform some fundamental NLP preprocessing tasks:

Sentence Segmentation and Tokenization: The first step in the process is to break the text apart into separate sentences. Segmentation can be as simple as splitting apart sentences whenever you see a punctuation mark. But modern NLP models often use more complex techniques that work on unstructured data. After sentence segmentation, we will split apart words whenever there’s a space between them and treat punctuation marks independently.

Predicting Parts of Speech for Each Token: Next, we’ll look at each token and try to guess it’s part of speech — whether it is a noun, a verb, an adjective, and so on.

source:https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854

After processing the whole sentence, we’ll have a result like this:

source:https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854e

Stemming and lemmatization: In this step, we standardize words by reducing them to their root forms. Lemmatization is done by having a look-up table of the lemma forms of words based on their part of speech.

Here’s what our sentence looks like after lemmatization.

source:https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854e

Stop word removal: In this, we filter out common words that add little or no unique information, for example, prepositions and articles (at, to, a, the). Only then can NLP tools transform the text into something a machine can understand.

Overview of the NLP pipeline

source:https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854e

APPLICATIONS:

Embracing NLP in our daily life is an easy task once we understand the tangible benefits instead of the abstract potential. With that in mind, here are some applications to highlight how widely NLP has impacted every sector in the industry.

Chatbots:

Customer experience is the most crucial aspect of any industry. It helps the companies improve their services and keep their customers satisfied. But interacting with every user manually and resolving their problems can be an arduous task. This is where Chatbots come into play.

Source:https://www.cosoit.com/image/catalog/blog/chatbots-nlp-to-nlu/chatbots-nlp-to-nlu-1.jpg

A chatbot is used for instantaneous question answering. It is mainly designed to understand natural language and deliver an appropriate response through natural language generation. They can solve up to 80% of all routine queries and route more complex issues to human agents.

Language Translator:

Have you ever used Google Translate to find out what a particular word means in a different language? I’m sure you must have experienced it at some point. The technique behind it is Machine Translation.

Machine Translation is the technique of rapidly converting the text in one language to another language while keeping the meaning intact.

Some years back, machine translation systems were dictionary-based systems, and they had limited coverage. However, due to the evolution in deep learning, the availability and use of data have become fairly accurate in converting the text from one language to another.

Business Intelligence(BI):

NLP enables companies to learn more about their customers and use those insightful predictions to create more effective strategies. Analyzing sentiments in unstructured data can boost market research. It sheds light on trends and business opportunities. In Business Intelligence, NLP empowers the users to search through their data in the same way search engines like Google or Yahoo enable them.

Autocorrect:

We all use autocorrect unintentionally. Autocorrect is nothing but a software that suggests correction in spelling or grammatical errors automatically while we type. Natural Language Processing plays an indispensable role in auto-correct software. Tools like Grammarly, for instance, use NLP to help you improve your writing, by detecting spelling, grammar, or sentence structure errors.

Source:https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.androidpolice.com%2F2019%2F08%2F20%2Fgrammarly-keyboard-synonyms%2F&psig=AOvVaw34ywm58n1lVeYuM4N5RQ-M&ust=1622201814210000&source=images&cd=vfe&ved=0CAIQjRxqFwoTCLC7iuni6fACFQAAAAAdAAAAABAJ

Challenges faced by NLP:

Human language is astonishingly complex. We behave and express ourselves in myriad ways. Each expression carries a huge amount of data. Although NLP has enabled the automatic handling of natural human language, it hasn’t been able to decipher our emotions and different states of mind.

As humans have increased interaction with machines every day, we expect computers to match our cognitive abilities. However, many user interactions fall outside the capabilities of the system. For instance, the machine might attempt to clarify what the customer wishes several times yet the user would have abandoned the interactive voice response system out of sheer frustration.

Future Vision:

Although the future is filled with challenges, the discipline is evolving at a very rapid rate. The buzz of NLP in the market is growing exponentially. It seems NLP will remove the current barriers in Big Data. According to researchers, the expected market value of NLP will reach $ 16 billion by 2021.We are likely to advance more in the coming years that will make complex technologies look possible.

NLP Online Demos:

• AllenNLP Sentiment Analysis: https://demo.allennlp.org/sentiment-analysis
• AllenNLP Text to SQL: https://demo.allennlp.org/atis-parser
• All AllenNLP demos: https://demo.allennlp.org
• HuggingFace Write with a Transformer: https://transformer.huggingface.co
• HuggingFace Multi-Task model: https://huggingface.co/hmtl
• Stanford CoreNLP Sentiment Analysis: http://nlp.stanford.edu:8080/sentiment/rntnDemo.html

Let’s Hack Some Code!

Here’s a list of the top widely used open-source libraries to use in your project.

• SpaCy (https://github.com/explosion/spaCy)
• Gensim (https://github.com/RaRe-Technologies/gensim)
• AllenNLP (https://github.com/allenai/allennlp)
• NLTK (https://github.com/nltk/nltk)
• CoreNLP (https://github.com/stanfordnlp/CoreNLP)

References:

• https://www.xenonstack.com/blog/evolution-of-nlp/
• https://towardsdatascience.com/your-guide-to-natural-language-processing-nlp-48ea2511f6e1
• https://www.dataversity.net/future-nlp-data-science/
• https://www.analyticsvidhya.com/blog/2020/07/top-10-applications-of-natural-language-processing-nlp/
• https://towardsdatascience.com/autocorrect-8c33f3b472a0
• https://towardsdatascience.com/introduction-to-natural-language-processing-nlp-323cc007df3d
• https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854e