10 Surveys About Popular NLP Areas

Transformers, text classification, transfer learning, etc.

Hello fellow NLP enthusiasts! One type of paper written by researchers and very useful for learners is the one of surveys. The surveys, if recent (considering the high speed with which the world of artificial intelligence is moving), are able to give a complete overview of an NLP field in a short time, which would be difficult to obtain by reading the individual articles. In this article, I recommend 10 surveys covering popular NLP areas. Enjoy! 😄

• Text Classification Algorithms: A Survey: An overview of text classification algorithms that cover different text feature extractions, dimensionality reduction methods, existing algorithms and techniques, and evaluation methods. The limitations of each technique and their application in real-world problems are also discussed.

Comparison of text classification techniques from https://arxiv.org/pdf/1904.08067.pdf.

• A Survey on Text Classification: From Traditional to Deep
  Learning: This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021, focusing on models from traditional models to deep learning. There’s a taxonomy for text classification according to the text involved and the models used for feature extraction and classification. The paper concludes by summarizing key implications, future research directions, and the challenges facing the research area.

Comparison of text classification techniques from https://arxiv.org/pdf/2008.00364.pdf.

• Topic Modelling Meets Deep Neural Networks: A Survey: Topic modeling has been a successful technique for text analysis for almost twenty years. When topic modeling met deep neural networks, there emerged a new and increasingly popular research area, neural topic models, with a wide range of applications in neural language understanding such as text generation, summarization, and language models.
• A Survey of the State of Explainable AI for Natural Language Processing: Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an overview of the current state of Explainable AI (XAI), considered within the domain of Natural Language Processing.

Overview of common combinations of explanation aspects from https://arxiv.org/pdf/2010.00711.pdf.

• A Survey on Contextual Embeddings: Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve groundbreaking performance on a wide range of natural language processing tasks. Contextual embeddings assign each word a representation based on its context, thereby capturing uses of words across varied contexts and encoding knowledge that transfers across languages. This survey reviews existing contextual embedding models, cross-lingual polyglot pretraining, the application of contextual embeddings in downstream tasks, model compression, and model analyses.

A comparison of popular pre-trained models from https://arxiv.org/pdf/2003.07278.pdf.

• A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models: Word representation has always been an important research area in the history of Natural Language Processing. This survey explores different word representation models and their power of expression, from the classical to modern-day state-of-the-art word representation language models (LMs).

Comparison of Classical, non-contextual and contextual (Context2Vec, CoVe, ELMo) Word Representation Models. Image from https://arxiv.org/pdf/2010.15036.pdf.

• A Survey on Transfer Learning in Natural Language Processing: Deep learning models usually require a huge amount of data which is not always attainable. Another limitation of deep learning models is the demand for huge computing resources. These obstacles motivate research to question the possibility of knowledge transfer using large trained models. This survey features the recent transfer learning advances in the field of NLP.

Inductive transfer learning methods. Image from https://arxiv.org/pdf/2007.04239.pdf.

• A Survey of Data Augmentation Approaches for NLP: Data augmentation has recently seen increased interest in NLP due to more work in low-resource domains, new tasks, and the popularity of large-scale neural networks that require large amounts of training data. Despite this recent upsurge, this area is still relatively underexplored, perhaps due to the challenges posed by the discrete nature of language data. This paper presents a comprehensive and unifying survey of data augmentation for NLP by summarizing the literature in a structured manner.

Comparing a selection of DA methods by various aspects relating to their applicability, dependencies, and requirements. Image from https://arxiv.org/pdf/2105.03075.pdf.

• A Survey of Transformers: Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. This survey provides a comprehensive review of various X-formers.

Taxonomy of Transformers. Image from https://arxiv.org/pdf/2106.04554.pdf.

• A Survey of Neural Network Techniques for Feature Extraction from Text: This paper aims to catalyze research discussions about text feature extraction techniques using neural network architectures. The research questions discussed here focus on the state-of-the-art neural network techniques that have proven to be useful tools for language processing, language generation, text classification, and other computational linguistics tasks.

Possible Next Steps

Possible next steps are:

• Deep dive into your favorite NLP area and read papers referred by its survey.
• Read other surveys of NLP areas. Have a look at this collection of surveys.

Thank you for reading! If you are interested in learning more about NLP, remember to follow NLPlanet on Medium, LinkedIn, Twitter, and join our new Discord server!