Photo by Radu Marcusu

🌻The Best and Most Current of Modern Natural Language Processing

Victor Sanh
May 22, 2019 · 7 min read

Over the last two years, the Natural Language Processing community has witnessed an acceleration in progress on a wide range of different tasks and applications. 🚀 This progress was enabled by a shift of paradigm in the way we classically build an NLP system: for a long time, we used pre-trained word embeddings such as word2vec or GloVe to initialize the first layer of a neural network, followed by a task-specific architecture that is trained in a supervised way using a single dataset.

Recently, several works demonstrated that we can learn hierarchical contextualized representations on web-scale datasets 📖 leveraging unsupervised (or self-supervised) signals such as language modeling and transfer this pre-training to downstream tasks (Transfer Learning). Excitingly, this shift led to significant advances on a wide range of downstream applications ranging from Question Answering, to Natural Language Inference through Syntactic Parsing…

“Which papers can I read to catch up with the latest trends in modern NLP?”

A few weeks ago, a friend of mine decided to dive in into NLP. He already has a background in Machine Learning and Deep Learning so he genuinely asked me: “Which papers can I read to catch up with the latest trends in modern NLP?”. 👩‍🎓👨‍🎓

That’s a really good question, especially when you factor in that NLP conferences (and ML conferences in general) receive an exponentially growing number of submissions: +80% NAACL 2019 VS 2018, +90% ACL 2019 VS 2018, …

I compiled this list of papers and resources 📚 for him, and I thought it would be great to share it with the community since I believe it can be useful for a lot of people.

Disclaimer: this list is not intended to be exhaustive, nor to cover every single topic in NLP (for instance, there is nothing on Semantic Parsing, Adversarial Learning, Reinforcement Learning applied to NLP,…). It is rather a pick of the most recent impactful works in the past few years/months (as of May 2019), mostly influenced by what I read.

Generally speaking, a good way to start is to read introductive or summary blog posts with a high-level view that gives you enough context before actually spending time reading a paper (for instance this post or this one).

Who said that naming models should be boring and sad? — Source: Moviefone

🌊 A new paradigm: Transfer Learning

These references cover the foundational ideas in Transfer Learning for NLP:

The Transformer architecture has become ubiquitous in sequence modeling tasks. — Source: Attention is all you need

🖼 Representation Learning:

🗣 Neural Dialogue:

🍱 Various picks:

As a good rule of thumb, you should read papers that you find interesting and spark joy in you! 🤷‍♂️🌟

🌍 General resources

There are plenty of amazing resources available you can use that are not necessarily papers. Here are a few:


Course materials:



🎅 Last advice

That’s it for the pointers! Reading a few of these resources should already give you a good sense of the latest trends in contemporary NLP and hopefully, help you build your own NLP system! 🎮

One last thing that I did not talk about much in this post, but that I find extremely important (and sometimes neglected) is that reading is good, implementing is better! 👩‍💻 You’ll often learn so much more by supplementing your reading with diving into the (sometimes) attached code or trying to implement some of it yourself. Practical resources include the amazing blog posts and courses from or our 🤗 open-source repositories.

What about you? What are the works that had the most impact on you? Tell us in the comments! ⌨️

As always, if you liked this post, give us a few 👏 to let us know and share the news around you!

Many thanks to Lysandre Debut, Clément Delangue, Thibault Févry, Peter Martigny, Anthony Moi and Thomas Wolf for their comments and feedback.


Stories @ Hugging Face