PyTorch-Transformers: An Incredible Library for State-of-the-Art NLP

Imagine having the power to build the Natural Language Processing (NLP) model that powers Google Translate. Well — we can now do this sitting in front of our own machines! The latest state-of-the-art NLP release is called PyTorch-Transformers by the folks at HuggingFace.

What is PyTorch-Transformers?

PyTorch-Transformers is a library of state-of-the-art pre-trained models for Natural Language Processing (NLP).

This library currently contains PyTorch implementations, pre-trained model weights, usage scripts and conversion utilities for the following models:

1. BERT (from Google)
2. GPT (from OpenAI)
3. GPT-2 (from OpenAI)
4. Transformer-XL (from Google/CMU)
5. XLNet (from Google/CMU)
6. XLM (from Facebook)

All of the above models are the best in class for various NLP tasks. Some of these models are as recent as the previous month!

Most of the State-of-the-Art models require tons of training data and days of training on expensive GPU hardware which is something only the big technology companies and research labs can afford. But with the launch of PyTorch-Transformers, now anyone can utilize the power of State-of-the-Art models!

Installing PyTorch-Transformers on your Machine

Installing Pytorch-Transformers is pretty straightforward in Python. You can just use pip install:

pip install pytorch-transformers

or if you are working on Colab:

!pip install pytorch-transformers

Since most of these models are GPU heavy, I would suggest working with Google Colab for this article.

Note: The code in this article is written using the PyTorch framework.

Predicting the next word using GPT-2

Because PyTorch-Transformers supports many NLP models that are trained for Language Modelling, it easily allows for natural language generation tasks like sentence completion.

In February 2019, OpenAI created quite the storm through their release of a new transformer-based language model called GPT-2. GPT-2 is a transformer-based generative language model that was trained on 40GB of curated text from the internet.

Let’s build our own sentence completion model using GPT-2. We’ll try to predict the next word in the sentence:

what is the fastest car in the _________

I chose this example because this is the first suggestion that Google’s text completion gives. Here is the code for doing the same:

The code is straightforward. We tokenize and index the text as a sequence of numbers and pass it to the GPT2LMHeadModel. This is nothing but the GPT2 model transformer with a language modeling head on top (linear layer with weights tied to the input embedding).

Awesome! The model successfully predicts the next word as “world”. This is pretty amazing as this is what Google was suggesting. I recommend you try this model with different input sentences and see how it performs while predicting the next word in a sentence.

Natural Language Generation using GPT-2, Transformer-XL and XLNet

Let’s take Text Generation to the next level now. Instead of predicting only the next word, we will generate a paragraph of text based on the given input. Let’s see what output our models give for the following input text:

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

We will be using the readymade script that PyTorch-Transformers provides for this task. Let’s clone their repository first:

!git clone https://github.com/huggingface/pytorch-transformers.git

GPT-2

Now, you just need a single command to start the model!

Let’s see what output our GPT-2 model gives for the input text:

The unicorns had seemed to know each other almost as well as they did common humans. The study was published in Science Translational Medicine on May 6. What's more, researchers found that five percent of the unicorns recognized each other well. The study team thinks this might translate into a future where humans would be able to communicate more clearly with those known as super Unicorns. And if we're going to move ahead with that future, we've got to do it at least a

Isn’t that crazy? The text that the model generated is very cohesive and actually can be mistaken as a real news article.

XLNet

XLNet integrates ideas from Transformer-XL, the state-of-the-art auto-regressive model, into pre-training.

You can use the following code for the same:

This is the output that XLNet gives:

St. Nicholas was located in the valley in Chile. And, they were familiar with the southern part of Spain. Since 1988, people had lived in the valley, for many years. Even without a natural shelter, people were getting a temporary shelter. Some of the unicorns were acquainted with the Spanish language, but the rest were completely unfamiliar with English. But, they were also finding relief in the valley.<eop> Bioinfo < The Bioinfo website has an open, live community about the

Interesting. While the GPT-2 model focussed directly on the scientific angle of the news about unicorns, XLNet actually nicely built up the context and subtly introduced the topic of unicorns. Let’s see how does Transformer-XL performs!

Transformer-XL

Google proposed a novel method called Transformer-XL (meaning extra long) for language modeling, which enables a Transformer architecture to learn longer-term dependency.

You can use the below code to run Transformer-XL:

Here’s the text generated:

both never spoke in their native language ( a natural language ). If they are speaking in their native language they will have no communication with the original speakers. The encounter with a dingo brought between two and four unicorns to a head at once, thus crossing the border into Peru to avoid internecine warfare, as they did with the Aztecs. On September 11, 1930, three armed robbers killed a donkey for helping their fellow soldiers fight alongside a group of Argentines. During the same year

Now, this is awesome. It is interesting to see how different models focus on different aspects of the input text to generate further. This variation is due to a lot of factors but mostly can be attributed to different training data and model architectures.

Training a Masked Language Model for BERT

The BERT framework, a new language representation model from Google AI, uses pre-training and fine-tuning to create state-of-the-art NLP models for a wide range of tasks.

BERT is pre-trained using the following two unsupervised prediction tasks:

1. Masked Language Modeling (MLM)
2. Next Sentence Prediction

And you can implement both of these using PyTorch-Transformers. So, let’s see how can we implement the Masked Language Model for BERT.

Problem Definition

Let’s formally define our problem statement:

Given an input sequence, we will randomly mask some words. The model then should predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence.

So why are we doing this? The model learns the rules of the language during the training process. And we’ll soon see how effective this process is.

First, let’s prepare a tokenized input from a text string using BertTokenizer :

This is how our text looks like after tokenization:

The next step would be to convert this into a sequence of integers and create PyTorch tensors of them so that we can use them directly for computation:

Notice that we have set [MASK] at the 8th index in the sentence which is the word ‘Hensen’. This is what our model will try to predict.

Now that our data is rightly pre-processed for BERT, we will create a Masked Language Model. Let’s now use BertForMaskedLM to predict a masked token:

Let’s see what is the output of our model:

Predicted token is: henson

That’s quite impressive.

This was a small demo of training a Masked Language Model on a single input sequence. Nevertheless, it is a very important part of the training process for many Transformer-based architectures. This is because it allows bidirectional training in models — which was previously impossible.

Analytics Vidhya’s take on PyTorch-Transformers

In this article, we implemented and explored various State-of-the-Art NLP models like BERT, GPT-2, Transformer-XL, and XLNet using PyTorch-Transformers. This was more like a first impressions experiment that I did to give you a good intuition on how to work with this amazing library.

Here are 6 compelling reasons why I think you would love this library:

1. Pre-trained models: It provides pre-trained models for 6 State-of-the-Art NLP architectures and pre-trained weights for 27 variations of these models
2. Preprocessing and Finetuning API: PyTorch-Transformers doesn’t stop at pre-trained weights. It also provides a simple API for doing all the preprocessing and finetuning steps required for these models.
3. Usage scripts: It also comes with scripts to run these models against benchmark NLP datasets like SQUAD 2.0 (Stanford Question Answering Dataset), and GLUE (General Language Understanding Evaluation).
4. Multilingual: PyTorch-Transformers has multilingual support.
5. TensorFlow Compatibility: You can import TensorFlow checkpoints as models in PyTorch
6. BERTology: There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT (that some call “BERTology”)

Have you ever implemented State-of-the-Art models like BERT and GPT-2? What’s your first take on PyTorch-Transformers? Let’s discuss in the comments section below.

Originally published at https://www.analyticsvidhya.com on July 18, 2019.