Learning to Write: Language Generation With GPT-2
A guide on language generation and fine-tuning language generation Transformer models with Simple Transformers. It’s easier than you think!
Transformer models are now state-of-the-art in most, if not all, Natural Language Processing tasks. Personally, I find language generation to be one of the most intriguing out of the myriad NLP tasks. There’s almost something human in being able to generate text that is not only grammatically correct but also cohesive and meaningful.
Transformers have risen admirably to the challenge of language generation with many models capable of generating impressive sequences of text. Out of these, the GPT-2 model, released over a year ago by Open AI, remains one of the best at language generation.
GPT-2 is a large transformer-based language model trained using the simple task of predicting the next word in 40GB of high-quality text from the internet. This simple objective proves sufficient to train the model to learn a variety of tasks due to the diversity of the dataset. In addition to its incredible language generation capabilities, it is also capable of performing tasks like question answering, reading comprehension, summarization, and translation. While GPT-2 does not beat the…
