PyTorch Reimplementation of OpenAI GPT-2 Small Model Released

Synced
Synced
Feb 20, 2019 · 3 min read
Image for post
Image for post

Github developer Hugging Face has updated its repository with a PyTorch reimplementation of the GPT-2 language model small version that OpenAI open-sourced last week, along with pretrained models and fine-tuning examples.

GPT-2 is a state-of-the-art language model designed to improve on the realism and coherence of generated text. It is the largest language model ever, with 1.542 billion parameters. To safeguard against potential misuse, OpenAI open-sourced only a small version of GPT-2.

Hugging Face Science Lead Thomas Wolf tweeted the news: “Pytorch-bert v0.6 is out with OpenAI’s pre-trained GPT-2 small model & the usual accompanying example scripts to use it.” The PyTorch implementation is an adaptation of OpenAI’s implementation, equipped with OpenAI’s pretrained model and a command-line interface.

The Hugging Face repository was first made available last November. It provides PyTorch implementation of BERT with Google’s pretrained models, examples, a notebook and a command-line interface to load any pre-trained TensorFlow checkpoint for BERT. The AI community has shown great interest in the project, which has received 3,700 stars on GitHub. Google BERT also recognized the Hugging Face contribution, declaring it “compatible with our pre-trained checkpoints and able to reproduce our results” on their GitHub.

The project also includes PyTorch reimplementations, pre-trained models and fine-tuning examples for OpenAI’s GPT model and Google/CMU’s Transformer-XL model.

Image for post
Image for post
Image for post
Image for post
Output example from pretrained GPT — 2 (picture Thomas Wolf)

Open AI last week made the controversial decision to not release their language model’s code and training dataset due to concerns it might be used for malicious purposes such as generating fake news. The decision challenged the AI community’s open-source philosophy and has divided researchers. Facebook Chief AI Scientist Yann LeCun — who never shies from controversy — mocked the decision in a tweet: “Every new human can potentially be used to generate fake news, disseminate conspiracy theories, and influence people. Should we stop making babies then?”

Founded in 2016, Hugging Face is a social AI and chatbot startup based in New York.

PyTorch Pretrained BERT: The Big & Extending Repository of pretrained Transformers is available on GitHub.

Journalist: Fangyu Cai | Editor: Michael Sarazen

2018 Fortune Global 500 Public Company AI Adaptivity Report is out!
Purchase a Kindle-formatted report on Amazon.
Apply for Insight Partner Program to get a complimentary full PDF report.

Image for post
Image for post

Follow us on Twitter @Synced_Global for daily AI news!

We know you don’t want to miss any stories. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

Image for post
Image for post

SyncedReview

We produce professional, authoritative, and…

Synced

Written by

Synced

AI Technology & Industry Review — syncedreview.com | Newsletter: http://bit.ly/2IYL6Y2 | Share My Research http://bit.ly/2TrUPMI | Twitter: @Synced_Global

SyncedReview

We produce professional, authoritative, and thought-provoking content relating to artificial intelligence, machine intelligence, emerging technologies and industrial insights.

Synced

Written by

Synced

AI Technology & Industry Review — syncedreview.com | Newsletter: http://bit.ly/2IYL6Y2 | Share My Research http://bit.ly/2TrUPMI | Twitter: @Synced_Global

SyncedReview

We produce professional, authoritative, and thought-provoking content relating to artificial intelligence, machine intelligence, emerging technologies and industrial insights.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store