A Simple Guide On Using BERT for Binary Text Classification.

The A-to-Z guide on how you can use Google’s BERT for binary text classification tasks. I’ll be aiming to explain, as simply and straightforwardly as possible, how to fine-tune a BERT model (with PyTorch) and use it for a binary text classification task.

Thilina Rajapakse
Jun 9, 2019 · 10 min read
Image for post
Image for post
Photo by Andy Kelly on Unsplash

Update Notice II

Update Notice I

1. Intro

Let’s talk about what we are going to (and not going to) do.

2. Getting set up

Time to get BERT up and running.

3. Preparing data

Before we can cook the meal, we need to prepare the ingredients! (Or something like that. <Insert proper analogy here>)

Much better, am I right?

4. Data to Features

The final step before fine-tuning is to convert the data into features that BERT uses. Most of the remaining code was adapted from the HuggingFace example run_classifier.py, found here.

Your notebook should show the progress of the processing rather than the ‘HBox’ thing I have here. It’s an issue with uploading the notebook to Gist.

5. Fine-tuning BERT (finally!)

Had your coffee? Raring to go? Let’s show BERT how it’s done! (Fine tune. Show how it’s done. Get it? I might be bad at puns.)

INFO:pytorch_pretrained_bert.modeling:Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
INFO:pytorch_pretrained_bert.modeling:Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

6. Evaluation

Time to see what our fine-tuned model can do. (We’ve cooked the meal, let’s see how it tastes.)

7. Conclusion

The Startup

Medium's largest active publication, followed by +705K people. Follow to join our community.

Thilina Rajapakse

Written by

AI researcher, avid reader, fantasy and Sci-Fi geek, and fan of the Oxford comma. On the hunt for an interesting PhD! www.linkedin.com/in/t-rajapakse/

The Startup

Medium's largest active publication, followed by +705K people. Follow to join our community.

Thilina Rajapakse

Written by

AI researcher, avid reader, fantasy and Sci-Fi geek, and fan of the Oxford comma. On the hunt for an interesting PhD! www.linkedin.com/in/t-rajapakse/

The Startup

Medium's largest active publication, followed by +705K people. Follow to join our community.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store