(tf 2.0 + keras + BERT layer)
Last time, we built an ENG version of the Text Classifier with a micro dataset. For this, we were utilizing a user-friendly framework Fast.ai. That method is based on Universal Language Model Fine-Tuning (ULMFiT).
This time, we have decided to experiment with BERT as long as its popularity as well as a variety of its usage are growing at a rapid speed. We are going to use BERT layer in a model applying Keras.
So here is what we have:
- A micro dataset in DE language (text and labels)
- Tesnorflow 2.0
- Keras
Of course, it is better to work with a larger dataset. To be precise, at least 200 samples are already a good way to go. But let’s be realistic. Usually, businesses don’t have that necessary data, but still want results. In this experiment, we are going to replicate this common situation in business, using a very small dataset.
Our approach to this experiment was inspired by the following article: BERT+KERAS+TF2.0 — fine tuning SPANISH.
Using Google Colab and Google Drive
To start fast prototyping, we use Google Colab (which, by the way, offers free GPUs within certain time limitations).
But most of the coding is done on a local PC. So we synchronize Google Drive with the local PC and then mount Drive on Colab notebook. Here is a vivid explanation on how to link Google Colab with your gDrive.
And in the next paragraphs, we are going to demonstrate you how to synchronize Google Drive with local PC, step by step.
1. We create a separate Google account for development (in such a way, we avoid having some private staff mounted on Google Colab).
2. Then, we install Open Drive to sync the local development directory with gDrive (as an alternative, you can also clone or pull your code to Colab environment and run it from there). So every time we apply local code changes, it transfers to the cloud storage.
3. In Colab Notebook, the first thing we need to do is to mount drive by using the following command:
It will require you to enter your authorization code provided by the link. Once it is done, you will see the following message:
To make sure that everything is OK, check content on the drive:
Use Tensorboard in the Notebook
Before installing all the required packages, you need to set up Tensorboard:
Then, map to the directory where training logs will be stored:
NOTE: we loaded our BERT MODEL directly to our working directory in GDrive. But in order to save Google Drive storage, you should better load and unzip model directly in Colab environment and simply use this path in the training process.
Install Necessary Packages in Colab
For this part, we will use Tensorflow 2.0:
And that’s it. All the other necessary packages are installed by default in Google Colab environment.
Preprocess and Training Code
This is a local python code which we will run later in Colab:
Run a Training
Go back to Colab notebook and run the following command which will start python script in colab environment on GPU:
Our model graph:
Watch Training Results
loss: 0.0723 — accuracy: 0.9794
Evaluate Model with Forms
Google Colab notebook provides us with a simple form where we type our own phrases to evaluate the model. Before using forms, we initiate our predictor method:
Example Colab Code for the form with four input fields:
Then we call model.predict using values from the form variables:
So, that’s all. To receive good results, you should also prepare a good dataset. With the help of examples, you train the model.
Our next experiment will be using BERT as a QA system (question/answer) which should be able to learn from a few pages of the text. Till then!
This article was originally published on ElifTech Blog.
