Sep 3, 2020

4 min read

Multilingual NLP — How to easily classify text inputs using Deep Learning APIs

Crowlingo aims to provide a fast way to use NLP technologies working for 100+ languages, so you don’t need to bother creating different models with their specific training data to get your process up and running.

The Crowlingo models take advantage of state of the art technologies to make them understand 100+ languages while focusing on the semantic value without any translation. It means that given a one language only dataset, you will be able to use it to train your custom model that will then be able to work on different languages.

In this tutorial, we will see with a few lines of code how to use your data to create a deep learning intent classification model working with many different languages using the Crowlingo APIs.

We will assume you have a working python environment to take advantage of the provided Python SDK PyCrowlingo. If you don’t want to be working on Python, you can find the API documentation here.

If you don’t have a valid Crowlingo API key yet, you can create a free account here.

Let’s see how to proceed :

Initializing the PyCrowlingo client

First if not done, install the SDK available on pip.

pip3 install PyCrowlingo

Then simply create the client with your API key.

from PyCrowlingo import Clientcrow_client = Client('[YOUR_API_KEY]')

Gathering the data

We are going to use a publicly available dataset that can be found here.
You can access the formatted data as csv as well as the full tutorial code here.

Training the model

At this point, there are 3 steps to make you model work :

DATA_PATH = './data.csv'
MODEL_NAME = 'CrowlingoTutorialIntent'// create the model
crow_client.model.create(MODEL_NAME, "clf")// upload the data
crow_client.classifier.upload_csv(MODEL_NAME, DATA_PATH, fieldnames=["text", "class"])// train the model
crow_client.model.train(MODEL_NAME, model_type="deep")

As we can see in the third call where we train the model, we specified model_type=”deep”. Crowlingo offers two types for the training: svm and deep. The SVM model is the best choice when you don’t have much training data and offers good results with a small training time. The Deep model type is the best when you have more training data, the training service will automatically build the best model to fit your data and find the best associated parameters to offer the optimal result.

The last step might take some time, let’s wait for our training to end and catch the resulting metrics:

metrics = crow_client.model.wait_training(MODEL_NAME)

Once we have a result in metrics, it will mean that our model has done training!

Understanding the metrics

When we gave the data to the model, the service has reserved a certain amount of this data as test, meaning that these data were not seen during the training. It is on this test data that we get a result when calling metrics, let’s have a closer look at it.

metrics['metrics']>> {'micro': {
       'precision': 0.9428270042194092,
       'recall': 0.9428270042194092,
       'f1': 0.9428270042194092},
    'macro': {
       'precision': 0.9555030729962458,
       'recall': 0.94979981014525,
       'f1': 0.9515538737815696},
    'weighted': {
       'precision': 0.9460385179935296,
       'recall': 0.9428270042194092,
       'f1': 0.9432564816628742}}

There are two things to understand: the average type (micro, macro and weighted) and the metrics (precision, recall and f1).

Here are efficient definitions taken from scikit learn :

Metrics :

precision : The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.

recall : The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.

f1 : The F1 score can be interpreted as a weighted average of the precision and recall

For any of these metrics, the closer you are to 1 the best the model performs.

Average type :

micro : Calculate metrics globally by counting the total true positives, false negatives and false positives.

macro : Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

weighted : Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.

Testing the model

Now that we have our new model, we can query it with this code:

res = crow_client.classifier.classify(MODEL_NAME, text)

Let’s have some fun and try it out. We are going to feed it some homemade examples with different languages.

TEST_WITH_TRAD = [('How do I cancel my insurance?', "Comment résilier mon assurance ?"),
('How do we say cow in german?', "Come si dice mucca in tedesco?"),('What is the meaning of life?', "В чем смысл жизни?"),
('Where were you born?', "あなたはどこで生まれましたか？"),
('tell me something interesting about crows', "dime algo interesante sobre los cuervos"),
('how do I set the language setting to german?', "Wie stelle ich die Spracheinstellung auf Deutsch ein?")]for test in TEST_WITH_TRAD:
    trad, text = test
    res = crow_client.classifier.classify(MODEL_NAME, text)
    print(f'{trad} : {res.classes[0].class_id}')>> How do I cancel my insurance? : insurance_change
>> How do we say cow in german? : translate
>> What is the meaning of life? : meaning_of_life
>> Where were you born? : where_are_you_from
>> tell me something interesting about crows : fun_fact
>> how do I set the language setting to german? : change_language

It seems that our model is doing the job, well done! Of course we only tried with 6 examples where a full test dataset would be much better to evaluate the model.