Introducing TamnunML
An easy to use open-source library for advanced Deep Learning and Natural Language Processing
Recent advances in Deep Learning for Natural Language processing are astonishing. We saw major breakthroughs in many NLP tasks including classification, question answering, translation and more. Most of these advances are thanks to two things: New powerful language models and transfer learning methods. Some even refer to is as NLP’s ImageNet moment. The idea is simple yet beautiful: Train a very big complex model on a huge amount of data and then use this model in downstream tasks like classification and pos tagging. Companies like Google, OpenAI, AllanNLP release models that beat state-of-the-art every time.
This is a great time to be an NLP practitioner. Almost any task can benefit from utilizing these methods. From our experience, fine-tuning BERT, for example, improved many models and products we work for a long time.
On the other hand, doing so is not very trivial. While there’s a lot of implementations of different methods in different frameworks, most of them require deep knowledge and work. Many methods and papers require lots of lines of code to implement.
Take BERT as an example. BERT is a Language Model released by Google that showed great results on many different tasks. From our experience, it works like magic. But Fine-tuning BERT is not easy, not only it requires knowledge in TensorFlow or PyTorch, but also takes lots of lines of code that could be prone to errors. Here I show to do it in plain pytorch:
Wouldn’t be awesome if we could just do fit().predict() ?
This is why we created TamnunML!
TamnunML aims to close this gap. We are building an easy to use (sometimes just one-liners) interface for utilizing powerful models and methods. At the moment, we’re focusing on Natural Language Processing and Transfer Learning but hope to expand this to a broad set of algorithms. TamnunML provides the excellent scikit-learn interface for more complex methods implemented using much more complicated frameworks like pytorch .
In tamnun fine-tuning BERT is as easy as fit().predict() !
Using these two lines, we are able to utilize BERTs power for our own work.
Another example is “Knowledge Distillation”, which is the process of transferring knowledge from one (usually big) model into another (usually much smaller). It’s very useful when we want to use the power of a very large model, like BERT, on relatively weak hardware (like CPU vs GPU). You can see a more comprehensive introduction to distilling BERT in this post:
As with fine-tuning BERT, Knowledge Distillation requires knowledge and work. In tamnun we are able to do it with very little code:
We are exploring many other methods to include in tamnun . One of them is Unsupervised Data Augmentation, which is a method that aims to help building models when labeled data is limited by using unlabeled data and augmentation techniques, the authors showed very good results using 20 labeled examples alone! Another one is Parameter-Efficient Transfer Learning for NLP where the authors suggest using “adapters” to fine-tune large models (like BERT) much more efficiently.
Both these methods look very promising in real-life NLP challenges.
We release TamnunML as an open-source library with the goal to expand its capabilities both by us and the Machine Learning community. We hope that you’ll find tamnun as useful as we did and start creating great things. You’re more than welcome to provide feedback and to contribute.
Interested in this type of work? We’re always looking for talented people to join our team!
