Transformers and Multimodal: The Same Key for all Data Types

The world of Machine Learning is undoubtedly fascinating, constantly growing, and capable of touching the most diverse sectors, from medicine to space racing, from catering to big manufacturing. There are countless fields of application for this technology and just as many techniques that have been developed over the decades, but they all have one thing in common: data.

Every Machine Learning model exists and works thanks to what it has been able, in one way or another, to learn from data. However, this data can take very different forms, for example, large amounts of text to train language models to generate sentences, understand context or irony, or identify anomalies. Or millions of images of objects, people and animals to create classification or object detection models, and even audio tracks to perform tasks such as identifying a song or its style.

Image by the author

All of this brings with it one big problem: dealing with such different data requires different techniques and therefore whole separate branches of Machine Learning have sprung up, each focusing on one of these data types. In particular Natural Language Processing (NLP) for linguistics, Computer…