This month, we spotted among others, articles about AI that can diagnose breast cancer with higher accuracy than experts! Let’s start, as usual, with the comic of the month:

1 — Breast Cancer Diagnosis

Interpret screen mammography

A recent evaluation of a AI system for breast cancer screening concludes that it is capable of surpassing human experts in breast cancer prediction.

It is essential to identify breast cancer at earlier stages of the disease when treatment can be more successful. Screening mammography is designed to perform such identification but is complex to analyze and lead to false diagnosis: some healthy patients are diagnosed sick (false positive) and some sick patients are diagnosed healthy (false negative).

This evaluation demonstrated an absolute reduction of 5.7% and 1.2% (USA and UK) in false positives and 9.4% and 2.7% in false negatives and thus surpassing human experts in breast cancer prediction!

2 — Here is Meena, the Universal Chatbot

On 27th January, a Google brain team introduced Meena, a new open-domain human-like chatbot, meaning Meena talks about any topic and it mimics the human ability to converse freely in natural language.

Unlike other state-of-the-art open-domain chatbots (MILABOT, XiaoIce, Gunrock, Mitsuku, and Cleverbot), Meena is an end-to-end Neural Network approach and do not rely on complex frameworks.

Traditionally, chatbot performance is measured through perplexity which measures how accurately the bot anticipates what people will say next. Interestingly, there is no proof that this measure correlates with the chatbot responses being “human-like”. To alleviate this issue, the authors proposed a new evaluation metric called Sensibleness and Specificity Average (SSA) relying on humans judging how chatbot responses make sense and are specific. Two things came up :

• the best Meena version scores 79% SSA, it outperforms state of the art open-domain chatbots and gets close to human performance (86% SSA)
• SSA and perplexity are strongly correlated: the more Meena responses are specific and accurate, the more it is able to predict people’s next answers. This is reassuring :)

With these learnings, the authors hope they can get even closer to human capabilities by reducing Meena’s perplexity hence increasing SSA performance.

Meena network has 2.6B parameters and was trained over 40B words over 30 days using 2048 TPU cores, impressive!

https://deepai.org/publication/towards-a-human-like-open-domain-chatbot

https://arxiv.org/pdf/2001.09977v1.pdf

3 — Creative AI: The Storytelling of AI Dungeon

AI Dungeon 2 is an AI-generated text adventure game. Unlike the original AI Dungeon that used an AI text generator to build scenes and choices for the player, the recently released AI Dungeon 2 is different in one major way: instead of the set commands and human-written storylines that traditionally limit player freedom, players of AI Dungeon 2 can type whatever they want. The game responds to the player’s text input thanks to a novel adaptation of GPT-2 :

In this blog post, the author tells the story of his adventure as “Henry the Wizard”. In a narrative way, he shares his learnings with us, starting from being skeptical at first and enthusiastic in the end.

https://lionbridge.ai/articles/creative-ai-the-storytelling-of-ai-dungeon/

4 — HiPlot: High-dimensional Interactive Plots Made Easy by Facebook

On January 2020, Daniel Haziza, Jérémy Rapin, and Gabriel Synnaeve from Facebook released HiPlot, an interactive tool that allows exploring high dimensional data.

Imagine you collected data from multiple trainings: epoch, dropout, embedding size, learning rate and so on. Hiplot let you explore these dimensions in a simple way, using parallel plots:

HiPlot is interactive, you can select the data you want to drill down by clicking on it. And it is really simple to install/use, pip install it and give it a try!

https://ai.facebook.com/blog/hiplot-high-dimensional-interactive-plots-made-easy/

5 — The Arrival of a Train at La Ciotat Station (1895) in Full-HD

“The Arrival of a Train at La Ciotat Station”, one of the first movies ever, was produced by Lumière’s brothers in 1895. The story goes that when the film was first shown, the audience was so overwhelmed by the moving image of a train coming directly at them that people screamed and ran away! It is probably a cinema myth, though :)

As you can imagine, the movie has aged a little bit. Hopefully, Denis Shiryaev ran a couple of neural-network-based algorithms to improve the situation:

• it upscales the input video up to 4K definition, using the GigaPixel AI tool from TopazLab
• it increases the FPS using Depth-Aware Video Frame Interpolation (Dain)

Denis Shiryaev says anyone could have done this and the credit should go to the authors of the algorithm that make them public on GitHub. However, it is quite funny to see how fast it got viral on the web, and how it went far beyond the data-scientist community.

Following the publication of the video, DeOldify released a colorized version of this video.

6 — Using ‘Radioactive Data’ to Detect if a Data Set was Used for Training

Another blogpost from Facebook.ai. The authors have developed a new technique to mark the images in a dataset so that researchers can determine whether a particular machine learning model has been trained using those images.

This is helpful to researchers and engineers to keep track of which data was used to train a model so they can better understand how it affects the performance of different neural networks.

The term “radioactive” data refer to the use of radioactive markers in medicine that are given to patients before radiography, so as to see a particular organ without harming the patient. Similarly, the “radioactive” marks in the data are harmless meaning they have no impact on the classification accuracy of models but are detectable with high confidence in a neural network.

Blog post: https://ai.facebook.com/blog/using-radioactive-data-to-detect-if-a-data-set-was-used-for-training/

Paper: https://arxiv.org/pdf/2002.00937.pdf

7 — Is Modern Facial Recognition Biased?

This article presents a review of studies about existing solutions for facial recognition. It turns out that many of them have biases such as Asian and African-American faces are falsely identified 10 to 100 times more than Caucasian faces.

In a nutshell, these studies warn against the use of facial recognition systems to make decisions impacting human lives and advocate them to be banned from public places e.g., college campuses, calling for more regulation in 2020.

Read more here: https://lionbridge.ai/articles/is-modern-facial-recognition-biased/

8 — Pandas 1.0.0 Released

First pandas major release in a decade! Pandas is a well-known cornerstone library to whoever need to manipulate data in python. It all started in 2011, and its popularity has been skyrocketing ever since.

No worries, it is not a huge / breaking release says the pandas core team. It is rather a symbolic milestone celebrating the growth of the pandas community.

For this occasion, the core team published this post to share its thoughts about the past decade and the next one.

9 — Machine Learning Co2 Impact

Do you ever wonder about the impact on the environment when you train your algorithms? This online tool lets you compute the Co2 emitted by your training based upon the GPU type and the Cloud provider.

It also gives you advice such as changing the region of computing to reduce your emission. A good way to empower data scientists :)

https://mlco2.github.io/impact/#compute

10 — Bayesian Product Ranking at Wayfair

Wayfair is an online store for housing furniture proposing more than 14M products to their clients. In this blog post, data scientists at Wayfair share their Bayesian approach to the problem of showing more appealing products to their customers.

They used the pystan package in Python to implement their solution. They also present other issues they encountered such as the model updating over time (as customer habits change all the time) and an interesting exploitation/exploration tradeoff: on one hand exploit the knowledge they already have to recommend appealing products, on the other hand, try new configurations to gather more data.

https://tech.wayfair.com/data-science/2020/01/bayesian-product-ranking-at-wayfair/?utm_campaign=Data_Elixir&utm_source=Data_Elixir_269

Best of AI: 10 Articles To Read in February 2020 was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

This tutorial explores how you can improve training time performance of your TensorFlow 2.0 model around:

• tf.data
• Mixed Precision Training
• Multi-GPU Training Strategy

I adapted all these tricks to a custom project on image deblurring, and the result is astonishing. You can get a 2–10x training time speed-up depending on your current pipeline.

Usecase: Improving TensorFlow training time of an image deblurring CNN

2 years ago, I published a blog post on Image Deblurring with GANs in Keras. I thought it would be a nice transition to pass the repository in TF2.0 to understand what has changed and what are the implications on my code. In this article, I’ll train a simpler version of the model (the cnn part only).

The model is a convolutional net which takes the (256, 256, 3) blurred patch and predicts the (256, 256, 3) corresponding sharp patch. It is based on the ResNet architecture and is fully convolutional.

Step 1: Identify bottlenecks

To optimize training speed, you want your GPUs to be running at 100% speed. nvidia-smiis nice to make sure your process is running on the GPU, but when it comes to GPU monitoring, there are smarter tools out there. Hence, the first step of this TensorFlow tutorial is to explore these better options.

nvtop

If you’re using an Nvidia card, the simplest solution to monitor GPU utilization over time might probably be nvtop . Visualization is friendlier than nvidia-smi , and you can track metrics over time.

TensorBoard Profiler

By simply setting profile_batch={BATCH_INDEX_TO_MONITOR} inside the TensorBoard callback, TF adds a full report on operations performed by either the CPU or GPU for the given batch. This can help identify if your GPU is stalled at some point for lack of data.

RAPIDS NVDashboard

This is a Jupyterlab extension which gives access to various metrics. Along with your GPU, you can also monitor elements from your motherboard (CPU, Disks, ..). The advantage is you don’t have to monitor a specific batch, but rather have a look on performance over the whole training.

Here, we can easily spot that GPU is at 40% speed most of the time. I have activated only 1 of the 2 GPUs on the computer, so total utilization is around 20%.

Step 2: Optimize your tf.data pipeline

The first objective is to make the GPU busy 100% of the time. To do so, we want to reduce the data loading bottleneck. If you are using a Python generator or a Keras Sequence, your data loading is probably sub-optimal. Even if you’re using tf.data, data loading can still be an issue. In my article, I initially used Keras Sequences to load the images.

You can easily spot this phenomenom using the TensorBoard profiling. GPUs will tend to have free time while CPUs are performing multiple operations related to data loading.

Making the switch from the original Keras sequences to tf.data was fairly easy. Most operations for data loading are pretty well-supported, the only tricky part is to take the same patch on the blurred image and the real one.

.… read the full article on sicara.ai

TensorFlow 2.0 Tutorial : Optimizing Training Time Performance was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Le MeetUp Computer Vision est né fin 2017 pour regrouper la communauté Computer Vision de Paris.

Le MeetUp CV est là pour permettre d’aborder des sujets de résolutions de problèmes techniques, de présenter l’état de l’art sur la recherche mais aussi d’approfondir les use case que permet la Vision.

Tous les 2 mois, la communauté Computer Vision, forte de 1700 membres début 2020, se réunit à Paris le temps d’un meetup. Le temps d’une soirée, des data scientists viennent présenter leurs apprentissages sur un sujet d’étude ou un cas client. Après les interventions, la communauté échange autour d’un cocktail. Voici quelques exemples de talks qui ont eu lieu :

Meetup Computer Vision #8 — Few shot learning for classification in production par Clément Walter, lead data scientist @Sicara.

Les capacités de classification des réseaux de neurones profonds ont démultipliés les cas d’application de la reconnaissance d’image. Cependant ceux-ci nécessitent d’important volumes de données pour être ajustés (trained) au problème considéré.
Le few-shot learning s’intéresse justement aux cas où ces volumes de données ne sont pas disponibles : comment ajuster un réseau lorsqu’on n’a qu’un seul représentant par classe ?
Cette présentation commencera par présenter le paradigme du few-shot learning et insistera sur la complexité de l’utilisation concrète d’algorithmes type Siamese en production. La présentation sera illustrée par du code open source. Vous pouvez retrouver les slides de présentation ici.

Meetup Computer Vision #9 — The photo lifecycle, from the photons to the eye, by Juliette Chataigner, Data Scientist @Meero.

Le talk était composé de 3 parties :
- Acquisition: Comparaison du post-traitement d’un appareil photo reflex numérique et d’un smartphone.
- Editing: Transformer une image brute en une belle.
- Conclusion sur un dernier problème : l’affichage (différents affichages, calibrage, profils…) et comment ils affectent l’image.

Meetup Computer Vision #10 — Pierre Marcenac, lead data scientist @Kili, “How to scale training data?”

“Il vaut mieux avoir un algorithme correct sur beaucoup de données, qu’un algorithme excellent sur peu de données”
Ainsi, la labellisation de données, même si parfois pénible, une étape critique dans un projet de machine learning. Lorsqu’on veut annoter des données en très grande quantité, il faut et une interface fluide, mais aussi du machine learning (pour faire de la pré-annotation par exemple). L’enjeu réside alors à réaliser le travail d’annotation sans dégrader la qualité des labels. Nous vous montrons comment Kili utilise le machine learning pour annoter de très grandes quantités de données sans dégrader la qualité du process.

Autres talks computer vision présentés lors des MeetUps de 2018–2019 :

• La mise en place d’un moteur de recommandation pour le e-commerce basé sur la similarité visuelle des produits pour augmenter le taux de conversion (Olivier Chancé, @Sicara)
• Le développement par Ubble d’une solution d’identification en ligne simple et fiable, en mettant un effort particulier sur la création d’algorithmes de détection de fraude. Disposant d’un niveau de données inégal en fonction des problématiques traitées, l’équipe a mis au point une solution unique mêlant computer vision classique et deep learning.
• L’entraînement d’un modèle qui reconnaît les références de sac à main dans des vidéos de défilés de mode et sur Instagram afin de prédire les ventes (Matthieu Montaigu et Kasra Mansouri, Data Scientists @Artefact)
• La détection en temps réel des attaques de guichets automatiques bancaires par un système embarqué (Grégoire Martinon, DS @Quantmetry).
• L’acquisition et la labellisation d’un dataset par crowdsourcing ainsi que la détection dans un contexte à forte densité d’objets, (Augustin Rudigoz et Bruno Peyrou, @Mobeye App).

Meetup Computer Vision Paris was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here you are : your model is running and producing a first set of results. However they fall far from the top results you were expecting. You’re missing one crucial step : hyperparameter tuning!

In this post, we’ll go through a whole hyperparameter tuning pipeline step by step. Full code is available on Github.

What is hyperparameter tuning and why you should care

A machine learning model has two types of parameters:

• trainable parameters, which are learned by the algorithm during training. For instance, the weights of a neural network are trainable parameters.
• hyperparameters, which need to be set before launching the learning process. The learning rate or the number of units in a dense layer are hyperparameters.

Hyperparameters can be numerous even for small models. Tuning them can be a real brain teaser but worth the challenge: a good hyperparameter combination can highly improve your model’s performance. Here we’ll see that on a simple CNN model, it can help you gain 10% accuracy on the test set!

Thankfully, open-source libraries are available to automatically perform this step for you!

Tensorflow 2.0 and Keras Tuner

Tensorflow is a vastly used, open-source, machine learning library. In September 2019, Tensorflow 2.0 was released with major improvements, notably in user-friendliness. With this new version, Keras, a higher-level Python deep learning API, became Tensorflow’s main API.

Shortly after, the Keras team released Keras Tuner, a library to easily perform hyperparameter tuning with Tensorflow 2.0. This post will show how to use it with an application to object classification. It will also include a comparison of the different hyperparameter tuning methods available in the library.

Hyperparameter tuning with Keras Tuner

Before diving into the code, a bit of theory about Keras Tuner. How does it work?

First, a tuner is defined. Its role is to determine which hyperparameter combinations should be tested. The library search function performs the iteration loop, which evaluates a certain number of hyperparameter combinations. Evaluation is performed by computing the trained model’s accuracy on a held-out validation set.

Finally, the best hyperparameter combination in terms of validation accuracy can be tested on a held-out test set.

Getting started

Let’s get started! With this tutorial, you’ll have an end-to-end pipeline to tune a simple convolutional network’s hyperparameters for object classification on the CIFAR10 dataset.

Installation step

First, install Keras Tuner from your terminal:

pip install keras-tuner

You can now open your favorite IDE/text editor and start a Python script for the rest of the tutorial!

Dataset

This tutorial uses the CIFAR10 dataset. CIFAR10 is a common benchmarking dataset in computer vision. It contains 10 classes and is relatively small, with 60000 images. This size allows for a relatively short training time which we’ll take advantage of to perform multiple hyperparameter tuning iterations.

Load and pre-process data:

The tuner expects floats as inputs, and the division by 255 is a data normalization step.

Model definition

Here, we’ll experiment with a simple convolutional model to classify each image into one of the 10 available classes.

Each input image will go through two convolutional blocks (2 convolution layers followed by a pooling layer) and a dropout layer for regularization purposes. Finally, each output is flattened and goes through a dense layer that classify the image into one of the 10 classes.

In Keras, this model can be defined as below :

Search Space definition

To perform hyperparameter tuning, we need to define the search space, that is to say which hyperparameters need to be optimized and in what range. Here, for this relatively small model, there are already 6 hyperparameters that can be tuned:

• the dropout rate for the three dropout layers
• the number of filters for the convolutional layers
• the number of units for the dense layer
• its activation function

In Keras Tuner, hyperparameters have a type (possibilities are Float, Int, Boolean, and Choice) and a unique name. Then, a set of options to help guide the search need to be set:

• a minimal, a maximal and a default value for the Float and the Int types
• a set of possible values for the Choice type
• optionally, a sampling method within linear, log or reversed log. Setting this parameter allows to add prior knowledge you might have about the tuned parameter. We’ll see in the next section how it can be used to tune the learning rate for instance
• optionally, a step value, i.e the minimal step between two hyperparameter values

For instance, to set the hyperparameter ‘number of filters’ you can use:

The dense layer has two hyperparameters, the number of units and the activation function:

Model Compilation

Then let’s move to model compilation, where other hyperparameters are also present…

…read the full article here

Hands on hyperparameter tuning with Keras Tuner was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Your team worked hard to build a Deep Learning model for a given task (let’s say: detecting bought products in a store thanks to Computer Vision). Good.

You then developed and deployed an API that integrates this model (let’s keep our example: self-checkout machines would call this API). Great!

The new product is working well and you feel like all the work is done.

But since the manager decided to install more self-checkout machines (I really like this example), users have started to complain about the huge latency that occurs each time they are scanning a product.

What can you do? Buy 10x faster — and 10x more expensive — GPUs? Ask data scientists to try reducing the depth of the model without degrading its accuracy?

Cheaper and easier solutions exist, as you will see in this article.

A basic API with a big dummy model

First of all, we’ll need a model with a long inference time to work with. Here is how I would do that with TensorFlow 2’s Keras API (if you’re not familiar with this Deep Learning framework, just step over this piece of code):

When testing the model on my GeForce RTX 2080 GPU, I measured an inference time of 303 ms. That’s what we can call a big model.

Now, we need a very simple API to serve our model, with only one route to ask for a prediction. A very standard API framework in Python is Flask. That’s the one I chose, along with a WSGI HTTP Server called Gunicorn. Our unique route parses the input from the request, calls the instantiated model on it and sends the output back to the user.

We can run our deep learning API with the command:

gunicorn wsgi:app

Okay, I can now send some random numbers to my API and it responds to me with some other random numbers. The question is: how fast?

Let’s load test our API

Read the full article on Sicara’s blog.

Optimize Response Time of your Machine Learning API in Production was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Quick reminder if you’re not familiar with the concept: we’re a deep tech company specialized in Computer Vision based in Paris. Every month we share our ten favorite AI-related articles (or other stuff). This is the digest of December 2019.

I know you’re probably not hungry after the Holidays but here’s the article’s menu: better and stronger deep fakes, socially and environmentally responsible data science, and Artificial General Intelligence. New decade, time to adapt, I’m kicking off with a meme instead of a comic. If you’re a data scientist under 30 years old and don’t get it, I’d love your feedback.

An AI-generated Christmas Song

I’m going to open up this Best Of AI with the song that has been stuck in my head for a few weeks. It’s a Christmas song and — brace yourself! — it’s not Mariah Carey. Interpretation is from a musician in Denmark. Lyrics are from a neural network.

Research scientist Janelle Shane trained a GPT-2 on 240 Christmas carols. Then she asked her model to produce a song about Rudolph the Red-Nosed Reindeer. The result is brilliantly terrible.

The interpretation is so pure that my family didn’t notice anything weird when I played this at Christmas dinner. Don’t hesitate to use this as background music while you keep reading!

What happened in the field in 2019?

2019 is over and I reckon it’s healthy to look back at what happened this year before jumping into a new year. Former Research Director at Netflix Xavier Amatriain can help you with that. His review gives a nice perspective of this year’s achievements and challenges in Machine Learning.

StyleGAN2: the revenge of deep fakes

Last year, Nvidia created a huge sensation when they introduced StyleGAN (the generative algorithm behind thispersondoesnotexist.com). Some people were like “awesome! we can do that now?” and others feared that it could be used in a very malicious way.

Apparently, it was not enough, so they just made StyleGAN2. They fixed some flaws of the first version (generated images often had water droplets on the background; now they don’t). They also added some psychedelic new features like a smooth latent space, which is responsible for the animation above.

You can find the full paper here and the code there. Here is a blog post explaining all the changes.

Deploy models to production without unfair bias

Machine Learning might be fun, but we have to keep in mind that it is not a game. The code we write influences people’s lives. Our algorithms learn from the example we provide them. As such, they are prone to replicate every unfair bias present in the data. For instance, gender or race can become a factor in credit decision or resume classification (even if it’s not an explicit input).

We want our algorithms to be better than us, not to reproduce our mistakes. Following this idea, Google released Fairness Indicators this month. It’s a suite of tools built on TensorFlow to help data scientists diagnose unfair biases in their model. A good step in the right direction!

Machine Unlearning

Just like people whose data is treated by AI algorithms have the right to know that they were not affected by unfair bias, they also have the right to ask for their data to be deleted. The problem is that any model trained with their data may have it memorized, and it would be incredibly expensive to re-train all your models every time a data instance is removed.

Researchers from the University of Toronto introduce a new way to train deep neural networks so that they can unlearn more easily when a data instance is removed from the training set. A great step towards total GDPR compliance, and it seems that it can also be used when some examples simply become irrelevant. You can find their article on arXiv.

What’s the environmental impact of your model?

Another responsibility as a data scientist is towards the environment. GPU computing has a big environmental impact: training state-of-the-art Deep Learning models now take up to years of computing time (of course they are distributed in many units, so we have them up-and-ready in a matter of days).

Following their recent paper on quantifying carbon emissions of Machine Learning, a team from Montreal published a website on which you can evaluate your own emissions in three clicks! The core idea is to include this information in future research papers so that we no longer ignore the environmental impact of our work.

For fun, I wanted to see what it took to train GPT-2 (the model that produced our beautiful Christmas song). One training process produces as much CO2 as a round trip between Paris and Los Angeles. You can only imagine the cost of the hyper-parameter tuning!

Solving differential equations with a neural network

To be honest, when I first read that there was now a neural network that can solve differential equations or calculate integrals, I didn’t care a bit. I just assumed this was something conventional solvers had been doing for decades, so why bother? Turns out I was wrong and it’s really a big deal: deterministic state of the art only reached 85% accuracy on function integration, but this new solution is close to 100%, with an inference time under one second!

The research paper is here, and here is an article in the MIT Technology Review that summarizes it very well.

NeurIPS 2019 Keynote: The future of Deep Learning According to Yoshua

It’s hard to imagine the best of AI this month without mentioning the NeurIPS conference. There were a lot of interesting talks on current and future challenges of Machine Learning. I can’t mention them all, so I’ll focus on the one that impressed me the most.

Founding father of Deep Learning Yoshua Bengio talked about the System 2 Deep Learning paradigm: what current Deep Learning is missing to match human intelligence; promising leads on how to face the challenges of compositionality, causality and out-of-distribution generalization. One of these leads is, of course, meta-learning, which has been a research interest at Sicara for some time now.

If you missed it, I can but strongly advise you to rectify this as soon as you get an hour of free time. If you don’t have an hour, the 12 first minutes may suffice as an introduction to these challenges that will surely shape the future of AI!

We need a better measure of intelligence than chess and Starcraft

As a scientist, what thrills me the most in the field of Machine Learning is Artificial General Intelligence (AGI). It refers to an AI that can learn any task that a human can. And it is all that current AI isn’t, as Yoshua Bengio explained.

This article is an interview with researcher François Chollet. This is about how we measure intelligence. And why we need to find better benchmarks than video games or board games, if we want to do more than designing AI that harness millions of examples and thousands of years of computing time to learn one specific task.

ObjectNet: the proof that you’re smarter than a CNN

A perfect example of the terrible generalization abilities of our Machine Learning algorithms was provided this month by MIT and IBM researchers. They spent three years designing ObjectNet. This dataset is like a parody of ImageNet, where objects are taken out of context or in odd positions, and shot at random angles. They used it to test object detectors trained on ImageNet, and — surprise! — the accuracy was cut in half. Here is the article on MIT News explaining everything.

I just love this. Seeing state-of-the-art algorithms trained for weeks on millions of images fail to recognize a hammer because it’s on a hand and not in a hand. It shows us how much progress we still have to make.

That’s it for December and subsequently for 2019. Now 2020 will be what you make it. Do you need data science services for your business? Do you want to apply for a data science job at Sicara? Feel free to contact us, we would be glad to welcome you in our Paris office.

The Best of AI Articles Published in December 2019 was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to the November edition of our best and favorite articles in AI that were published this month. We are a Paris-based company that does Agile data development.

This month, we spotted articles about AI that can identify who wrote each scene in Shakespeare’s Henry VIII, and teach non-native speakers how to pronounce English words! Let’s start, as usual, with the comic of the month:

1 — Predict Impact of a Song on our emotions

Machine Learning, Music and Emotions

In a recent article researchers describe how they trained machine-learning algorithms to predict what features in a song would impact people’s emotional responses.

They predicted brain and heart activities as well as physiological response using features based on music dynamics such as timbre, harmony, etc…

This work helps to understand how music affects human experience and has applications in music emotion recognition and neuroscience.

2 — A Mobile App to Improve English Pronunciation of Non-Native Speakers

How to improve your English pronunciation if — like me — you do not always understand why your sentence is wrongly enunciated? A startup used machine learning to tackle this challenge! Blue Canoe created a mobile app directing its users to repeat sentence prompts. Speech-recognition technology then analyzes the recordings and uses machine-learning models to point out the differences. When users spend 10 minutes per day on the app, personalized feedback informs students precisely how they mispronounced words. The startup started by digitizing a 20-year-old methodology called the Color Vowel System. Then, they hired linguists to listen to users’ recordings and tag the problems. Recordings are then used to improve machine-learning models.

3 — Drones to spot missing people

While California was last month the third state to frame Police use of facial recognition softwares, Police Scotland unveiled this month a new drone using computer vision to search missing and vulnerable people reported BBC. Its recognition software is lightweight enough to be used on a smartphone and uses an optical camera and a sensor detecting heat. Police Scotland’s air support unit detailed aspects of its drone to argue it will not be used to spy citizens: ”We’ll comply fully with all the human rights legislation — in fact a data protection impact assessment has been carried out and we review that yearly. Also, before we deploy we’ll use social media to tell the public this is what we’re doing. ”In addition, its blue light and the sound of its rotors are supposed to alert people of its presence, believes BBC.

4 — Video recognition by Facebook

PySlowFast — Facebook’s video recognition system — is now available on GitHub and its mechanisms explained in a preprint paper. The main intuition of this system is to reproduce primate’s eye cells. These cells are either functioning at low frequency and focusing on fine details either responding to swift changes. Transposed to this video’s recognition system: the video is treated at a low and at a higher temporal rate. The lower to recognize static areas and the higher to recognize dynamic areas. This model has been confronted to two popular datasets: DeepMind’s Kinetics-400 and Google’s AVA and achieved state-of-the-art results on both.

5 — Full Release of Controversial GPT-2 Text Generating AI

Last November 5, OpenAi finally released the largest version of its controversial model GPT-2, claiming they have not found “strong evidence of misuse so far”. GPT-2 is a deep learning model able to output credible text from a minimal prompt (demo here). This full version was not originally released last February because OpenAi was concerned it could be used to automatically produce Fake News (summary of the debate here). They motivated this late release by the following arguments:

• this model version has only a marginally greater “credibility score” compared to already released version (according to a survey by Cornell University).
• they acknowledge that “GPT-2 can be fine-tuned for misuse” but argue that “despite having low detection accuracy on synthetic outputs, ML-based detection methods can give experts reasonable suspicion that an actor is generating synthetic text”
• they conducted in-house detection research and developed a that has detection rates of ~95% for detecting 1.5B GPT-2-generated text. By releasing this version they aim “to aid the study of research into the detection of synthetic text, although this does let adversaries with access better evade detection”.

6 — French BERT (CamemBERT) now available!

The French version of BERT has been released on huggingface/transformers repo! BERT or Bidirectional Encoder Representations from Transformers is a method based on pre-training language representations which obtained state-of-the-art results on a wide array of Natural Language Processing tasks (Google explanations here). This French version has been trained on 138 GB of French text and is available both in Pytorch and Tensorflow 2. This release is the achievement of a collaboration between Facebook AI, INRIA and Sorbonne Université.

7 — Recommending Apps in Google Play Store

In an interesting blog post, Deepmind explained its approaches implementing recommendation algorithms for Google Play Store, in order to “help users discover personalized apps”. The first approach using LSTM (Neural Network used to treat sequences) has been replaced by Transformers, which improved the model performance, but also increased the training cost. Third and final solution was to implement “an efficient additive attention model that works for any combination of sequence features, while incurring low computational cost”. In addition, the blog post introduced recommendation bias problem and how they deal with it: “For instance, if app A is shown in the Play Store 10 times more than app B, it’s more likely to be installed by the user, and thus more likely to be recommended by our model”. They detailed refinements they introduced in re-ranking recommendations and optimizing for multiple objective, such as relevance, popularity, or personal preferences.

8 — Determine Who Wrote each Shakespeare’s Henry VIII. Scenes

A new approach on a century-lasting debate!

Some literary analysts believe that Shakespeare did not write his play Henry VIII alone but has been helped by John Fletcher, the writer who replaced him as playwright of the King’s Men after his dead. In the mid-nineteenth century, literary analyst James Spedding already proposed a division based on the use of eleven-syllable lines. In 1962, an influential analyst divided the play between Shakespeare and Fletcher based on their distinctive word choices, for example Fletcher’s uses of ye for you and ’em for them. And last month, Petr Plecháč of the Czech Academy of Sciences in Prague claimed he has studied the problem using machine learning to identify the authorship at a more accurate level (not only attributing scenes): “Our results highly support the canonical division of the play between William Shakespeare and John Fletcher proposed by James Spedding”.

9 — Increase Solar Panel Energy Production

A startup named Heliogen aims to increase solar panel energy production by using advanced computer vision software. Such technology’s impact should not be limited to increase energy production. By accurately aligning mirrors Heliogen expects to be able to reach temperatures over 1,000 degrees Celsius. Such high temperatures could be used for the industrial applications that currently account for roughly 75 percent of the energy demand through fossil fuel production. In addition, this technology could ultimately provide an alternative to gasoline for powering automobiles by “spliting carbon dioxide and water molecules to produce clean-burning fuels like hydrogen”, the article explains. This AI-backed technology could therefore be a step to successfully use solar energy in fields where still dependent to fossil fuel.

10 — Self-training with Noisy Student improves ImageNet classification

In an article submitted last November 11, three researchers explained how they obtained 87.4% top-1 accuracy on ImageNet, which is 1.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. ImageNet is a famous image database often used to measure the performance of image classification neural networks. To achieve this result, they first trained an EfficientNet on labeled ImageNet images and use it to label 300M unlabeled images (it creates pseudo-labels, as these labels are not the ground-truth but a prediction). This first EfficientNet is called the Teacher. Then they trained a larger EfficientNet — called the student — learning to classify both ImageNet and newly labeled images. They iterate this process by using the larger EfficientNet as Teacher, i.e to re-label the dataset of 300M unlabeled images. During the learning of the student, they injected noise such as data augmentation, dropout, stochastic depth to the student so that the student neural network is forced to learn harder from the pseudo labels. But during the pseudo-labelling of the 300M unlabelled images, the teacher is not noised so that the pseudo labels are as good as possible. These researchers stressed that the “main difference between [their] work and prior works is that [they] identify the importance of noise, and aggressively inject noise to make the student better”. The following results show this impact of noise on the network’s results:

Do you need data science services for your business? Do you want to apply for a data science job at Sicara? Feel free to contact us, we would be glad to welcome you in our Paris office

This article was originally published on Sicara’s blog:
https://www.sicara.ai/blog/11-2019-best-of-ai-november-2019

Read the October edition
Read the September edition
Read the July edition
Read the June edition

Some articles we recently published on our blog:

Face Detectors: Understand DSFD and the State-of-the-art Algorithms
Determine Your Network Hyper-parameters With Bayesian Optimization
Deep Learning Memory Usage and Pytorch Optimization Tricks

The Best of AI: New Articles Published This Month (November 2019) was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read the original article on Sicara’s blog here.

Welcome to the October edition of our best and favorite articles in AI that were published this month. We are a Paris-based company that does Agile data development. This month, we spotted articles about AI that can solve physics problems, paint portraits, judge criminals, play video games and even recognize smells! Let’s start, as usual, with the comic of the month:

1 — AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning

The DeepMind’s bot AlphaStar managed to enter the Grandmaster league in Starcraft II. This league is the highest of the seven ranked leagues of the game. The developers made three different versions of the agent play against real players on Battle.net. The most advanced version got ranked, on average, in the top 0.15% of all players.

By the way, if you want to develop your own Starcraft II bot, you can, just like DeepMind, use the official Blizzard’s API client that provides full external control of the game. If you want to take a look at the official research paper, together with DeepMind’s pseudocode, detailed architecture and datasets with game replays to train your bots, they are available here.

Read AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning — From DeepMind Blog.

2 — Solving Rubik’s Cube with a Robot Hand

OpenAI trained a robot hand that is capable of manipulating a Rubik’s cube. To be clear, this achievement is not really about solving the cube, but rather about developing an extremely dexterous agent able to perform interactions with the environment with a high degree of precision.

The whole training process took place in a simulated environment. The new method, called Automatic Domain Randomisation, generated harder and harder environments as the agent trained. For more challenge, the developers introduced various perturbations during the robustness tests with a real robot hand. My favorite one is a cute plush giraffe curiously poking the cube with the tip of its nose! Others include throwing a blanket on the robot hand or making it solve the cube while wearing a rubber glove.

Read Solving Rubik’s Cube with a Robot Hand — From OpenAI Blog.

3 — Loss landscape

Neural nets are ubiquitous. But what really happens inside of them? This remains a mystery even to their developers. This new project takes you on a journey to a mesmerizing world of weirdly satisfying loss landscapes. Some of the visualizations produced by the Loss Landscape project are:

• LR Coaster that lets you ride along the minimizer during the learning rate stress test,
• Sentinel that explores the optimization process of a convolutional net,
• WALTZ-RES that shows the difference between two ResNet networks, with and without skip connections,

and many more!

Visit Loss Landscape — By Javier Ideami.

4 — Turn Python Scripts into Beautiful ML Tools

Streamlit is a new open-source Python framework built for machine learning engineers. As the developers promise on their website, it is “The fastest way to build custom ML tools”. Using Streamlit, you can build sleek web apps to serve your models in just a few lines of Python code!

Here are the core principles of the framework:

• Scripts are awesome: every Streamlit app is a stateless Python script.
• No callbacks: every widget is a variable!
• Information reuse: data and computations are cached in Streamlit’s data store that lets it safely persist information.

Try it now and see for yourself!

Read Turn Python Scripts into Beautiful ML Tools — From Towards Data Science

5 — Can you make AI fairer than a judge? Play our courtroom algorithm game

COMPAS is an algorithm used in the US courts. It looks at the defendant’s criminal history and outputs a “risk score”. This score reflects how likely the person under trial is to become a recidivist.

It turned out that the algorithm is racially biased, even though the score doesn’t take race into account. This piece lets you tweak the algorithm’s parameters and make it fairer!

Read Can you make AI fairer than a judge? Play our courtroom algorithm game — By MIT Technology Review

6 — A neural net solves the three-body problem 100 million times faster

The three-body problem is a classic physics problem of calculating the trajectories of three bodies given their initial positions and velocities. The first specific version of this problem, formulated in the 17th century, involved calculating the motion of the Earth, the Sun and the Moon.

It turned out to be an extremely hard problem to solve, since the resulting dynamic system is chaotic, except for a small number of edge cases. So far, a closed-form solution to this problem has not been found. Therefore, the solutions are generally calculated numerically, requiring enormous computational resources.

Researchers from the University of Edinburgh trained a neural network on the solutions produced by the state-of-the-art solver named Brutus. As a result, this network is able to accurately predict the motion of three bodies up to 100 million times faster than the solver.

Read A neural net solves the three-body problem 100 million times faster — From MIT Technology Review

7 — Learning to Smell: Using Deep Learning to Predict the Olfactory Properties of Molecules

We’re no longer surprised by AI models that can see and hear things. But what about other senses? Google came up with a model that is able to figure out how different things smell by predicting smell descriptors from molecules. It can distinguish smells like vanilla, chocolate or citrus, but also more complicated ones such as spicy, beefy or creamy.

Further research in this area could make it possible to develop digital scents and to create molecules with completely new smells. It would also be incredibly useful to help those who can’t smell appreciate scents like everyone else.

Read Learning to Smell: Using Deep Learning to Predict the Olfactory Properties of Molecules — From Google AI Blog

8 — Unsupervised Doodling and Painting with Improved SPIRAL

You may already be familiar with generative adversarial networks that create photorealistic high-resolution images. Human drawings, on the other hand, are rarely photorealistic, and yet we’re able to tell what’s in the picture, which means that they somehow capture the “essence” of objects. This “essence” is a high-level representation that incorporates human knowledge and structure.

SPIRAL++ is a GAN framework that learns how to paint like a human artist. With a limited number of brush strokes and without supervision, the algorithm learns to draw objects that are clearly recognizable by humans. This article lets you click on any image painted by the generator network and see the whole process of its creation, stroke by stroke.

Read Unsupervised Doodling and Painting with Improved SPIRAL — By John F. J. Mellor et al.

9 — Visualizing Tensor Operations with Factor Graphs

Have you ever felt lost looking at some formula containing multidimensional tensor operations and trying to figure out what it does? You’re not alone. Tensor operations can be difficult to wrap your head around.

But don’t be discouraged! Here’s a beautiful technique — called factor graphs — that produces powerful visualizations and helps us understand what’s happening when we work with multi-dimensional arrays of data.

Read Visualizing Tensor Operations with Factor Graphs — From Rajat’s Blog

10 — Smiles beam and walls blush: Architecture meets AI at Microsoft

The Microsoft Research Artist in Residence program developed Ada, the first AI-powered pavilion that can sense our emotions and change its colors and lighting in response.

Named after Ada Lovelace, Ada is a two-story photo-luminescent structure created using cutting-edge fabrication techniques such as 3D digital knitting. It is able to pick up on our voice tones, choice of words and facial expressions and use that information to infer our mood in real time. Whether or not it actually understands our feelings, it sure looks fascinating!

Read Smiles beam and walls blush: Architecture meets AI at Microsoft — From Microsoft AI Blog

This article was originally published on Sicara’s blog: https://www.sicara.ai/blog/10-2019-best-of-ai-october-2019

Read the September edition
Read the July edition
Read the June edition

Some articles we recently published on our blog:

Face Detectors: Understand DSFD and the State-of-the-art Algorithms
Determine Your Network Hyper-parameters With Bayesian Optimization
Deep Learning Memory Usage and Pytorch Optimization Tricks

Thanks to Hugo L., Fatima K. and Raphaël M.

The Best of AI: New Articles Published This Month (October 2019) was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Mixed precision training and gradient checkpointing on a ResNet

Read the original article on Sicara’s blog here.

Shedding some light on the causes behind CUDA out of memory ERROR, and an example on how to reduce by 80% your memory footprint with a few lines of code in Pytorch

Understanding memory usage in deep learning models training

In this first part, I will explain how a deep learning models that use a few hundred MB for its parameters can crash a GPU with more than 10GB of memory during their training !

So where does this need for memory comes from? Below I present the two main high-level reasons why a deep learning training need to store information:

• information necessary to backpropagate the error (gradients of the activation w.r.t. the loss)
• information necessary to compute the gradient of the model parameters

Gradient descent

If there is one thing you should take out from this article, it is this:

As a rule of thumb, each layer with learnable parameters will need to store its input until the backward pass.

This means that every batchnorm, convolution, dense layer will store its input until it was able to compute the gradient of its parameters.

Backpropagation of the gradients and the chain rule

Now even some layer without any learnable parameters need to store some data! This is because we need to backpropagate the error back to the input and we do this thanks to the chain rule:

The culprit in this equation is the derivative of the input w.r.t the output. Depending on the layer, it will

• be dependent on the parameters of the layer (dense, convolution…)
• be dependent on nothing (sigmoid activation)
• be dependent on the values of the inputs: eg MaxPool, ReLU …

For example, if we take a ReLU activation layer, the minimum information we need is the sign of the input.

Different implementations can look like:

• We store the whole input layer
• We store a binary mask of the signs (that takes less memory)
• We check if the output is stored by the next layer. If so, we get the sign info from there and we don’t need to store additional data
• Maybe some other smart optimization I haven’t thought of…

Example with ResNet18

Now let’s take a closer look at a concrete example: The ResNet18!

Continue reading “Deep Learning Memory Usage and Pytorch Optimization Tricks”

References

Memory Usage

Why is so much memory needed for deep neural networks?

Gradient Checkpointing

• Fitting larger networks into memory.
• Explore Gradient-Checkpointing in PyTorch

Mixed precision training

Mixed-Precision Training of Deep Neural Networks | NVIDIA Developer Blog

Are you looking for Image Recognition Experts? Don’t hesitate to contact us!

Deep Learning Memory Usage and Pytorch optimization tricks was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read the full article on Sicara’s blog here.

Welcome to the September edition of our best and favorite articles in AI that were published this month. We are a Paris-based company that does Agile data development. This month, we spotted articles about AI surveillance, Deepfake, a documentary from the 60s and much more. Let’s kick off with the comic of the month:

1–1960 documentary on AI as seen in 1960

Let’s jump in 1960, we are ten years from HAL 9000 and the first personal computers but people are already thinking about the emergence of Artificial Intelligence. From the late 1950s to the early 1960s, newspapers were full of articles about it.

It’s during that time that “The Thinking Machine” first aired. Back then we were already thinking about recognizing handwriting, playing games (checkers) and telling stories. This movie feels surprisingly contemporary, I advise you to take a look at it.

Read 1960 documentary on AI as seen in 1960 — By Harry McCracken.

2 — TensorFlow 2.0 is released

Google announced the final version of Tensorflow 2.0. It provides a comprehensive ecosystem of tools for developers, companies, and researchers who want to push the state-of-the-art in machine learning and build scalable ML-powered applications.

This new version includes :

. Ability to run models from any device (from an iPhone to a server in the cloud)
. Up to 3x faster training performance
. Standard dataset interface
. Eager execution as default
. Tight integration of Keras
. Access to TensorFlow’s low-level API

Read TensorFlow 2.0 is released — By TensorFlow.

Notice that PyTorch 1.3 has been released the 10th of October.

3 — How to make technology work for society ?

A lot of articles warn us on the negative impact of Artificial Intelligence over employment. MIT has published a report which tries to answer an enlightening question: how does Artificial Intelligence may help American build better careers as technological changes occur?

It moves the debate describing opportunities to suppress low-skilled job. It also makes the distinction between productive innovations and innovation which only perform as good as humans on effortless jobs (they are called “so-so technology”). For instance, self-check-out at pharmacies or supermarkets are making a less interesting improvement than an efficient waste sorting.

It ends by proposing policies to help AI and human task force be an efficient combination rather than a fight for employment. Take a look at their blog post if you want to understand the point of view of the MIT on this subject and see their advice.

Read How to make technology work for society— From MIT news

4 — Detecting patient pain

As improvements are done in the field of operating machines only with our brain, a team of researcher from MIT and elsewhere has developed a system that detects patient pain by analyzing brain activity.

This huge progress could help doctors diagnose and treat pain in unconscious and non-communicative patients. The model, based on the hemoglobin oxygenation, also allows generating “personalized” submodels to fit our different perceptions. It has a 87% accuracy and may soon be used in hospital, according to Dr Lopez-Martinez.

Read Detecting patient pain— From MIT News

5 — Facebook and Microsoft join forces to fight Deepfake

In July we talked about the best and scariest example of AI-Enabled Deepfake. In September, Microsoft and Facebook have provided a huge open-source labelled dataset to allow collective brainpower to work on this new threat. They are also funding this project with more than $10 million.

Read Facebook and Microsoft join forces to fight Deepfake — From Facebook Blog

6 — US government overcomes European GDPR law

If you live in a European country, you must have heard about General Data Protection Regulation (GPDR), a law which aims to protect users privacy. It has significantly bolstered consumer rights.
The CLOUD Act, introduced last year, is now threatening this law. It allows the American government to retrieve any information saved in a US company datacenter.
More and more affairs show that US government uses this law to overcome the GDPR, threatening the link between Brussel and Washington.

From US government overcomes European GDPR law — By European Views

7 — Google Fined for targetting children

Did you know that one of the most popular YouTube channels in the US is a kid channel? During the past year, this type of channels had an exponential growth. This September YouTube has been fined $170m to settle allegations it collected children’s data without their parents’ consent.

It’s important to understand that the illegal harvesting of children’s data was “extremely lucrative” for Google. This case and the presence of paedophiles on the platform forced YouTube to stop the monetization of these videos and to block comment access.

This shows how the US wants to protect children from the persuasive strength of AI-based targeted ads.

Read Google Fined for targetting children— By Associated Press

8 — The Global Expansion of AI Surveillance

In 1984, when George Orwells wrote its “Big Brother” dystopia, the fear of the global surveillance of the population has been put in our brains.

This report, published by Carnegie, studies the evolution of AI Surveillance over 176 countries with questions such as which and how governments are using it. The number of countries using it is rapidly increasing, and according to the report, already 64 countries are using facial recognition.

Carnegie has summed up his report through 8 key-finding. Read The Global Expansion of AI Surveillance — From Carnegie

9 — New Google Multilingual Speech recognition

The vocal assistant market is still growing rapidly, some challenging points that this technology is yet to overcome are the size of the model, the need for a model for each language, and latency.

Google made improvements towards solving these issues with a new end to end model which is a single model to understand them all. The experimentation has been done in India, this model can understand 9 different languages with low latency and fewer parameters than other states of the art models.

The learning is done by providing labelled speech. To prevent bias from the unequal distribution of training data by language, the architecture has been built to separate different languages. It outperforms state of the art monolingual models.

Read New Google Multilingual Speech recognition — From Google AI blog

10 — Artificial Intelligence Can’t Think Without Polluting

As artificial intelligence growth, the amount of computation used to train models is also continuously getting bigger and bigger. Especially in leading companies which use a lot of energy to create their state of the art models.

Even if for now, AI does not seems to weight much of the global energy balance, we may start to think about its impact. Multiple solutions exist, like starting to measure power consumption and to reward efficiency instead of accuracy.

This article is linked to the 3rd article (How to make technology work for society), it may become more and more important to focus on impactful models and stop using our energy and our time on "so-so Technologies".

Read Artificial Intelligence Can’t Think Without Polluting— By April Glaser

The Best of AI: New Articles Published This Month (September 2019) was originally published in Sicara's blog on Medium, where people are continuing the conversation by highlighting and responding to this story.