SimGANs - a game changer in unsupervised learning, self driving cars, and more
Apple’s Learning from Simulated and Unsupervised Images through Adversarial Training (S+U Learning) lays down the blueprint for training state-of-the-art neural nets from only synthetic and unlabelled data. In this post we will see why this has huge potential, and apply it to an interesting problem: autonomous driving. We will refer to an implementation of SimGAN: https://github.com/wayaai/SimGAN, and to my 2nd favorite company behind waya.ai: comma.ai, throughout the post.
If a simple and intuitive introduction to generative adversarial networks (GANs) is needed:
The core idea behind Apple’s S+U learning is that labelled data is often scarce and expensive. However, synthetic labelled data can be generated using powerful engines like Unity or with other methods. The problem is that the gap between the synthetic and real datasets leads to a network trained on synthetic data not generalizing well enough to the real world. What S+U learning suggests is that with a real (and not necessarily labelled) dataset this gap can be bridged. A generator can learn to refine synthetic data such that it’s closer to the real dataset’s distribution while maintaining the synthetic data’s annotations (i.e. it’s labels remain valid) through adversarial training.
The following links will provide relevant code snippets for the concepts being explained. The refiner (generator) refines the synthetic data such that:
- Annotations are preserved by minimizing the difference between the refined and synthetic data with a self-regularization loss term.
- The synthetic data looks real (standard concept of GANs).
The refiner takes as input a synthetic data sample, and outputs a refined sample of the same dimension(s). The discriminator takes as input a data sample and classifies it as refined or real.
The GAN architecture and training procedure is standard, but S+U Learning suggests two simple and intuitive methods for improving the quality of generated data that applies to GANs in general.
- In current GAN architectures, discriminators are only trained on data generated by the most current generator. This leads to it forgetting features of previously generated data, and means the generator can reintroduce features to trick the discriminator. Generated data is generated data and should always be classified by the discriminator as such. A history buffer of previously generated data can be used so that the discriminator is trained on data from the current generator, along with data from past generators.
- In current GAN architectures, discriminators classify data as real/generated by analyzing the input data globally. This means the generator can get away with introducing some local artifacts in the generated data to trick the discriminator. Since any local patch of data should look real, the discriminator should instead break the data into local patches and classify each patch as real/generated, averaging these local adversarial losses for a more balanced global adversarial loss.
Applying this powerful approach in unsupervised learning to autonomous driving
Imagine you’re comma.ai,
and you have a large amount of real unlabelled driving data collected by Dash. While your current method of labelling data is awesome, it’s not quite cutting it, and you only have a small subset of labelled data. Using a SimGAN, you could train a refiner network to refine synthetic data from Grand Theft Auto
such that it looks like it came from the distribution of your real dataset and it’s annotations are preserved. Now you can train your production models on this nearly unlimited refined labelled dataset, and use your smaller real labelled dataset as validation/test.
I haven’t taken any of the self driving car courses out there, but I’ve seen that they use GTA and simulated environments to train their models. With a technique like this, their software could be closer to real world ready.
There seem to be many possible applications of SimGANs in the real world and autonomous driving is just an interesting one I chose to use as an example.
About Waya.ai
Waya.ai is a company whose vision is a world where medical conditions are addressed early on, in their infancy. This approach will shift the health-care industry from a constant fire-fight against symptoms to a preventative approach where root causes are addressed and fixed. Our first step to realize this vision is easy, accurate and available diagnosis. Our current focus is concussion diagnosis, recovery tracking & brain health monitoring. Please get in contact with me if this resonates with you!
