Torchmeta: A Meta-Learning library for PyTorch
Torchmeta is a collection of extensions and data-loaders for few-shot learning & meta-learning in PyTorch. Torchmeta received the Best in Show award at the Global PyTorch Summer Hackathon 2019. The library is open-source, and you can try it with pip install torchmeta.
What is Meta-Learning?
If there is one lesson we have learned from deep learning, it is that large-scale models you can train on large datasets are capable of generalizing well. For some applications, such as image or speech recognition, these deep neural networks are now reaching super-human level of performance.
Now what happens when you don’t have access to a large amount of data? After all, unlike deep learning in its current form, humans are capable of learning to perform new tasks very quickly, and very efficiently from only a few demonstrations. Getting models that can perform well from only a handful of training examples is particularly challenging, and thus requires some level of prior knowledge on the task to solve, effectively giving a “head start” when learning something new. For example, this prior knowledge can be introduced in the model explicitly by a domain expert — e.g. through regularization, or architectural choices.
Alternatively, this prior knowledge can be acquired from past experience; this is the approach taken in meta-learning. In meta-learning, we are leveraging the experience from a number of different meta-training tasks, with the objective of improving the performance and learning efficiency (i.e. the number of training examples necessary) on new downstream tasks.
If you want to learn more about meta-learning, check out the Meta-Learning Tutorial at ICML 2019 by Chelsea Finn & Sergey Levine.
Creating the “Gym” of Meta-Learning
The motivation behind creating Torchmeta was to facilitate the evaluation of meta-learning algorithms on different datasets, with as little changes as possible. Its design is inspired by OpenAI Gym, which made reinforcement learning more accessible by providing a common interface for a wide range of environments. The adoption of Gym as a standard tool allowed most open-source projects to be agnostic to the choice of environment, and made testing on multiple environments seamless.
Similarly, Torchmeta introduces, under a unified interface, data-loaders for a wide range of standard few-shot classification and regression problems. As of version 1.3, the following datasets are available in Torchmeta:
- Few-shot regression (toy problems)
- Sine waves (Finn et al., 2017)
- Harmonic functions (Lacoste et al., 2018)
- Sinusoid & lines (Finn et al., 2018) - Few-shot classification (image classification)
- Omniglot (Lake et al., 2015, 2019)
- Mini-ImageNet (Vinyals et al., 2016, Ravi et al., 2017)
- Tiered-ImageNet (Ren et al., 2018)
- CIFAR-FS (Bertinetto et al., 2018)
- Fewshot-CIFAR100 (Oreshkin et al., 2018)
- Caltech-UCSD Birds (Hilliard et al., 2018, Chen et al., 2019)
- Double & Triple MNIST (Sun, 2019)
All these data-loaders are fully compatible with the PyTorch ecosystem, including PyTorch’s DataLoader and the torchvision package. They generate random batches of tasks based on the corresponding dataset, each of them containing a training and a test dataset — which is common practice in meta-learning. While you can have full control over how the data-loaders are defined, Torchmeta also includes helper functions for the most popular benchmarks, with useful defaults from the literature.
Modules for Meta-Learning
In addition to the data-loaders, Torchmeta also provides a thin extension of PyTorch’s nn.Module, called MetaModule, to simplify the implementation of certain meta-learning algorithms. These meta-modules leaves you the option to manually specify the parameters of your modules with full computational graphs. This allows you, for example, to backpropagate through an update of the parameters, which is a key ingredient of gradient-based meta-learning methods (Finn et al., 2017; Finn, 2018; Grant et al., 2018; Lee et al., 2019; Raghu et al., 2019) and in various hybrid methods (Rusu et al., 2018; Zintgraf et al., 2019).
By default (i.e. with no extra parameter), the meta-modules behave identically to their module counterparts in PyTorch. Therefore the creation of models compatible with these meta-learning methods feels very natural with Torchmeta, and only requires minimal changes to your existing PyTorch models. You can also interleave meta-modules with standard nn.Module instances to enable fast adaptation of only some parts of the model (Raghu et al., 2019).
Improve reproducibility in Meta-Learning research
Reproducibility in meta-learning can be very challenging, especially when it comes to data loading, due to the lack of standard for some of the datasets used in the literature. For example, while the Mini-ImageNet dataset was introduced by Vinyals et al. (2016), the split used by Ravi & Larochelle (2017) is now widely accepted in the community as the official dataset. And this situation still exists as of today for some datasets (e.g. CUB). It can be hard to keep track of the “correct” version the meta-learning algorithms should be evaluated on.
With a number of few-shot learning and meta-learning datasets and standard splits available, the objective of Torchmeta is to provide all the necessary tools to make the development and the reproducibility of meta-learning algorithms as easy and accessible as possible.
Conclusion
The future is bright for meta-learning in PyTorch, and there are a number of great open-source projects that have been released lately. To mention two in particular, learn2learn provides implementations of some standard meta-learning algorithms, and higher is a library that enables higher-order optimization for existing PyTorch models. Torchmeta nicely complements these other libraries by providing a unified interface for a variety of datasets, and a set of tools to simplify the development of meta-learning algorithms. This makes the evaluation of these methods on different benchmarks seamless, and therefore is a crucial step towards better reproducible research in meta-learning.
To learn more about Torchmeta, check out the examples available in the repository of the project, as well as this implementation of MAML for a more detailed showcase of all the features of Torchmeta.
Finally I would like to thank the PyTorch team for organizing the Global PyTorch Summer Hackathon, and their incredible support; this was an amazing experience! I would like to also thank my wonderful colleagues at Mila for their support and feedback during the development of Torchmeta.
