Google and Microsoft Open Sourced These Two Frameworks to Train Deep Learning Models at Scale
GPipe and PipeDream are new frameworks for high scale training in deep learning solutions.
Microsoft and Google have been actively working on new models for training deep neural networks. The result of that work has been the release of two new frameworks: Microsoft’s PipeDream and Google’s GPipe that follow similar principles to scale the training of deep learning models. Both projects have been detailed in respective research papers(PipeDream, GPipe) which I would try to summarize today.
Training is one of those areas of the lifecycle of deep learning programs that we don’t think of as challenging until the model’s hit certain scale. While training basic models during experimentation is relatively trivial, the complexity scales linearly with the quality and size of the model. For example, the winner of the 2014 ImageNet visual recognition challenge was GoogleNet, which achieved 74.8% top-1 accuracy with 4 million parameters, while just three years later, the winner of the 2017 ImageNet challenge went to Squeeze-and-Excitation Networks, which achieved 82.7% top-1 accuracy with 145.8 million (36x more) parameters. However, in the same period, GPU memory has only increased by a factor of ~3.
