Shake-Shake regularization with Interactive Code [ Manual Back Prop with TF ]

GIf from this website

Shake shake regularizer is a simple yet powerful method to generalize our model, and additionally the implementation is very simple.

And as always lets train our model using different optimizers to see which gives us the best results on the CIFAR 10 data set. Below are the list of cases that we are going to implement.

Case a) Shake-Keep Model with Auto Differentiated Adam (Per Batch)
Case b) Shake-Keep Model with Auto Differentiated Adam (Per Image)
Case c) Shake-Shake Model with Manual AMSGrad (Per Batch+ No Res)
Case d) Shake-Shake Model with Manual AMSGrad (Per Image + No Res)

Residual Layers

Image from this website

Before reading on lets just review what a residual layer is, as seen above we do some mathematical operation from the given input. And…