AIsaturdaysOgbomosho Week 10
This is a review of what we learnt today 7th of july, 2018 using Andrew Ng’s Deep Learning specialization course. To accelerate our learning, we watch the videos during the week and do a walk-through the code while solving the programming assignments during weekends.
Deep Learning models have so much flexibility and capacity that overfitting can be a serious problem if the training dataset is not big enough. Sure it does well on the training set, but the learned network doesn’t generalize to new examples (test set) that it has never seen!
Regularization reduces overfitting. There are several regularization techniques.
- L2 Regularization
- Dropout
- Others are Data Augmentation, early stopping etc
This week’s task is learning to use Regularization in deep learning models.
Using a 2D dataset from France’s past 10 games. The goal is to use a deep learning model to recommend positions where the France’s goal keeper should kick the ball so that the French team’s players can then hit it with their head.
Each dot corresponds to a position on the football field where a football player has hit the ball with his/her head after the French goal keeper has shot the ball from the left side of the football field.
- If the dot is blue, it means the French player managed to hit the ball with his/her head
- If the dot is red, it means the other team’s player hit the ball with their head
NON — REGULARIZED MODEL
We tried out a non-regularized model first.
parameters = model(train_X, train_Y)
print ("On the training set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)
The training accuracy was 94.8% while the test accuracy was 91.5%. This is more like the baseline model.
REGULARIZED MODEL
- L2 regularization: Computing the cost and back propagation using the L2 regularization:
parameters = model(train_X, train_Y, lambd = 0.7)
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)
The training accuracy was 93.8% while the test set accuracy increased to 93% better than the baseline model.
2) Dropout: is a widely used regularization technique that is specific to deep learning. It randomly shuts down some neurons in each iteration. The idea behind drop-out is that at each iteration, you train a different model that uses only a subset of your neurons. With dropout, your neurons thus become less sensitive to the activation of one other specific neuron, because that other neuron might be shut down at any time.
parameters = model(train_X, train_Y, keep_prob = 0.86, learning_rate = 0.3)print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)
The test accuracy increased again to 95% !
A common mistake when using dropout is to use it both in training and testing.
Observations from the assignment showed that:
L2 regularization and Dropout are two very effective regularization techniques.
L2 regularization makes the decision boundary smoother
Dropout should be used (randomly eliminate nodes) only in training.
Regularization hurts training set performance! This is because it limits the ability of the network to overfit to the training set. But since it ultimately gives better test accuracy, this helps the system.
AISaturdayOgbomosho wouldn’t have happened without fellow ambassadors Temiloluwa Ruth Afape, Adegoke Toluwani and Mhiz Adeola Lawal, our Partner Intel.
Thanks to our Ambassador Daniel Ajisafe for the write up.
A big Thanks to Nurture.AI for this amazing opportunity.
follow us on twitter
