Building Emotional State Predictor using Deep Learning
In this post we will implement a neural network model which recognizes various emotions from facial expressions. The reader is assumed to be already familiar with the basics of Convolutional Neural Network.
1. Problem Statement
Given an image containing a single face, the aim is to build a predictive model to determine the emotion of the person from the facial expressions. The emotions identified belongs to these classes: Angry, Sad, Happy, Scared, Shocked and Disgust.
Assumptions: The image contains only one face. The image is gray scaled and is of the size (224,224).
2. Data and Preprocessing
Dataset is collected using Bulk Bing Image Downloader. Make sure to download at least 50 images per class for good prediction. For this blog post, I am using 59 images per class. The number of images per class should be equal to avoid Class Imbalance problem.
For a better performance, you should try raising the number of training examples per class. The data directory structure should be as follows:
where class_01 and class_02 etc. are class labels. Make sure to remove .pickle file from each class folders.
In case of multiple faces in a single image, Use the script below to detect faces in image and auto crop it. It would make the classifier accurate as it will not bother about features other than facial expressions. We use dlib and OpenCV to do this.
To select all the images, we use the glob module.
The first step of pre processing is to convert the image into grayscale. To do that, we use the following code.
Second step of the pre processing is image encoding. VGG16 network will encode the (224,224) sized image into (4096,1) sized vector. To implement the encoding, we use Keras and NumPy.
get_image_feature is the main function which takes in the image path, and returns the encoding of the image.
After retrieving the filenames using glob module, we call
get_image_feature and save the encoded training examples as
We do the same with validation images.
3. Understanding the Problem statement
The problem of emotion detection is a multi class classification problem. So, one hot encoding of the output labels is necessary as Softmax with categorical cross entropy will be used. We can use
np.eye for one hot encoding, but as this is an explanatory post, I am writing the encoding explicitly.
Now, training labels can be defined using this encoding. Each encoding has the shape (1,6)
In a similar way, validation labels is defined.
4. Designing the network
Now that input preprocessing is done, we are ready to design the network. The problem in hand is a multi class classification problem, so Softmax with cross entropy is used at readout layer. To prevent over fitting, Dropout layer can be used. The image shown below is the summary of the network used in building the classifier.
rmsprop is used although Adam optimizer is generally used.
5. Training the network
59 training images per class is very less and can lead to overfitting easily. Hence, early stopping is used while monitoring validation loss.
On running this code, we get the following output
Validation accuracy is just 33% but it is better than random guessing which is around 17%. The validation accuracy is low because of less training examples.
6. Visualizing the results
We use the following code to view the results in IPython notebook.
Running this code on the test images, we get really good predictions.
The same can be extended as a web application. I have made a webapp which recognizes the emotions Happiness and Sadness. Click here to view the demo of the same. Code of the same is available on Github.
Here are a couple of useful tips that may improve the performance of the model:
- Use more training examples
- Add Dropout layer to the network to combat overfitting.
- Explore more about early stopping
That’s all is required to train your emotion detector. If you don’t have a good GPU, do check out floydhub or other cloud GPU providers.
Thanks for reading! Stay tuned for more posts!
Originally published at sethuiyer.xyz.