Smile Detector using Haarcascades

Facial expression detection always plays a crucial role in analyzing human emotions and behavior. It has always been an easy task for humans, but achieving the same with a computer algorithm is slightly challenging. Smile detection is a special task in facial expression analysis with various potential applications such as photo selection, user experience analysis, smiling payment and patient monitoring. In this blog, we are going to build a smile detector using OpenCV and Haar cascades that take in a live feed from a webcam. The smile/happiness detector that we are going to implement would be a raw one. There may exist many better ways to implement it.

What you need to build a Smile Detector:

1. Visual Studio Code ( https://code.visualstudio.com/ ) or Anaconda Navigator ( https://docs.anaconda.com/anaconda/navigator/ )
2. OpenCV ( https://opencv.org/ )
3. Haar cascades (link provided in the next section)

About Haar cascades:

Haar cascades are algorithms that can detect objects in images regardless of their location and scale. It can also run in real-time, making it possible to detect objects in video streams. The algorithms use edge or line detection features to detect objects. These algorithms are stored in their respective XML files in the Haar cascade git repository. To know more about Haar cascades check out this link, OpenCV Haar Cascades.

For our smile detector we need two Haar cascade files:

1. haarcascade_frontalface_default.xml
2. haarcascade_smile.xml

Make sure to download these files from the Haar cascade git repository.

All set? Alright, now let’s dive into the code.

CODE:

Importing the libraries, loading the haar cascade files and capturing video from the webcam

For the smile detector, we only need the cv2 module to proceed with. After importing the module, we need to call the downloaded cascades. To call them we need to use a function called CascadeClassifier. Following that, we create an object ‘cap’ and use the VideoCapture function to capture videos from the webcam. The function only allows two arguments, 0 if the webcam is internal and 1 if it is external.

import cv2
faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
smileCascade = cv2.CascadeClassifier('haarcascade_smile.xml')
cap = cv2.VideoCapture(0)

Creating a while loop and detecting happiness

Now, since I have done most of the code inside a while loop, instead of dividing the code and copy-pasting in different code blocks, I have copy-pasted the entire while loop below. To explain it better, I have divided the code into six sections.

1. Sec 1: This code initiates an infinite loop (to be broken later by a break statement), where we have true and frame being defined as the cap.read(). Basically, true is a boolean regarding whether or not there was a return at all, at the frame is each frame that is returned. If there is no frame, you won't get an error, you will get None. Then, we define a new variable called gray and convert frame to gray using the cvtColor function. Notice that here we are using BGR2GRAY because OpenCV only reads colors as BGR (Blue Green Red).
2. Sec 2: The detectMultiScale function returns four tuple values: x, y, w and h, which are the coordinates of the rectangle that will detect the face. This function accepts 3 arguments: the grayscale frame, a scale factor that reduces the size of the image and the minimum number of accepted neighbor zones.
3. Sec 3: Now that we have our face coordinates, the next task would be to draw the rectangle. For that, a for loop is used with the 4 tuple values in face. Inside the for loop, the rectangle function is used to draw the rectangle of the detected face. The rectangle function accepts 5 arguments: the frame, coordinates of the upper left corner, coordinates of the lower right corner, color of the rectangle (I chose dark blue) and the thickness of the edges of the rectangle. Following that, I have also created the rectangle on the grayscale image and stored it in gray_temp.
4. Sec 4: This code is similar to the one with the faces. Here I have given the same arguments, scaleFactor as 1.3 and minNeighbors as 5 but you can experiment it with your model to get the perfect result.
5. Sec 5: Here, using the if loop I am checking if the person is smiling or not. If smiling, then I am displaying the text “Smiling” using the putText function. This function accepts 8 arguments: the frame, the text to be displayed, the position of the text, font style, font scale, color ( I chose red), the thickness of the text, and the line type.
6. Sec 6: Despite being a video stream, we still use imshow function to display the result. The if loop statement runs only once per frame. Basically, if we get a key, and that key is a ‘q’, we will exit the while loop with a break.

while True:
    true, frame = cap.read()
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) //Sec 1
    face = faceCascade.detectMultiScale(gray, scaleFactor=1.3, minNeighbors=5) //Sec 2
     for (x,y,w,h) in face:
        cv2.rectangle(frame,(x,y),(x+w,y+h),(0,0,204),2)
        gray_temp = gray[y:y+h, x:x+w] //Sec 3
        smile = smileCascade.detectMultiScale(gray_temp, scaleFactor= 1.3, minNeighbors=5) //Sec 4
        for i in smile:
            if len(smile)>1:
                cv2.putText(frame,"Smiling",(x,y-50),cv2.FONT_HERSHEY_PLAIN, 2,(255,0,0),3,cv2.LINE_AA) //Sec 5
    cv2.imshow('RESULT', frame)
    if cv2.waitKey(1) & 0xFF==ord('q'):
        break //Sec 6

Releasing the webcam

This code releases the webcam and closes all the imshow() windows.

cap.release()
cv2.destroyAllWindows()

Results:

After experimenting with the scaleFactor and minNeighbors a little bit, I was able to get the desired result. Sorry for the bad image quality 😅. I had to crop out my desk and phone 🤷🏾‍♂️.