How to stop touching your face? Hands and face detection with Python

When the hands get close to the face, there is an alert sound

Introduction

I usually write about business development, marketing, and growth hacking. This post is different. Well, our life became very different too…

The intention is to illustrate a (very) simple way to utilize AI and ML to detect hands and face. I’ll be using already existing models and libraries, so there is no training needed.

The library that I ended up using for face detection (cvlib) can also detect 80 common objects in context. So it can be useful for other applications.

The full code is available here: https://github.com/borismay/dont_touch_your_face

If you recall, there was a very viral post: Snagging Parking Spaces with Mask R-CNN and Python, I used pretty much the same approach.

It runs on my laptop with i7 CPU, without GPU, and as you can see, it is pretty close to real-time. So one can port it to other platforms, or use it at home to develop healthy and life-saving habits :-)

You’ll be amazed by how many times you touch your face!

So, stay safe and healthy and let’s see how it works.

Finding the right classifier

For face detection, I tested several existing libraries.

You can refer to detectors.py that implements all the tested detectors. There is also a Streamlit UI to fine-tune the detection parameters. Everything is available in the Github repository.

Streamlit UI to fine-tune various detection parameters

Here is the verdict:

1. cv2.CascadeClassifier — this is an embedded classifier that comes with the Open-CV library. There are multiple parameters that can be configured, not very well explained in the documentation. Fortunately, there is a StackOverflow post that can walk you through. The overall performance is not sufficient. It only works when you look straight to the camera
2. face_recognition — a package that is built on top of the dlib. There are 2 detection modes: ‘hog’ and ‘cnn’. It provides reasonable results with ‘cnn’, however on my simple hardware it was very time consuming
3. And the winner is… cvlib — Underneath, cvlib is using an AlexNet-like model trained on the Adience dataset by Gil Levi and Tal Hassner for their CVPR 2015 paper. It produced very robust results and fast performance. In addition to face detection, it can detect 80 common objects in context with a single line of code. For objects detection, it uses YOLOv3 model trained on COCO dataset

Finding a reliable hand detection library was a little bit more challenging. I used Victor Dibia's work described in this post: How to Build a Real-time Hand-Detector using Neural Networks (SSD) on Tensorflow (2017), GitHub repository, https://github.com/victordibia/handtracking.

Putting it all together

The flow is the following:

1. Configure the web camera and initialize its parameters
2. Initialize the detectors
3. Infinite loop:
   read the image
   detect face and hands
   if they’re in close proximity, warn the user

Webcam

Creating the webcam object, and configure the image size:

cap = cv2.VideoCapture(0)
cap.set(3, 1280/2)
cap.set(4, 1024/2)

The webcam’s parameters’ description can be found here.

Capturing a video frame is pretty simple too:

ret, frame = cap.read()

However, the image from the camera is in BGR format. In order to feed it to the classifier, you need to convert it to RGB:

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

Face and hands detection

I defined a base class to provide a similar interface to all the detectors I’ll be using:

class Detector:
    detector_params = {}
    detector = None

    def __init__(self):
        pass

    def set_detector_params(self, params):
        self.detector_params = params

    def detect(self):
        pass

Each detector will inherit from this class and implement its methods, including the constructor. Such that, after creating a detector object it will have the same interface.

So, for example, this is the implementation of the FaceDetector, and how it will be called in the code:

class CVLibDetector(Detector):
    def __init__(self):
        self.detector = cv

    def detect(self, rgb_image):
        # returns an array of (top, right, bottom, left)
        objects, confidences = self.detector.detect_face(rgb_image)
        # change to an array of (x, y, w, h)
        return [(top, left, bottom - top, right - left) for (top, right, bottom, left) in objects]
FaceDetector = CVLibDetector()
face = FaceDetector.detect(rgb)

Face detection results

As I mentioned, I used How to Build a Real-time Hand-Detector using Neural Networks (SSD) on Tensorflow (2017), GitHub repository, https://github.com/victordibia/handtracking as a starting point. I needed to update a few lines of code, to make it work with TensorFlow 2.0 and without the GPU.

Detect the face touch

Since it is a 2D image, there is no trivial way to model the hands and face position in 3 dimensions.

I assumed that if the bounding boxes of the detected hands and face will intersect, it will be declared as a touch event.

In order to implement the detection, I used shapely library. It is a very useful library for geometry calculations, I use it a lot for geospatial analysis.

Each detected object is represented by the following parameters:

x, y, w, h = obj

I’ll turn each detected object into a polygon and then will calculate the intersection between the hands' polygon and the face polygon:

def obj_to_poly(obj):
    x, y, w, h = obj
    return Polygon([(x, y), (x+w, y), (x+w, y+h), (x, y+h)])
def objects_touch(face, hands):
    if face and hands:
        face_poly = obj_to_poly(face[0])
        for hand in hands:
            hand_poly = obj_to_poly(hand)
            if face_poly.intersects(hand_poly):
                return True
    return False

If these objects intersect, the function will return True.

That’s it.

I spent a couple of hours during the weekend putting it all together, with a purpose to demonstrate how easy it can be to utilize machine learning and AI with python. There are plenty of already available trained libraries that enable us to build powerful applications with minimum effort.

And one more thing. Maybe this prototype can be used also to increase our self-awareness to touching the face. Download it, run it just for 30 minutes, you’ll see the result.

Stay home and keep your hands away from your face.