Inside AI

HAND DETECTION AND FINGER COUNTING AI using Opencv python

How I build Real-time finger counting model from scratch

Introduction :

Hey you! If you are newborn family member of COMPUTER VISION and want to evolve and understand some cool opencv tools and stuffs. In this article we will be going from scratch to full understanding and implementation of opencv using python on finger counting project.

Code for this project can be found at my Github

Basics of opencv python are also available at my Github

I will break this process in certain steps for better understanding as below:

Step 1: Setting-up the avg. background values in region of interest.

Step 2: Apply blurring and Thresholding on hand entered in ROI to find contours.

Step 3:Use technique known as Convex hull to grab polygon around it along its vertices (x,y).

Step 4: Using some mathematics to count no. of finger raised. I will be going to explain math related stuffs later on.

Step 5: Put all stuff together to build our project

Step 1:

In computer vision projects we are more concern about an object in an image or video of our interest.Background should not create noise or distract our machine to give wrong and unexpected results.In this project we are only concern about our hand entering in ROI. To get segmented out our hand we will turn all the pixels to 0(zero) except our hand pixels.This can be achieve by taking abs. difference of our calculated avg. background image with current frame/image.

In above code we first grab a single frame to select our ROI using cv2.selectROI(image) which returns x,y,width,height of our ROI. Then for next 60 frames we modify each pixels of back to become weighted average of each frame. This can done using cv2.accumulateWeighted(), which takes parameter like current frame,backgound frame(back),learning rate.

Part 2:

This part deals with image pre-processing. As our hand is now get segmented, we should apply blurring and Thresholding. Doing this we can now easily find boundaries of hand.

After getting background image(back),we must apply cv2.convertScaleAbs() It generally Scales, calculates absolute values, and converts the result to 8-bit. Then convert background image(back) and current frame into Grayscale images.Take absolute difference of grayscale images .Apply Thresholding using cv2.threshold(image,min_thresh,max_thresh,method_of_thresholding.). Now our image is prefect to get its external contours . It returns contour(list of points(co-ordinate) of boundary),their hierarchy. Take those list of points which covers maximum area(probably it must be our hand segment’s boundary co-ordinates).I’m creating a copy of img ,you will get it in next step(it’s just for visualization purpose.)

Part 3:

Convex hull draws a polygon by connecting points around most external points in a frame. First we will calculate the most extreme points(top,bottom,left and right). Then we can calculate their intersection and that must be center of our hand. Next we will calculate the distance for the point furthest away from our calculated center.Then using some ratio(depends hand to hand) of that distance we create a circle. Any point outside the circle and far away enough from the bottom ,should be extended fingers. For more mathematics behind this check this.

In this code, first call a function cv2.convexHull(contour) and assign it to a variable name conv_hull. Then we grab the most extreme points(top,bottom,left and right).It’s just a simple co-ordinate geometry.With the help of this points, grab hand center(cx,cy).Now we can estimate the value of radius(radi). Then create circular_roi of shape same as img of all pixels of value zeros.Draw a circle of our desired radii on it.Then do bitwise_and() operation between img2 and circular_roi and store it as mask.

part 4:

Now its time to count finger raised. For doing so, we first grab contours again of mask image and find co-ordinates of bounding box and apply some condition on each box to count weather it is a contour of finger or wrist or something else.

As shown above. We grab contours.Made a variable named circumference. Then we loop for each bounding box/rectangle found by contour. Calculate its orientation(m_x,m_y,m_w,m_h) and then apply two condition on it.That is out_wrist_range and limit_pts. We want to make sure that contour region is not at very bottom of the hand, if the person has lot of arm in ROI we may get some contour points at bottom and no useful for us. We can avoid this by simple math. We can say the boxes which are far bellow from center of hand it must be a contour of our arm and we can neglect it .Therefore the bounded box which are say 25% above center are part of finger raised. The second condition we can make sure of is that the no. of points along contour does not exceed 25% of the circumference of circular ROI. This is because machine can count certain points which are above center but not of fingers. This may because of some noise some external contours may pop up. So we define some limit(limit_pts) for this purpose.

Part 5:

Its time to combine all together what we have learned so far.

In above code we grab first frame for selecting our ROI. After having our ROI we calculate weighted avg. of first 60 frames and then took absolute difference background image and current frame in every iterations while reading from webcam and assign it to new variable(img). Take threshold of grayscale img. Later we find extreme points(co-ordinate) of polygon using conver hull function by finding contours first. Tacking intersection of those points we get center points of our hand. Apply bitwise_and() operation between thresholded image and circular_roi having circle of some radii. If hand size is small make the radii small and vise-versa. Again grab the contour boxes and apply conditions for counting fingers raised. For visualization purpose we mark selected ROI’s rectangle on frame which cap object is reading. Also we show mask and weighted image to recheck our logic and code works properly.

The final output should look like this . Hope you enjoy this project.