Real-Time Sign Language Recognition System to Aid Specially Abled.
The Sign Language is a strategy for correspondence for hard of hearing — moronic individuals. The gesture-based communication is a significant technique for correspondence for hard of hearing specially-abled people. As communication via gestures is all around organized code signals, each motion has a significance allotted to it.
It can be utilized to communicate complex implications by joining fundamental components. Over the most recent quite a long while there has been an expanded enthusiasm among the specialists in the field of gesture-based communication acknowledgment to present methods for connection from human-human to human — PC cooperation.
Many research works related to Sign languages have done as the American Sign Language, the British Sign Language, the Japanese Sign Language, and so on. But very few works have been done in Sign Language Number Recognition to date. Tragically challenged individuals depend on communication through signing mediators for interchanges.
In any case, finding experienced and qualified mediators for their everyday undertakings all through life period is a troublesome undertaking and unreasonably expensive. Subsequently, the human — PC connection framework will end up being a dependable and steady answer for such people. There has been an across the board research around there completed over two decades.
Sign Language is a well-structured code gesture, where gesture has the meaning assigned to it. Finding an accomplished and qualified translator each time is an exceptionally troublesome assignment and excessively expensive. Besides, individuals who are not hard of hearing, never attempt to get familiar with the gesture-based communication for cooperating with the hard of hearing individuals.
That turns into a reason for detachment of the hard of hearing individuals. In any case, if the PC can be modified so that it can make an interpretation of gesture-based communication to the text group, the contrast between the ordinary individuals and the hard of the hearing network can be limited.
In the proposed framework that can perceive the different number i.e. till number 5, of Sign Language Number for Human-Computer communication giving progressively precise outcomes at any rate conceivable time. It won’t just advantage the almost senseless individuals of India yet additionally could be utilized in different applications in the innovation field.
How does the Proposed System work?
- Acquire the image from the Front Camera:
a. Cap = cv2.VideoCapture(0)b. Ret, IMG = cap.read()
2. Acquiring frame in real-time:
With the above two lines a & b, it will capture images in frames. Figure 1, below illustrates capturing a number two.
3. Image Preprocessing:
The processing of the image captures from the front camera, with the Gaussian blur creates the contour of the hand. Then process the convex_hull of the contour of the hand and get the convex defect of the image. Figure 2, shows the conversion of the hand image.
4. Transformation:
In simple words, convexity defect is a cavity in an object (blob, contour) segmented out from an image. That means an area that does not belong to the object but located inside of its outer boundary -convex hull. The image below, Figure 3, shows it better than millions of words. Areas between fingers (all marked with arrows) in this schematics of a hand contour are the convexity defects.
5. Blurring:
Image Blurring refers to making the image less clear or distinct. It is done with the help of various low pass filter kernels.
Gaussian Blurring: Gaussian blur is the result of blurring an image by a Gaussian function. It is a widely used effect in graphics software, typically to reduce image noise and reduce detail. It is also used as a preprocessing stage before applying our machine learning or deep learning models. Figure 4, shows the extraction of the biggest Contour by Gaussian obscuring.
6. Threshold:
For each pixel, similar edge esteem is applied. If the pixel esteem is littler than the limit, it is set to 0, else it is set to a most extreme worth. The capacity cv.threshold is utilized to apply the thresholding. The principal contention is the source picture, which ought to be a grayscale picture. The subsequent contention is the edge esteem which is utilized to characterize the pixel esteems. The third contention is the most extreme worth which is doled out to pixel esteems surpassing the edge, illustrated in Figure 5.
Implementation of Number_Indian Sign Language Recognition:
# Imports
import numpy as np
import cv2
import math# Open Camera
capture = cv2.VideoCapture(0)while capture.isOpened():# Capture frames from the camera
ret, frame = capture.read()# Get hand data from the rectangle sub window
cv2.rectangle(frame, (100, 100), (300, 300), (0, 255, 0), 0)
crop_image = frame[100:300, 100:300]# Apply Gaussian blur
blur = cv2.GaussianBlur(crop_image, (3, 3), 0)# Change color-space from BGR -> HSV
hsv = cv2.cvtColor(blur, cv2.COLOR_BGR2HSV)# Create a binary image with where white will be skin colors and rest is black
mask2 = cv2.inRange(hsv, np.array([2, 0, 0]), np.array([20, 255, 255]))# Kernel for morphological transformation
kernel = np.ones((5, 5))# Apply morphological transformations to filter out the background noise
dilation = cv2.dilate(mask2, kernel, iterations=1)
erosion = cv2.erode(dilation, kernel, iterations=1)# Apply Gaussian Blur and Threshold
filtered = cv2.GaussianBlur(erosion, (3, 3), 0)
ret, thresh = cv2.threshold(filtered, 127, 255, 0)# Show threshold image
cv2.imshow("Thresholded", thresh)# Find contours
contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)try:
# Find contour with maximum area
contour = max(contours, key=lambda x: cv2.contourArea(x))# Create bounding rectangle around the contour
x, y, w, h = cv2.boundingRect(contour)
cv2.rectangle(crop_image, (x, y), (x + w, y + h), (0, 0, 255), 0)# Find convex hull
hull = cv2.convexHull(contour)# Draw contour
drawing = np.zeros(crop_image.shape, np.uint8)
cv2.drawContours(drawing, [contour], -1, (0, 255, 0), 0)
cv2.drawContours(drawing, [hull], -1, (0, 0, 255), 0)# Find convexity defects
hull = cv2.convexHull(contour, returnPoints=False)
defects = cv2.convexityDefects(contour, hull)# Use cosine rule to find angle of the far point from the start and end point i.e. the convex points (the finger
# tips) for all defects
count_defects = 0for i in range(defects.shape[0]):
s, e, f, d = defects[i, 0]
start = tuple(contour[s][0])
end = tuple(contour[e][0])
far = tuple(contour[f][0])a = math.sqrt((end[0] - start[0]) ** 2 + (end[1] - start[1]) ** 2)
b = math.sqrt((far[0] - start[0]) ** 2 + (far[1] - start[1]) ** 2)
c = math.sqrt((end[0] - far[0]) ** 2 + (end[1] - far[1]) ** 2)
angle = (math.acos((b ** 2 + c ** 2 - a ** 2) / (2 * b * c)) * 180) / 3.14# if angle > 90 draw a circle at the far point
if angle <= 90:
count_defects += 1
cv2.circle(crop_image, far, 1, [0, 0, 255], -1)cv2.line(crop_image, start, end, [0, 255, 0], 2)# Print number of fingers
if count_defects == 0:
cv2.putText(frame, "ONE", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 2,(0,0,255),2)
elif count_defects == 1:
cv2.putText(frame, "TWO", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 2,(0,0,255), 2)
elif count_defects == 2:
cv2.putText(frame, "THREE", (5, 50), cv2.FONT_HERSHEY_SIMPLEX, 2,(0,0,255), 2)
elif count_defects == 3:
cv2.putText(frame, "FOUR", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 2,(0,0,255), 2)
elif count_defects == 4:
cv2.putText(frame, "FIVE", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 2,(0,0,255), 2)
else:
pass
except:
pass# Show required images
cv2.imshow("Gesture", frame)
all_image = np.hstack((drawing, crop_image))
cv2.imshow('Contours', all_image)# Close the camera if 'q' is pressed
if cv2.waitKey(1) == ord('q'):
breakcapture.release()
cv2.destroyAllWindows()
RESULT:
A communication via gestures acknowledgment framework proposed for human–PC cooperation utilizing Image Processing Technique was executed effectively with precision equivalent with those of ongoing commitments.
Hope this helps :)
Follow if you like my posts.
For more help, check my Github:- https://github.com/Afaf-Athar/Hand_Gesture_Number
More Refrences:
1. https://link.springer.com/article/10.1007/s11831-019-09384-2#:~:text=Sign%20language%20recognition%20is%20a,and%20to%20perceive%20their%20meaning.
2. https://link.springer.com/chapter/10.1007/978-3-319-38771-0_54
3. https://ieeexplore.ieee.org/abstract/document/6470093
