Indian Sign Language Recognition Using Machine Learning — part2
As we have seen in Part 1 what is Sign Language and how it can be used to make a life of people with hearing and speech disabilities easier and make them independent by providing online educational systems by using recent technologies like Machine Learning. Now let’s see how such systems can be developed in part 2.
Typography:
Sign language fonts of all sign languages are made by adding English letters of that language and hand gestures for that English letter which is called “Typography”.
Implementation Methods :
1. Sensor-based :
In sensor-based methods, data gloves or motion sensors extract the gestural data. This method requires wearing hand gloves with sensors. Such methods are not more comfortable for users.
2. Vision-based :
In a vision-based method, image processing is used for extracting gestures and this approach is comfortable for users. The images are captured from In vision-based methods, and image processing cameras, no extra devices are needed.
Steps Involved :
The steps involved in designing a sign language recognition system are as follows:
Gesture Acquisition and Gesture Recognition
The basic step of a Gesture Recognition system is to capture gestural data i.e. gesture acquisition. A laptop camera or mobile camera can be used to capture the images. To track the movement of people with hearing and speech disabilities, segmentation is required. Segmentation separates regions of interest (ROI) of the background. Tracking is a technique used to know the position of a gesture. Then the next step is feature extraction. Feature extraction is done to extract important features after the completion of tracking and segmentation. The whole process of recognition is divided into two stages, i.e., training and testing. Training is the initial stage where the classifier is trained using the training database. The steps involved in the training stage are the creation of the database, pre-processing, extracting features, and training.
Pre-Processing :
The pre-processing step has filtering, resizing, and segmentation of images. Commonly used segmentation techniques are the HSV colour model, Background subtraction, Otsu’s thresholding, skin colour-based segmentation, motion-based segmentation, Euclidean Distance, etc. The images or videos of the test database are also pre-processed during the testing phase to extract the region of interest.
Feature Extraction :
Feature Extraction is a very important step in sign language recognition. This step extracts important features after completing the tracking and segmentation steps. Feature extraction is mandatory for sign recognition performance. The most important decisions in designing sign language recognition systems are selecting appropriate features and feature extraction methods. The feature vectors are obtained in this step. These feature vectors are given as input to the next step, i.e., the classification step where these vectors are used to train the classifier. Common feature extraction techniques are Eigenvalues and Eigenvectors, Fourier Descriptors, HU Moments, etc.
Classification :
For the Classification of the signs which are given as input to the system, a classifier is used. Classification has two steps: the training phase and the testing phase. During the training phase, the feature vectors obtained in the feature extraction step are given to the classifier. The testing phase involves the identification of the class equivalent to the sign. The commonly used classifiers are SoftMax Classifier, ANN, SVM, Eigenvalue weighted Euclidean distance classification, OH (36 bins) with PCA at various distance-based classifiers, Mahalanobis Distance, Bayesian classifier, etc.
Machine learning algorithms for the classification:
- Support Vector Machine (SVM)
SVM classifier is a popular machine learning algorithm that is used for the classification of captured images. It is a popular pattern recognition technique with supervised learning.
- Artificial Neural Network (ANN)
ANN is a particular type of machine learning algorithm for training the system to work like the human brain. Applications of ANN are character recognition, speech recognition, signature classification, speech recognition, etc. A deep neural network (DNN) is also ANN. In most cases, ANN works as an adaptive system that changes its structure depending on external or internal information data which flows through the network during the learning phase.
- Convolutional Neural Network (CNN)
In deep learning, CNN is a part of DNN which can be used for image classification, image recognition, object detection, and face recognition. So, it is used for feature representation generation.
- Hidden Markov Model (HMM) and Finite State Machine (FSM)
HMM, and FSM is used for gesture recognition dynamically. It is a primary choice of gesture recognition technique. In dynamic gesture recognition, both are robust faster alternatives for the neural network.
Fig. 9 shows an example of every step of sign recognition. The vision-based approach is explained with examples very clearly.
Takeaway :
This article presents a review of the Indian Sign Language Recognition system as well as I contributed my own ideas based on my study and findings. More focus is given to Education-based online systems to remove the hurdle of the availability of learning institutes for people with hearing and speech disabilities. This application can be easily used by both deaf and non-deaf people. Non-deaf people can use it to learn sign language to interact with people with hearing and speech disabilities. For example Sign language interpreters, teachers of deaf and dumb learning institutes, and parents of deaf and dumb students. The dependency of people with hearing and speech disabilities on learning institutes can be reduced and the “LITERACY RATE” of people with hearing and speech disabilities can be increased.
Thank You for reading !!! 😊
Special thanks to Krupa for reviewing this article.
