Vidya G | VIT Chennai | December 2019
Introduction
Visual language is the natural way of communication used to aid the deaf, hard of hearing or mute, but very often they face educational, employment and social difficulties due to the language barrier with the general public. To facilitate a convenient and intelligent two-way communication,a gesture -based vocalizer that can translate visual language into speech which can be understood by all is built.
The flex sensors are fitted to a glove along each finger’s length and the thumb. The degree of bend generates a voltage variation at the output of the flex sensor. This output is given to the analog to digital converter of the micro-controller. The data is further processed and sent to the receiver where corresponding voice output is obtained.
While there are different visual methods for communication such as Cued speech, Sign supported English (SSE), Makaton, BSL (British sign language), ASL (American Sign Language, ISL (Indian Sign Language) etc. it becomes highly difficult for the disabled to interpret such variety in actions or signs.
In order to tackle this issue, we designed a GUI were custom symbols and corresponding meanings ie words/sentences can be saved by different communities based on their convenience. This system can also help the bed-ridden patients or physically challenged to be semi-independent.The vocalizer is also integrated with a GUI to accommodate the variety of sign languages that different communities use. The gTTS API supports several languages including English, Hindi, Tamil, French, German and many more. The speech can be delivered in any one of the two available audio speeds, fast or slow.
Field Survey
For the survey, we visited the C.S.I.Hr.Sec. school for the deaf located in Santhome, Chennai on February 20th, 2020. The students were articulate with the sign language. There are two sign languages followed, ASL(American sign language) and ISL(Indian sign language). Depending on what the student is taught and individual choice, they use either of those.
The teachers mentioned that most of the students form their own signs for communicating with friends and acquaintances, which emphasizes the importance of the device being customizable to suit their needs. Later, we met two teachers with hearing impairment, who grew up in different parts of the country and so one of them was convenient with ASL, while the other used ISL. ASL requires the use of both hands whereas ISL requires only one.
After having a conversation with these two teachers, the students, and the principal, we came to realize how important it is for the differently-abled to be able to customize and use the device to their ease and we decided to implement the same.
Architecture
The controller takes the input obtained from the flex sensors and the accelerometer which is then processed to APR 9600 which is a voice module through which voice output is obtained. The GUI takes in custom signs and presets its meaning as intimated by the designer.
Flex Sensors: The sensors are attached to the gloves along the fingers. This device is very similar to a potentiometer or a variable resistor. Resistance is caused due to the bending of the sensor which varies for different signs that are made by the glouse. The processed analog values obtained at the microcontroller are passed through the ADC channel after which they are compared with the preset threshold values for gesture recognition of a particular sign. It is then transmitted through the receiver module in a serial manner. Corresponding messages in the form of voice are obtained through the speaker.
Arduino Uno: The micro -controller board, Arduino Uno uses the ATmega328. It has 6 analog inputs, 14 digital I/O pins (of which 6 are PWM outputs), a crystal oscillator of 16 MHz, and a USB connection. It is low cost and contains everything that is required of a microcontroller. The processed analog values obtained at the microcontroller are compared with the preset threshold values for gesture recognition of a particular sign. It is then transmitted through the receiver module in a serial manner. Corresponding messages in the form of voice are obtained through the speaker.
Voice Synthesis: The voice module generates a signal output, which is then amplified with the use of amplifying circuitry and the output voice is given at the speaker. In the prototype version, the user makes a gesture with the glouse and holds it for 2 seconds to make sure its recognized. The system gives the voice output faster than the arbitrary 2 seconds limit.
Graphical User Interface: Graphical User Interface gives the user of this device the freedom to edit the word/sentence mapped to the gesture. GUI is designed using python libraries and made simple and minimal so anyone who has less idea about technology can also use it. In our GUI we have made provision to enter the phrases as well as to select the choice of language. Hence the product can work easily in different regions.
Google Text to Speech API :There are several APIs available to convert text to speech in python. One of such APIs is the Google Text to Speech API commonly known as the gTTS API. gTTS is a very easy to use tool which converts the text entered, into audio which can be saved as a mp3 file. The gTTS API supports several languages including English, Hindi, Tamil, French, German and many more. The speech can be delivered in any one of the two available audio speeds, fast or slow.
Encouraging results were obtained from the gesture vocalizer to rehabilitate the deaf/mute. It can also be extended for use in advanced telerobotic surgery and virtual interactive gaming.
Limitations of the System
• It is not possible to perform the same gesture exactly twice. The gestures in motion usually differ strongly while executing them in time and amplitude. Thus, the motion variances should be taken into account by the user.
• In mobile scenarios, like jogging or driving the system will not be able to give on the move interaction and users with non-gestural movements will have enormous effect on signals of acceleration.
•Wired hardware might also turn to be inconvenient when the user is in motion.
Future Work
- It can be applied in pragmatic reality for replacing the devices with traditional peripherals like keyboards and mouse with the hand glove.
- Machine activities can be controlled at sites that require assistance beyond the robotic control system.
- Commercial viability and portability can be achieved by making the system completely wireless.
- A light weight suit can be used to gathers signals from animals.
- It can also be used to aid patients with paralysis which would make them semi-independent.