Creating Sign Language Lens on Snapchat
In early 2021, Snap Inc. launched a feature called Hand Joint tracking as part of the Snapchat app. This means that when the user opens the Snapchat app and lifts their hand in front of the camera, the camera and detect all the fingers of the hand including the thumb and can track their progress as the user moves their hand.
Using this new feature, I decided to create lenses that could detect sign language. I did not intend to create a sign language translator. The primary goal was to create something fun and hopefully educational, inline with the overall idea of Snapchat as a very informal app. If the lens(es) are also used as a translator during a conversation, then that can be considered icing on the cake.
I wanted to work with just the letters for now as more complicated phrases involve motion and other complexities. Therefore I created a little over 10 lenses, each focusing on only 1 sign language alphabet. This makes for a clean and simple experience.
The response from the community has been fantastic, with each lens being viewed tens of thousands of times. I am glad that so many people got introduced to sign language through this.
Due to the success of the lenses, I decided to write this post to share my experience in others. I do so in the hope that others can take this forward and create more sign languages experiences. The more experiences there are out there, the more the community benefits :)
Please feel free to post in the comments if any of the instructions are confusing or clarifications are needed.
You are of course also welcome to try out these lenses if you like. You will need to install Snapchat on your phone. Once you do, create an account, then go and search “Adarsh Hasija”. You will see all the lenses.
Alternatively, you can also view the demoes of each of these lenses. They are available on YouTube:
https://youtube.com/playlist?list=PLRgLU2UYCK3B0ahloMyw9QatlAc5J53-P
Creating Sign Language Lenses
To create lenses, one has to first download Lens Studio. That is the tool that is used to design lenses.
In the samples section, select Hand Joints
When the project opens, look at the preview area on the right. Select the webcam option to access the live preview. Allow permission to access your camera if asked.
When the camera preview comes, put your right hand in front of it. In the preview, there should now be an outline around your hand. Now sign the letter A. Click the area on the screen where your hand is displayed with a border. In the logger area, in the bottom middle of the screen, some text will appear. These are the coordinates of every joint that was visible to the camera.
Copy this line and paste it in the HandGesturesController file below the comment stub: To add custom pos. Give it an appropriate name. In this case we will use the name ASL_A_RIGHT. according to which character you signed with your hand.
Now we need to obtain some graphic images to display. We have 2 scenarios to cover. The first is a hint image, which we show when the lens first loads, to tell the user what they have to do. The other is an image of the actual letter. This acts as confirmation that the user has done the sign correctly. Once you have the images, drag and drop them in the Textures subsection of the Resources section.
The hint icon should be a transparent icon. Like so:
The image of the actual letter should not be transparent. Something like this:
Now that we have all the necessary resources, its time to wire everything up.
Attach the letter (not the hint) to the palm. This means that when the user performs the sign successfully, the letter will appear over the palm of the hand on the screen. To do this, find the “Center” section in the objects pane.
Click the Plus on the top left and select Image Object
When it is created, open the inspector on the right and add the name of the image under texture, like so:
Now its time to modify the hint image and text. In the Object inspector, find the Hint Logo Image.
Right click and select Duplicate. Rename the new Object according to which sign language character you are working on.
Click this new object to view its settings in the inspector on the right. Change the details to add the image of the hint, like so:
Now go back to the left objects inspector and select the Hint Text option
This is where you can modify the instruction text that is displayed to the user. In the Inspector on the right, modify this text as per your wish. As an example, you can say: “Sign the letter A with your hand”
Set it up for the right hand
Now its time to setup the actual trigger. In the Objects inspector, go to the Hand Triggers section and select one of the triggers
Rename it to “Sign_Detection_Right”
In the inspector on the right, set it up for custom gesture detection and use the same name that we put in the HandGestureController file.
For Response on start, use the same name and add “_START” at the end.
For Response on end, add “_END” at the end.
Now return to the Objects inspector on the left and select the On Start option under “Sign_Detection_Right”
In the Inspector that opens on the right, replace all instances of “INDEX_FINGER_START” with “ASL_A_RIGHT_START”
Go back to the Object menu and select On End. Repeat the process. Replace all instance of “INDEX_FINGER_END” with “ASL_A_RIGHT_END”
Repeat for the left hand
Go back to the point above where you first previewed your right hand in the camera. Do the same for left hand. When you copy the coordinates from the Logger and paste it in the file, name it ASL_A_LEFT. Remember that the first code was named ASL_A_RIGHT.
Go back to where it says “Set it up for the right hand”. Repeat the same steps for the left hand. Remember to follow appropriate naming conventions. When setting up names, do not end it with “_RIGHT”. End it with “_LEFT”.
At the end your object inspector should look like this. There should be a “Sign_Detection_Right” and a “Sign_Detection_Left”
Now you can try out your lens. You can do so in the preview within Lens studio or send it to your device (view it within the Snapchat app on your device). It should look something like this:
