Skeleton Detection Using a Video
How we want to extract emotions from a dancer’s move
We are Jérôme and Dylan, two students at the School of Engineering and Architecture of Fribourg in Switzerland. Our study plan include two semester projects and for the first one we chose to work on a skeleton detection application in collaboration with the HumanTech Institute.
What are our project goals ?
This skeleton detection application has many usage but the main goal was to create a web application to extract the skeleton from a dancer and then store it into a database. In a second phase, we would like to detect the emotion that the dancer is expressing to automatically correlate some emotions by their dances.
During this project we reached the main goal, it means we developed a web application where the user may upload a video or a picture and then extract the skeleton. After the extraction process is done, the user has some choices :
- Download the data as JSON or as CSV
- Download the whole body or some parts
- Download the picture / video with the skeleton superimposed or only the skeleton on a black background.
How we managed to assemble everything ?
During the first part of the project, we analyzed existing technologies that we could use and we chose to use ReactJS for the front-end. For the back-end we chose to use Docker with 3 containers. The processing container uses Flask and OpenPose. OpenPose is a real-time multi-person system to detect human body, hands, face and feet. The storage container is a MongoDB NoSQL database. And we have a third container with Express to take a look at the database.
The limitations of our skeleton detection application
We encountered some issues while developing this application. One of the issues was the speed of the extraction. OpenPose provides a demo and we can use it in real time with our webcam. But with our project, extracting the skeleton of a video could take up to 4h. We were surprised to see the time difference between the demo and our application. Later we found out the demo is coupled with TensorFlow (a machine learning tool) and the precision criteria is lower on the demo than in our application. In order to speed up the execution for our application, we should use TensorFlow or cut the video in small clips with a reduced quality and extract the skeleton in parallel and then reassemble all the small clips into a video.
What about the future ?
This project is an open door to many other projects. We took time to think about the future and we ended up finding many ideas. We could use this project to help a chiropractor to follow the progress of their patients while walking. We could virtually add some colors to each fingers of the dancer and create art. We could create a game where the player has to reproduce some emotion in order to win points… the possibilities are infinite.
A part of the project is also the detection of the emotions, we didn’t reach this goal but we proposed a solution. We could extract the skeleton of five clips where the dancer express happiness and then when we extract the skeleton, we check if there are any occurrences in our database; if there are occurrences then we detect the emotion, otherwise we ask the user which emotion was expressed in the dance.
A self-reflection about our work
At the end of the schedule, we are satisfied because we reached all of our milestones in time and we reached the main goal with providing an application to extract the skeleton from a video or a picture, store the data, download the data and the results. The group cohesion was great and even with the COVID-19 limitations we could cooperate and help each other.
Thank you for reading our article and we hope we created interest in skeleton and emotion detection !
