Constructing Skeleton Picture
It took long to find a suitable and well-performed open source package to construct proper skeleton for input images.
We first had a look into the package using posenet implemented by pytorch. This package provides two usages, it can either run the image_demo.py, an inference on an input folder of images and outputs those images with the keypoints and skeleton overlayed, or we can run the web_cam_demo.py which uses OpenCV to capture images from a connected webcam. The result of web_cam_demo.py is overlayed with the keypoints and skeletons and rendered to the screen. For example, here is one of the outputs of the image_demo.py. Git repository: https://github.com/rwightman/posenet-python.
However, the accuracy of the package is not as good as our requirement. Out of 50 images within the folder, only 5 of them were output with a proper skeleton. Most of the results have an incomplete skeleton plotted on the picture. For example:
In addition to that, the package is likely to give an output with nothing plotted on the picture if it is given a picture of a person’s back, which appeared twice out of 57 input images. For example:
We then decided to do some research and find a better package to do the job. We found that the package openpose which outputs reasonable skeleton picture and a high accuracy. With the same 50 pictures we tested for posenet, this module correctly plotted 45 of the input pictures. Here are the success and failures.
As we can see, the module is affected by the flying loose clothes of the dancer. It incorrectly detected her left leg onto her clothes. Moreover, in the third picture, dancer showed a very professional and difficult move, the module is not capable of plotting the skeleton correctly. However, users are not likely to be told to preform such move, and the second skeleton picture is actually not far away from the correct skeleton. Therefore, we have found the module that suits our project. Git repository: https://github.com/CMU-Perceptual-Computing-Lab/openpose.
However, there are places to be improved. First, the module is relatively difficult to install and compile. A number of software and packages are required to be install before running the module. The other disadvantages is that, the cpu version of the module takes about five seconds to produce skeleton picture, it is too slow for our project.
