Create Filters with PoseNet and ml5.js
What and Why ml5.js?
ml5.js aims to make machine learning approachable for a broad audience of artists, creative coders, and students. The library provides access to machine learning algorithms and models in the browser, building on top of TensorFlow.js with no other external dependencies.
What is PoseNet?
PoseNet is a machine learning model that allows for Real-time Human Pose Estimation.
Pose estimation :
Pose estimation is the task of using an ML model to estimate the pose of a person from an image or a video by estimating the spatial locations of key body joints
· Can be used to determine single or multiple poses
· One version of the algo can detect one person/pose and other version can detect multiple person/poses
· Why 2 versions you say ? the single person/pose version is faster
The first question that probably pops up in your head when we deal with any machine learning model is probably,
‘What are the inputs and outputs?’
The poseNet model expects an image as an input and as for output it will give you an array of x y co- ordinates and confidence score.
The co-ordinates are the key points in the poseNet skeleton
poseNet skeleton does not refer to an literal skeleton , haha but it actually refers to the 17 pose key points in the below image
Note : PoseNet works with Event handling
.on(‘pose’, …)
An event listener that returns the results when a pose is detected. You can use this with .singlePose() or .multiPose() or just listen for poses if you pass in a video into the constructor.
poseNet.on(‘pose’, callback);
look at the console output snippet
And that’s how you get the x,y co-ordinates of the pose key points. Few of the higher confidence value implies that those are the visible pose keypoints i.e., left & right eye,ear and Nose. The rest of the confidence values for elbow and hip and knee are very low because they were not in the image/video.
Create Filter :
After getting the co-ordinates , all that you have to do is
- get a transparent images as per your choice and position them accordingly
note : the filter size should change according to the distance between the camera and Screen. This can be carried out by calculating the distance between the left & right eye.
Checkout my github for the ‘black shades’ filter using poseNet & ml5.js
References :
