Getting Started with ML5.js — Tutorial Part II: PoseNet
_________________________________________________________________
PSA: It is a key value for us at AIxDESIGN to open-source our work and research. The forced paywalls here have led us to stop using Medium so while you can still read the article below, future writings & resources will be published on other platforms. Learn more at aixdesign.co or come hang with us on any of our other channels. Hope to see you there 👋
Machine Learning/ML is changing the world, and it has gradually evolved into a new medium and design material for artists, designers, and creatives; however, algorithms and programming can become a great entry barrier for many to understand and explore the capability that ML offers.
ML5.js exists precisely for this purpose. With their entry-level interface, ML5.js made ML more approachable and easy to use, specifically with designers and artists in mind.
In November and December 2021, we hosted a series of ML5.js workshops with Erik Katerborg — a creative technologist, lecturer teaching Creative Media and Game Technologies at the Rotterdam University for Applied Sciences, and active AIxDesign community member.
To spread the word and invite more people to make their first steps into coding, we created this article as a written version of the workshop for anyone to follow at any moment in time. If video is more your thing, you can also watch a recording of the workshop on our Youtube channel here.
We have published Part I of this workshop series. Make sure to check it out here.
In Part II, we will look at how to build your own body detection game in the browser.
Let’s build with PoseNet!
The project that we will develop is using ML5.js’ PoseNet API to build a simple web interface where you can draw graphics using your body postures.
Now, let’s walk through the workshop tutorial. If you are in a rush, make sure to check out the finished project/code at the end.
Step 0. Setup
To start, go to the template that Erik has set up and click “Remix your own”
Similar to our first workshop, you will be able to see your own project in the web editor. A list of files is displayed on the left.
You can change the viewport layout, project name and turn off the “Refresh App on Changes.”
In index.html, we should be able to see some familiar code blocks from the Image Classifier workshop, like “message” to indicate the status of the project. Two new items to pay attention to are video and canvas.
<canvas id=”canvas” width=”640" height=”360"></canvas>
// Canvas provieds a way to draw in the browser
<video id="video" width="640" height="360" autoplay></video>.
// Video creates a region that displays the webcam video stream
These two HTML elements are shown in the Preview/Show panel. Now that we have seen how to access the webcam and the video footage, we can disable the video and only work with canvas. Edit the video line as below:
<video id="video" width="640" height="360" style = "display:none" autoplay></video>
Step 1. Understand script.js
Let’s select script.js from the left panel and understand what does each line represents.
In the variable section, we have the following elements:
const div = document.querySelector(“#message”)
// div contains the on-screen message that will be displayed const video = document.getElementById(“video”);
// video captures and displays the video stream from the webcamconst canvas = document.getElementById(“canvas”);
// canvas serves as a container that we would "draw" the camera feed in it later const ctx = canvas.getContext(“2d”);
// ctx is the drawing helper variable
The following two lines delineate the property of the style and line weight of the stroke. In this case, the stroke style will be red and the width will be 3.
ctx.strokeStyle = 'red';
ctx.lineWidth = 3;The next section is for accessing the camera with debugging code if there are multiple cameras. We do not need to fully understand or edit this section of the code.
The last section in script.js, function drawCameraIntoCanvas(), is where you define how to draw in the browser region. Right now, we are drawing the video stream into a 640x360 region. If your video is having a certain distortion, make sure to edit the width and height to display the video correctly. The last line in the function means we are calling the function 60 times per second, realizing an animation effect for the frames. Without the line, we are only drawing the video once.
Step 2. Load PoseNet Model from ML5.js
Now let’s head over to the “Quick Start” section for PoseNet on ML5.js https://learn.ml5js.org/#/reference/posenet and implement them step by step.
Because in the HTML file, we have already created a video element and accessed the webcam video stream. We can start from the second section and copy-paste it into our code. Also, to notify the user once the model is loaded, we added a line in the function modelLoaded(). So go ahead and copy and paste the following section in script.js.
// Create a new poseNet method
const poseNet = ml5.poseNet(video, modelLoaded);
// When the model is loaded
function modelLoaded() {
console.log(‘Model Loaded!’);
div.innerHTML = "PoseNet model loaded!"
}
If you refresh the preview panel, you should be able to see the text when the model is loaded.
Step 3. Create Poses
Next, we need to detect poses and create a variable to store all the poses that are generated in the video stream.
In the last section of “Quick Start”, PostNet will listen to events and store all the results in “poses”, which is a new variable we should create. In script.js, add the following line after “ctx.lineWidth = 3”.
Let poses = []Then copy and paste the last section of “Quick Start” below const poseNet. This way, model PoseNet will be loaded and your poses in the video will be detected and stored in “poses.” Your code should look like this.
Step 4. Draw Poses
In the function drawCameraIntoCanvas(), everything will run 60 times per second, creating an animation effect. This is where we would want to put our pose drawings. First, let’s see what is currently being detected by PoseNet. In this function, add
function drawCameraIntoCanvas() {
ctx.drawImage(video, 0, 0, 640, 360);
console.log(poses) window.requestAnimationFrame(drawCameraIntoCanvas);
}
Then, open the web console and see what is being detected. We can see once there’s a movement, PoseNet registers the positions and movement of your body. They are sequentially stored in the “poses” variable you created earlier.
ML5.js PoseNet page has provided some boilerplate for us to use. What we will use for this workshop is in this file.
Scroll down and copy and paste the entire function drawKeypoints()and function drawSkeleton() sections after function drawCameraIntoCanvas() in script.js. Your code should look like this.
These two functions can draw body keypoints and skeleton to the canvas based on the PoseNet generated poses. If we look closely at the drawKeypoints() function, you can see each pose’s keypoint can be drawn if the confidence score is high enough.
To call the newly added functions, add the following in the function drawCameraIntoCanvas(). This way, the drawing functions will be drawn 60 times every second.
drawKeypoints()
drawSkeleton()
Refresh the preview panel and look at how PoseNet is detecting your body position and movement!
Step 5. Draw with Poses
Now let’s experiment with drawing with poses!
First, in function drawCameraIntoCanvas(), comment out the following two lines. This hides your video and the drawn skeletons. Refresh the preview panel, you should be able to see no video but only your poses.
// ctx.drawImage(video, 0, 0, 640, 360);// drawSkeleton()
Then add in the following two lines, which can help us experiment with the drawing color.
ctx.fillStyle = ‘rgb(255,0,0)’;let redvalue = 0.
In the function drawKeypoints(), change ctx.stroke() to ctx.fill().
Add a few new lines in drawCameraIntoCanvas() function to achieve different visual effects. If you’ve done everything correctly, you should be able to see a shadowing effect that documents the trace of your body position and movements.
function drawCameraIntoCanvas() {
// Create shadowing effect
ctx.fillStyle = "rgba(255,255,255,0.05)";
// Create white background
ctx.rect (0,0,640,360);
ctx.fill();
// ctx.drawImage(video, 0, 0, 640, 360); redvalue++
if (redvalue > 255) redvalue = 0
ctx.fillStyle = `rgb(${redvalue},0,0)`
drawKeypoints() // drawSkeleton()
// console.log(poses) window.requestAnimationFrame(drawCameraIntoCanvas);
}
Last but not least, we can further edit the code to make PoseNet only traces our arms. Edit function drawKeypoints() to the following code and you should be able to see your arms’ movements only.
function drawKeypoints() {// Loop through all the poses detected
for (let i = 0; i < poses.length; i += 1) { // For each pose detected, loop through all the keypoints
// Define variables and access pose generated data
let leftWrist = poses[0].pose.leftWrist
let rightWrist = poses[0].pose.rightWrist if(poses[0].pose.leftWrist.confidence > 0.2){
ctx.beginPath();
ctx.arc(leftWrist.x, leftWrist.y, 10, 0, 2 * Math.PI);
ctx.fill();
}
if(poses[0].pose.rightWrist.confidence > 0.2){
ctx.beginPath();
ctx.arc(rightWrist.x, rightWrist.y, 10, 0, 2 * Math.PI);
ctx.fill();
}
}
}
Now with PoseNet and ML5.js, you can draw abstract images with your body, just like a digitized version of Jackson Pollock!
Ran Into Any Issues? No Worries.
You can always review the workshop video on Youtube: https://www.youtube.com/watch?v=BRs1xouy-k4
In addition, Erik has provided the finished file for the project that you can access at: https://glitch.com/~ml5-workshop-pose-finished.
Conclusion & Next Step
We hope you had fun and gained some new perspectives and ideas to bring back to your own practice. And of course, a big shout out to our brilliant host Erik Katerborg who showed us the potential and fun creative applications of doing Machine Learning in the browser using ML5.
We’d love to hear your key takeaways in a reply or a post if you’re willing to share! Please leave a like and share with your friends if you found the article helpful!
Links to Workshop Materials
To rewatch the workshop, please head over to our Youtube Channel where we have uploaded the recording:
This is Erik’s Glitch account where you can find all his projects: https://glitch.com/@KokoDoko.
Again, you can always access the finished project:
Click below to check out Part I of the ML5.js workshop series, which will show you how to use a machine learning model to classify an image.
Helpful Resources
Beginners guide to ML5 by the Coding Train — https://thecodingtrain.com/learning/ml5/
Made with Tensorflow.js Youtube channel — https://www.youtube.com/watch?v=h9i7d4R36Lw&list=PLQY2H8rRoyvzSZZuF0qJpoJxZR1NgzcZw
ML5.js documentation — https://learn.ml5js.org/
Interested in More Workshops?
Stay tuned to our Eventbrite page for upcoming workshops, keynotes, and networking events!
Thank you, Erik!
This tutorial is fully based on the workshop developed and hosted by Erik for AIxDesign in November & December 2021. So big shout out to Erik for this amazing work! Erik Katerborg is a creative technologist and lecturer in creative media at the Rotterdam University for Applied Sciences. He is especially interested in making technology and coding more accessible to people with a creative or design background.
To stay in touch, you can connect with Erik on Linkedin, Twitter, or Instagram.
About AIxDesign
AIxDesign is a place to unite practitioners and evolve practices at the intersection of AI/ML and design. We are currently organizing monthly virtual events (like this one), sharing content, exploring collaborative projects, and developing fruitful partnerships.
To stay in the loop, follow us on Instagram, Linkedin, or subscribe to our monthly newsletter to capture it all in your inbox. You can now also find us at aixdesign.co.
