The Face
Final project in Creative Coding class
Display
Don’t forget to turn on your radio!
My code on P5.js(It needs some time to load, be patience and remember to turn on your camera.): https://editor.p5js.org/evaallen1997/sketches/B5towPhMF
Inspiration
From the proposal, My goal is to make a 3D face effect using the code. After several tries and referred to some online resources online, I found a package for face landmarks detection which can detect the face points using p5.js. Therefore, I wanted to make a likely 3D effect in a 2D setting.
Process
First, I got the image from Github(https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection) showing the face key points. There are 478 points detected on a person’s face where I could get all their X, Y positions. Then I first chose points and connected them in the vertical line to make a face mask in procreate. This step allows me to document the points in each lines, which will later be concatenate in each array in p5.
The next step is to connect them into p5.js. There are 49 lines of it and each line’s points were put in an array. I used for loop to draw every line one by one. This process needed a lot of patience as I always got errors during the process when connecting one point mistakenly.
Then, I wanted to add the shadow lines to make the mask more three-dimensional. I tested different versions for the lines. I decided to go with the third effect. I also added some afterimage.
For the background, I struggled a lot. I didn’t know how it could have a 3D effect. eventually, I decided to make a foreshortening effect using circles. Therefore, when the user gets closer to the camera, the background would be closer the camera as well and vice versa.
For the last step, I made a button called “stop capturing”. It would allow the camera to stop capturing the real-time image and show a static image based on the last-minute tracking. I stored the points when I clicked the button and then used those points to draw the static image.
Challenges
How to mimic a 3D effect in a 2D setting?
I used several methods to do this: First, I mapped the 3D face points on a 2D canvas, retaining same distance between points and leaving a sense of 3D space feeling. Secondly, I also made line shadow for the face, which would be cast with the face movements. When the user lift his/her heads more, then the line shadow will gets longer at the bottom of the face. The same situation when the head turn left, right or down. Thirdly, in order to make a 3D effect, I create a background that would scale based on the distance between the user and the camera. Therefore, it would create a sense of real physical space.
How to show a static image based on the last-second face track?
As I wanted to make a pause feature that will outline the last-second face when the user pauses the video, I first created a pause button. It would allow the user to pause the real-time tracking. Then I stored all the important data(ex. face points, distance between the face and the camera, shadow end points…) in the array. With those data, I drew the effect I want on the canvas again.
How to keep the code tide and efficient so that the detection speed could catch up with my actual moving speed?
Due to p5’s own capacity, It was not fast to run face detection with lots of data. Along the process, I need to be careful when using for or while loop. I learned to divide my work into parts and processes, so I could test whether it could work smoothly. I also learned to test a small part(like points on a line) first and adjust the effect based on that so that I would not spend too much time and find out it is not the effect I want in the end.
What I have learned…
- Test whenever it could be test. Test with the small parts and check the general effect as early as possible.
- Be patient with the bug. Check syntax, check logic, check the output and input.
- It’s really helpful to draw the diagram first to sort out the logic before writing the code.
Resources
- CP2: Face Mesh Detection in p5.js: https://www.youtube.com/watch?v=exrH7tvt3f4
- Face Tracking With P5JS: Part 2: https://www.youtube.com/watch?v=Agpnp1G7zTM
- Github face detection: https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection
In Creative Coding class, I…
In my first class, I was really nervous because I started as a newbie in p5.js and as well as coding. However, with the help and support from my classmates and Lucas, I was no longer panic when I ran into bugs and was able to tackle the problems alone by learning for solutions online. I developed the skills not only of how to write codes but also how to solve problems. I still remembered for the first homework, I couldn’t distinguish from plagiarism and inspirations in codes. Now, I totally understood the difference and knew the importance of citing the original authors. I had a lot fun in this course and explored the different fun effects using coding that I never imagined I could do before. Another lesson I learned in this class is the balance between user behaviors and automatic control. Sometimes, leaving a bank space for users is a fantastic way to create work creatively.
Overall, thanks to all my classmates. I always learned a lot from their fantastic works and working process. Thanks to my teacher — Lucas, who is always really patient and supportive. He teaches the course in a way really easy to be understood.