The Alias of Your Dreams

Become your favorite actor or your crush’s fantasy and send them a pic! Clone the repo or get the first Beta release and try it out for yourself!

openFrameworks

The app is built on a framework called openFrameworks. It’s a C++ framework designed for creative coding, so you can develop really awesome and visually intriguing apps, and with all of the addons available the possibilities are endless. The app is also based on our very own Grace Avery’s OS X app as featured in a previous blog post which achieves the same result, just on Macs.

The App

Building the app posed a couple of problems. The first and most apparent “problem” is that openFrameworks libraries are written in C++ and iOS apps are, primarily still, written in Objective-C. Luckily, there’s a way to combat the issue via Objective-C++. Essentially, the app is written both in Objective-C and C++ with both syntaxes mixed in to a single file in some cases. There’s some meddling around in the Build Settings required to get the app to compile correctly at first since you’ll get a bunch of type reference errors unless you use C++99 for the language dialect and libstdc++ standard library as oppose to the libc++ standard library. Secondly, ARC, or Automatic Reference Counting, isn’t a thing in the app. Garbage collection is all handled manually just like in the good ol’ days, so the app is sorta primitive in that sense. And also pointers. Pointers everywhere! They aren’t so much problems as they are a nuisance.

How Does This All Tie Together?

So enough with the nitty gritty… how does the app work? You can look at the video as 2 layers. The first (bottom) layer is what the camera is capturing (a.k.a. your face) and the second layer (top) is the face you have chosen to become. If you just slap the image you want on top of the bottom layer, it’ll be like plastering a poster onto your face, so we need a way to “cut” the face out of that poster. If we break that process up, we first need to actually find a face. That’s all handled through a Face Detection library, ofxFaceTracker, and a Computer Vision library,ofxOpenCV. The image (whatever face you chose) is converted into a matrix via ofxOpenCV and that matrix is fed into an ofxFaceTrackerinstance. Once we get that tracker set up we can get the image points of the face. This will allow us to create the mesh that will be our cut out face. For those interested in the code, here it is:

src.clear(); // src is the raw image src.loadImage(face); if(src.getWidth() > 0) { Mat cvImage = toCv(src); // create the point matrix srcTracker.update(cvImage); // update the face tracker with the new face srcPoints = srcTracker.getImagePoints(); // get the face points }

The same thing is done for your face on the bottom layer. For video face detection, we use the ofxFaceTrackerThreaded library to conduct calculations asynchronously to better performance and not bog down the system. The reason for using the threaded face tracker to get the points of your face on camera is that those points are constantly recalculated and found again almost every new frame of the video (sorry, can’t help dropped frames), so performing these operations asynchronously is crucial.

What you’re left with is two sets of face points that can be converted into a mesh. Each set of points represents a face whether it be your face or the selected face. The points in each mesh can be mapped to each other 1–1, so essentially it’s like taking a rubber mask and stretching it to cover your face while keeping the major features as close together as possible (in this case, the eyes, eyebrows, node, and mouth are the places with the highest resolution or most points). Once those points are glued to each other, the points on the top layer are completely controlled by the points generated by your face. That’s why when you open your mouth or move your eyebrows the top face also follows them.

The resulting mesh looks like this when mapped onto your face:

In-Depth Look

If you’re really interested in the nitty gritty code of the app, I definitely recommend cloning the repo or downloading the first beta release code. Everything regarding the processing of the face image and the camera face is found in the ofApp.cpp file in the Application group. Remember, you need to have downloaded openFrameworks along with the necessary addons (ofxOpenCV, ofxCV, and ofxFaceTracker) and then throw the code three levels below the root of the openFrameworks directory (it’s easiest to put the app folder in openFrameworks/apps/myApps/. That’ll ensure all of the OF libraries are found and that the project runs correctly. Also, another thing to note is that this project will only run on physical devices because of the use of the front facing camera. You’ll get unknown architecture errors if you try to use the simulator.

Lastly, face swap responsibly and have fun!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.