Future of Web: Programmatic Headshot Cropping with Facial Recognition
Written and developed by: Patrick Organ and Tomasz Mieczkowski
Plenty of websites have profile pictures of their users. The problem that they all encounter, however, is how to automatically convert any picture into a proper profile picture (or headshot, in our case).
We have came up with a reasonable solution, but before we share it with you, allow me to walk you through how this process is normally taken care of:
- Resize the photo: if the photo came straight from the photographer, the dimensions might be astronomical, and there’s just no need for your website images to be larger than 1000px.
- Convert the photo: photos might come to you in so many formats, but for the purpose of a user image, it should be a JPG 99% of the time.
- Compress the photo: uncompressed photos may take up twice as much space and slow down the website load, while degrading the quality by 15% is nearly unnoticeable.
- Crop the photo: you may have desired proportions in mind, if it is too wide, cut the left side and right side evenly; too tall, and cut the top and bottom. Makes sense, right?
It’s all good when you have total control over every image uploaded to your website, but what happens if the images are uploaded by your user? Despite of providing your users with cropping functionality, soon enough you realize that a lot of pages end up with photos like this:
Facial recognition to the rescue!
In short, we’ve revamped the way images will be cropped in the future for all of your website templates using automated facial detection.
Finding the right algorithm for facial detection was not easy, and after considerable research we discovered the Face++ service. They do an amazing job by providing accurate data coupled with blazing fast responses. Using their service meant that our initial step was instantly complete.
The Face++ service is simple to use: provide a photo, and get back the location and size of the face. All that is fine and dandy, but where do we go now that we have all the parameters? Let’s get a little technical:
- Define cropping orientation: this is dependent on the thumbnail size. For this example, let’s assume you want square thumbnails (400px by 400px) so that the aspect ratio is 1:1 (aspect ratio is the proportional relationship between an image’s width and height). If the image is in landscape format (width is larger than height) at 800px by 600px, you’ll be looking at an aspect ratio of 4:3, which is greater than 1:1. This means you’ll be resizing to height and cropping to width. If an image is in portrait format (height is larger than width) at 600px by 800px, the aspect ratio is 3:4 — which is smaller than than 1:1. You will need to resize to width and crop to height.
- Resize the image: Once you know what orientation you have, let’s resize the image as per the instructions above. Let’s assume an image in portrait format for this entire example and go from there. First, we need to resize to width (if image was landscape, you would resize to height first). For this example, to retain proportion we have 600/400 = 800/x, giving you the x=400*800/600. As you see, image resizes to dimensions of 400px by 533px. This results in 133px of desired crop (533px — 400px = 133px).
- Calculate facial recognition deltas: Now let’s apply the facial recognition and then calculate the deltas. Since we are cropping height, we need to know the distance from the top of the face to the top of the image, and the distance from the bottom of the face to the bottom of the image. See code sample.
- Calculate the remaining offset and initial crop: If the delta of bottom offset to top offset (established in step 3) is greater than the desired crop (established in step 2), then the initial crop should be the difference between the two. See code sample.
- Calculate remaining crop: Our formula for portrait photos gives 130% tolerance towards the bottom of the crop for remaining offset. What this means is that we shift the face up by the factor of 30%. In some cases, you may want to keep the crop even. As you can see below, final results show that round photos look slightly better without the application of 130% tolerance
Ta-da! Is this an improvement over a picture of someone’s chin and chest? Absolutely. Can there be a better algorithm that would result with those thumbnails looking even more perfect? Of course, anything can be improved! The moral of the story is: the path of least resistance is not always the best path. Find a way to improve the web, even if it’s just one thumbnail at a time.
Google’s 2nd Commandment:
It’s best to do one thing really, really well.
Hey! Don’t think we would leave you hanging without an actual sample to play with. You deserve to have a little fun after all, right? Check out the website below for a full implementation of this example, or download the PHP library on GitHub.