Fun with Sitecore and face recognition (like in Facebook)

Jack Spektor
Aug 28, 2017 · 5 min read

Around a month ago I’ve seen a post by @markstiles about integration between the Sitecore and Microsoft Cognitive Services. I was really impressed by how much intellegent you can make your sitecore build.

So, I thought — ok, there is one thing that is not covered here — face recognition. Microsoft provides a great API which I’m going to use in this article to show how to add support for Facebook-like face recognition in your website.

Here is brief steps that show how it was done to achieve face recognition of the Sitecore media library. Note this is only a brief notes, but for the full source code it’s best to look into the GitHub repository.

If you are not interested in implementation notes you can skip this part. I’m planning to release the module as a Sitecore package and will cover how to configure it in my next article. So stay tuned!


Step 1: Detect faces

First we need to detect faces on the photos. For this I’ve decided to extend the Image template in Sitecore to store information about the detected faces.

Then when we upload a new photo we’ll store it in the media library and invoke the pipeline that will do some face recognition magic.

To detect faces on the image we’ll call the “Detect” method of the FaceAPI on the image.

Step 2: Let’s create a repository of persons to identify

Azure Face API has the concept of “persons” and “person group”. Each person has faces that are assigned to it. This is how Face API knows how to identify the person. But to train the Face API to use these images we need to train the person group that this person is related to.

We’ll create the person and person group hierarchy in Sitecore, but will sync it with the Face API using item:saved handler.

Now we have the information about our detection candidates in Azure Face API database. But we still don’t have any idea how they look like.

Step 3: Provide Face API with person faces information

Now as we have the information about faces on the image (as well as information about detection candidates), we can use it to let the users to identify the users on the photo. In this way the users themselves will tell Face API how to know how to identify the face on the photo.

For this we’ll implement the handy JS code which will show the tag on the person and will handle the click event that will allow users to select who is on the photo.

A little bit of test data…

Hooray! Now we have all the data to actually identify people on the photos.

What’s next?

Step 4: Train the FaceAPI using our data

Now we have all the data. But our FaceAPI data is not trained yet. For this I’ve created a custom command that I will call from Sitecore ribbon.

This will make a call to the FaceAPI and call the “Train” operation on the PersonGroup.

Now our genious FaceAPI is trained to identify people, so it’s better go use it!

Step 5: Implement the search index that will identify the person using Face API

FaceAPI has it’s own usage limits, so it’s best to make as little calls to the API as we could. For this we’ll create a custom search index that will store the information about the identified persons on the photo.

To identify the face we’ll call the “Identify” operation on the face.

Basically the index will go through the each photo in the media library and will call the FaceAPI to identify all the faces on the photo.

Last step: Train the FaceAPI and show the info on the tag

Now we have almost everything in place. We have the data about identified persons in the index. So the last thing we should do is to actually show it on the image.

For this we’ll create a handy HtmlHelper extension that will find data about the image in the index and then it would be handled by our JS code.

That’s it! It’s working!

Everything is in place. Exciting! Let’s see the results now!

Looking good!

Thanks for reading!

)

Jack Spektor

Written by

Sitecore MVP 2018–2019, Sitecore Developer in AKQA, traveler and lover of life.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade