Using Power apps and cognitive services to recognize people in Office 365 environment?
Some time ago, I thought about creating a Canvas app with the Microsoft Power Apps platform, something which can be useful in the company and also use Microsoft Azure cognitive services.
Finally, I decided to use Face API of cognitive services to make some fun with face recognition to detect my colleague’s faces and information from the photos. most of the companies have yearly events and lots of photos from employees. search within thousands of photos to find only a specific person wouldn’t be easy. with the use of Microsoft cognitive services together with the help of Office 365 API it would be easy.
First of all, we need to have at least one photo of each colleague and it’s available already in the Office 365 profile picture. So we can use Office 365 API user search to find all the employee list from active directory and recognize the face from the profile picture and register the user as a person in cognitive service. also, each office 365 user profile information can be stored as a person meta data in cognitive service and detected face from the profile photo with a unique face Id will be assigned to that person object.
from now on we can just take a photo of a colleague using a smartphone camera or just an existing image to recognize the persons in the photo.
I dig a little bit deeper into the implementation of the solution. To simplify the maintenance of the entire workflow and process, I’ve created most of the business logic and services in a REST API using Function Apps, and then made a Custom connector from it in Power App in order to call these endpoints directly from my canvas app or also from a custom Flow. Therefore, I should not update the Canvas app at all after a few minor changes in API.
by capturing photo with camera, the photo will be send to the API in order to be analysed and find the person face. it detect also all the faces in a photo and find the persons base on the faces.
Integration with SharePoint
It is also possible to find similar faces in a bunch of event’s photos. the only thing that should be done is analyzing each photo content during storing the photo and identifying all faces. If the photo gallery already exists, then we need to execute the process one time for all pictures.
imagine that there is a photo gallery in SharePoint and we want to find all photos of person A. after identifying the faces in the photo, we need to store all the face Ids in a Large list of all faces in the cognitive resource database. Also, we can store the photo metadata, like SharePoint Item Id or Picture URL, as user data per each face. for instance the location of the face in the photo, so if there is more than one face in a photo we can find the location of the person later on. This process can be done also by using Microsoft flow.
After we identified and stored all the faces from Photo Gallery into a Large Face List, we can start taking a photo or just using the person office 365 profile picture to find all other similar faces in the face list. The result of the process can be a list of face Ids with user metadata (the picture URLs). so we can show all the photos using the URLs.
of course, it makes sense to integrate this feature as a plugin or a work flow in SharePoint. Imagine that during uploading photos to the SharePoint, this service can find and recognize the faces in the picture and add some tags (person’s name) to the item.
