iOS 11 Tutorial: Vision framework

Apple has introduced several new cool frameworks at WWDC 2017.

ARKit — a framework that provides APIs for integrating augmented reality into your apps.

Core ML — provides an API where developers can provide a trained model, some input data, and then receive predictions about the input data based on the trained model.

Vision — a framework to apply high-performance image analysis and computer vision techniques to identify faces, detect features, and classify scenes in images and video.

And much more new frameworks and improvements to exist frameworks.

In this tutorial I am going to focus on Vision framework.

Vision framework allows you to:

1. Detect face rectangle and face landmarks (face contour, median line, eyes, brows, nose, lips, pupils position)
2. Find projected rectangular regions surface.
3. Find and recognizes barcodes.
4. Find regions of visible text.
5. Determine the horizon angle in an image.
6. Detect transforms needed to align the content of two images.
7. Process images with Core ML model.
8. Track movement of a previously identified arbitrary object across multiple images or video frames.

Under the hood there are 4 base class categories:

• VNRequest and derived classes — describe your analysis request. It has request completion handler and array of results.
• VNObservation and derived classes— describe an analysis result.
• VNImageRequestHandler, VNSequenceRequestHandler — processes one or more VNRequest on given image.
• VNErrorDomain

Lets do some coding

Install Xcode 9 Beta to make everything below working.

1. Create an UIImage instance and save it as a result.
2. Create an instance of VNDetectFaceLandmarksRequest with completion handler. The handler will be called with analysis results.
3. Create an instance of VNImageRequestHandler from our source image and perform request created earlier.
4. Inside completion handler try to cast results to [VNFaceObservation]
5. Iterate over VNFaceObservation instances. It could more than one face on an image.
6. Try to get face landmarks from VNFaceObservation. Save the face bounding box. The coordinates are normalized to the dimensions of the processed image
7. Save faceContour to temporary array if given landmark has it. Also here we may get the rest of face features. (median line, eyes, brows, nose, lips, pupils position)
8. Call custom function to draw face landmarks on source image.

Complete source code is available on GitHub.

kravik/facevision

Next iOS11 Tutorial: How to measure objects with ARKit

If you want to get latest news and tuorials about iOS development follow me on Twitter