Use Apple Vision Framework (iOS) to calculate the size of border around an input image

Highlights:

• Apple iOS Vision Framework
• Image processing without 3rd party libraries (such as OpenCV etc.)
• On device image processing on iOS

Last year, I worked on an interesting problem which forced me to go to CS basics. The ask was to process an input image and calculate the border area in both iOS. There were a lot of obstacles and constraints to find the best suited solution. I enjoyed the journey so much that I felt compelled to share it…here in form of a medium story.

Problem Statement

So, first things first. let’s put the problem statement together with all the constraints. Please see the screenshot below containing three examples of images with possible border variants (color, size, location etc.)

Image 1: 3 Examples of images with borders in different colors and sizes

The ask was to calculate the area occupied by this border. Basically, the image itself is the valid use of space (UIView or UIImageView in case of iOS) and the border around it could be there for multiple reasons. For example, the image could be small and the programmer may have used incorrect resizing properties (aspect fill or fit etc.). No matter what the reason is, the border around the image looks wrong and constitutes a bad user experience. And hence the ask to be able to calculate its size.

Constraints:

The only constraint was to keep the size of compiled product as small as possible. Hence using third party libraries or available frameworks (e.g. OpenCV) was totally out of scope. Otherwise this would have been a much simpler task since OpenCV is a rich library with several available options to process a given image.

Similarly, I also tried to come up with a model (using Xcode and CreateML) to finish the requirements. But this idea didn't go much further due to 2 reasons :

• After training the CoreML model, I was not able to get the results in acceptable range.
• The size of model file was again an issue.

To be fair to Apple’s CreateML, I would have succeeded in getting around the first issue but the second one (size of model file) was still going to be there.

The Solution for Apple iOS scenario:

The Solution on Apple side was to use Vision Framework to do the heavy legwork. VNDetectRectanglesRequest Class (Vision FW) can be used to detect multiple rectangles within an input image. Further, you can specify the input parameters such as aspectRatio, quadratureTolerane and minimumConfidence etc. to further refine this request. This request returns an array of observations. These observations are akin to sudden changes in image and can be used to detect different areas say in a credit card, business cards or a document. In our case, we will use to detect the start of a plausible border area. The idea is that if there is a continuous color with no observations (no contour change) in a given region, then this region can be thought of as a border in that direction.

Here is the summary of steps involved:

• Pass an input Image to Vision framework
• Make a VNDetectRectanglesRequest to detect small rectangles (regions) with specified parameters
• Make a minimum size rectangle (green Color in image below)that will fully contain all of these returned observations (red colored small rectangles in image below)
• The area inside this master rectangle is your valid Image. And anything outside is area occupied by the border

Image depicting Vision FW in action at various Steps:

Image 2

In the image above, the first subImage is the input image. The second subImage shows all the observations (returned by Vision FW) in red color. And finally the third subImage shows the minimum size master green rectangle containing all these observations.

Source Code:

Here is the source code for this implementation using SwiftUI framework

GitHub - VipulArvind/BorderSize

P.S. : I will update the article to add Android details later on…