How to draw bounding boxes with SwiftUI
While building an awesome machine learning library for mobile devices called Amplify, I learned some pretty cool tricks with SwiftUI and I wanted to share one of them since I couldn’t seem to find any good documentation on it. In order to build a sample app in SwiftUI that demonstrates the power of machine learning, I actually needed to do some complex things using SwiftUI. One of those things was the ability to draw bounding boxes. Bounding boxes are very useful for debugging UI, but they’re also the data structures that are returned from ML Frameworks. If you already know how to do this in UIKit, you might find the SwiftUI approach confusing like I did.
When you use your machine learning models to detect objects or text in an image, you are usually returned a bounding box i.e. where the thing the model has detected is located in the actual image. If you’ve used CoreML or MLKit previously, you might be familiar with using a bounding box. The first thing to note is what is actually being returned to you which can be easily missed in documentation of any of these machine learning frameworks. You might think when you see the bounding box model, “Oh all I have to do is plot x, y, width, and height as a CGRect”. But this wouldn’t be correct because what is returned to you is actually a ratio of the x, y, width, and height so that you can plot this bounding box on an image size of your choosing. So now let’s look at what this means conceptually.
let trueX = boundingBox.x * image.size.widthlet trueY = boundingBox.y * image.size.height let width = boundingBox.width * image.size.width let height = boundingBox.height * image.size.height
Now the above code might be how it would look if you were using UIKit in your view controller to place an overlay on your UIImage, but remember we decided to try this in SwiftUI.
One of the first things you learn about SwiftUI if you read anything is that the layout of a child element is inferred from the parent element. Which is great! Meaning you never spend time setting height and width for children elements because naturally you don’t need to because they are automatically inferred to make sure children fit nicely within their parents’ layout.
Well, this feature is great most of the time. Except in our case we actually need to get this inferred width and height of our parent element, the image, in order to actually display the bounding box because as we now know, we only get ratios back. So what the heck do we do?
After hours of searching online and going through pages upon pages of SwiftUI documentation it’s finally clear: we need to call something called the GeometryReader. So in the below code, I create the image i want to display using normal SwiftUI practice, then I call .overlay() to add a bounding box as the overlay, then inside there I can call the GeometryReader to make sure I can get the width and height of the parent element which in this case is our Image object. Then from there I just create the Rectangle and call .path on it to create the CGRect of the bounding box and as you can see I multiple the geometry of the image to pull the true values out from the ratio I was given. See the code below:
My full SwiftUI file looks like this:
and you get a solution that looks like this with the above code:
Full code example of how to use SwiftUI to display machine learning results from Amplify Predictions is here but this same code above can be used when receiving bounding boxes from any ML framework.
So just to recap what we did, we used SwiftUI to display an overlay in an image that is a rectangle using the height and width of that image. I welcome feedback on this solution and would love to know if anyone figured out another way to do the same thing. After all my research, this seemed to be the easiest way for me to pull out the size of the image from the parent element that I could find in the SwiftUI documentation.
