Nostalgic Image Recognition on iOS with Swift and OpenCV

Like most people when travelling abroad, I want to bring back a souvenir that will remind me of my trip. I also enjoy flipping through the various photo albums from different trips and events but sometimes it can be a struggle to sift through them all. I pondered if I could put together a small application with some pattern recognition that would let me scan my personal items and could open the associated trip’s album.

The Setup:

• Apple Developer Program Account (Simulator doesn’t have a camera)
• Xcode 9.2
• OpenCV 3.3.1 iOS Pack
• iPhone X (any iOS device with a decent camera will do)

Setup a new project

Open Xcode, in the menu select File > New > Project, then choose the “Single View App” option.

Give your application a name, choose Swift as the language, and remove the Core Data and Testing options since we’re just prototyping for now.

Using the device’s camera which requires the user’s permission. iOS will prompt the user automatically and only requires a message to give the user context. In the Navigator click on Info.plist, right-click on the blank space and add a new row. The key will be “Privacy — Camera Usage Description” and the value can be whatever message would like to be displayed.

Import OpenCV

Download OpenCV 3.3.1 for iOS to any folder, then drag it into the project. When prompted, select “Copy Items if needed” if it isn’t already selected. After a few seconds, there will beopencv2.framework in the Navigator and under “Linked Frameworks and Libraries” which is at the bottom of the “General” tab for the your application’s target.

OpenCV relies on some core iOS frameworks. Click on the “+” and add both the CoreMedia and AVFoundation frameworks to make the complete list of linked frameworks look like this…

Find the Nostalgia Trigger

Take the picture with the device, move it to your computer, resize it to a width of 640 while keeping its aspect ratio (OpenCV captures at 640x480 by default) and crop out the unique part of the item via Preview.

When choosing the unique part for the application to process, it’s important to note that OpenCV is going to do edge/pattern detection to figure out if it’s a match. Initially I thought of capturing the “DUBLIN” on the front, but font’s edges aren’t crisp enough. The small green shamrock ended up being a much better option.

Once the image is ready, import the image into the project. In the Navigator, click on Assets.xcassets, right-click on the empty space below AppIcon, and select “New Image Set”. Then rename the new image set to “item1” and drag your image into the “1x” square.

Create an OpenCV Wrapper

Swift is great, but OpenCV is written in C++. In order for them to work in the same project, make a class that will access all the OpenCV goodness that the rest of our Swift code can interface with.

In the xCode menu click on File > New > File, select “Objective-C File” and click next. On the next screen, give it an appropriate name (ex. NostalgiaCamera), make sure you have your target selected and the “File Type” is an empty file.

When prompted with “Would you like to configure an Objective-C bridging header?” be sure to click “Create Bridging Header”. This is component that allows your new class to be visible in your Swift code.

Follow the same process to also create a “Header File” with the same name…

Lastly, rename the NostalgiaCamera.m file to NostalgiaCamera.mm so that Xcode will treat it like an Objective-C++ file.

Write Some Code

This will setup OpenCV extensions of its CvVideoCamera and CvVideoCameraDelegate.

The last part is taking advantage of the (project name)-Bridging-Header.h ex. (NostalgiaScanner-Bridging-Header.h) file that Xcode created for us by adding a single line to it. This will expose both the class and the protocol to Swift.

#import "NostalgiaCamera.h"

Hook it up to the UI

The extended CvVideoCamera requires an ImageView. It’s up to you whether you want to create the ImageView with the storyboard or with code. This example is using a storyboard for the sake of simplicity.

In the Navigator click the Main.storyboard and then the View for the ViewController, search for the ImageView object and drag it onto the canvas, then add constraints for everything but the bottom with a value of 0. Also add the “Aspect Ratio” constraint to avoid stretching of the image. Clicking on the constraint allows the ability to specify an aspect ratio value to “3:4".

Next, link the new ImageView to the ViewController code and give it the name “imgView”.

Only thing left to do is to add the rest of the ViewController code…

Done! 🙌

There should be something that is functioning like the clip at the start of the article. Adding more images or changing the url to point towards a Google Photos album or opening another application are both possibilities. The full source for this project is also available on Github.

This just scratches the surface of vision frameworks and vast amount of new and upcoming use cases. If you’re curious, I definitely urge you to explore. 😁