Speeding-up OpenCV Features

This post introduces the project I’m doing for OpenCV as part of the Google Summer of Code.

Image features

Features are very useful stuff in computer vision. They look something like this:

Detected key-points on the image

The color circles on this picture are our goal. Each circle represents a single feature-point that has been selected by a feature detector.

Image features are one way how you can work with images in computer vision. One particularly nice thing you can do with features is stitching (also called the panorama).

Matches between key-points detected on two images while creating the panorama

The key problem in image stitching is to identify the same point on different images. When we know the correspondence, we can align the images accordingly and create the panorama.

To be able to identify key-points across images we attach each key-point with a unique descriptor. The descriptor contains information about surrounding area of each key-point. It is a bit hard to visualize, but it may look a bit like this:

Key-points with size and orientation

Features that have the same descriptor should represent the same point across different images (for example the same scene taken from a different angle). If it works you may obtain this:

Image features are useful far beyond stitching images. Check these videos on object tracking and SLAM (the way how robots navigate) if you want to see some more examples.

Speed!

To be able to create panoramas with lots of images in them without draining all your battery, we need to be able to detect features in them quickly and efficiently.

Cars running on the highway and flying drones move fast. If we want to use this algorithm for our self-driving vehicles it needs to be quite fast.

To get seamless motion estimation we would like to run our algorithm at ~60 fps. Of cause, the key-point detection is just one part of the whole processing, so it needs to be much faster than that.

Medium.com homepage in Chrome profiler

To put some perspective on what means fast: medium.com home page takes 2197 ms to load on my computer. Detecting and describing features on the image above takes currently 38.303 ms. That is 57.35x faster.

And while Chrome needs 2197 ms just to display a bunch of images and text we need to be 57.35x faster with some complex image analysis algorithm? Crazy.


I will be publishing follow-up posts about this project in the upcoming weeks. Especially I’d like to share some tricks I used to make the algorithm faster. If you are interested in C++ or OpenCV performance subscribe.

If you want to learn more about image features in OpenCV check the documentation or tutorials.