Optical flow and motion detection
Hi everyone! In the last post, we saw a little bit about motion detection using Background Subtraction in OpenCV. Really cool, isn’t it?! We saw that is possible to analyze a frame sequence and segment an object of the image just with the difference between the pixels of a frame and a scene model. NICE! But if we want to know the velocity of a moving object? Or it’s direction? This information we can achieve with Optical Flow (The mathematical of the technic is not our aim here, but as I say, it is always good to check under the hood. Like that you will know, not only how to use, but also how it works)!
Optical Flow gives us the capacity of motion detection in a frame sequence, admitting that:
- the intensities of the pixels are constant through time;
- exists a neighborhood of pixels that move together.
It is kind of obvious, right? A digital image is compound by pixels with different intensities values (e.g.: 8bits, intensity of 0–255). An object in the image can be compound by pixels of completely different intensities, but think that it is made by a sequence of pixels in the same neighborhood and with similar intensities. Now think that this object moved in the next frame (e.g.: a displacement to the right), and both observations become true.
As I said, this technique gives us the ability to find both the direction and amplitude of the motion. This is really interesting! Think about how many application you can use such kind of information. You can, for example, to predict when an impact will occur in a target object, or to spare processing of your system analyzing just objects that move in your interest direction or to stabilize your video stream.
OpenCV has two types of Optical Flow algorithm, and we’ll focus on Dense Optical Flow, that is, we’ll obtain the motion of the entire image. Check later in the other kind of optical flow detection, where are chosen just a few features in the image, and these features are them tracked. Let’s code!
This is an example of a script to detect the main direction of motion in a video. Let’s comment on some important parts. Well, as Optical Flow calculates the motion vector of two consecutive frames, the first step is to store an initial frame. Lines 40 and 41. The Dense Optical Flow is calculated by the algorithm based here, and through the function calcOpticalFlowFarneback, line 60. This function returns a ndarray with the same number of rows and columns that both images in the calculation, but with two channels: the first one with the x-coordinates, and second, y-coordinates.
In OpenCV we can use the function catToPolar to get the magnitude and direction (angle) of the motion through the previous coordinates, line 61. Take a look in the argument angleInDegrees = True. If False (default), it returns in radians. In degrees, the directions are:
- purely to the left — 0 / 360º;
- purely down — 90°;
- purely to the right — 180°;
- purely up — 270°.
In line 62 we divided the angles by 2 (we’ll see why later. Obs.: this step is not necessary). Finally, in line 63 we store the actual frame as the frame that will be used for the calculation in the next. Now it is just fun! Optical Flow calculated and ready for visualization. Let s go.
In line 65 we filter just the direction where the motion has a magnitude higher than a threshold. This removes some motion noises (not noises like salt & pepper) which are not movements, but just a small change in pixels intensities. Then, in line 66, we calculated the mode of these movements to see which one are predominant in the image. There is a lot of ways to do that: mean, median, maximum magnitude, etc., and that will depend on what you want to do.
From line 67 to 89, we create a kind of “accumulator” to generate certain inertia in the motion direction. I particularly like to use this kind of approach including in real industries applications, as this technique has the capacity to considerably reduces oscillations in your signal (nice! This kind of information became a 1D signal). The size of this accumulator will be of your choice, just remember: how much big it is, bigger will be the inertia. If you want to see this accumulator, just run the code with the argument “-p” or “ — plot”. And the direction is identified by this accumulator in lines 102 to 112, where we get the accumulator that has the bigger mean.
Take a look in the ifs. See that we do not take the same values of angles above, but with a little dephasing of 10°. This is because we can’t move purely in one direction. So, if you suddenly move 1º of 90º, for example, ops! You already in another direction.
Now to see the magnitudes and directions, we use the HSV color space, where: Hue is the direction; Saturation is… saturation! We leave saturation at maximum to make the colors very vivid; and Value is the magnitude. That is, for each direction of motion, we will have a certain color, and according to the magnitude of this motion, the intensity of this color will be stronger or weaker. Remember we divided the angle by 2 earlier? It’s because of it! The H channel in HSV color space of OpenCV has a range of 0–179, not 0–360, so we have to have values only between 0–180. Also, anyway, 360 is beyond cv.imshow’s default 255 (uint8). Another thing is to normalize the magnitude to the values 0–255 on line 115. This operation makes you see even very low magnitudes, because even if it’s an extremely small value, they will be in the range 0–255, and will be shown (it will be the noises I commented above). However, when you have larger magnitudes, these movements will be practically invisible. Ah! Of course! Line 116 converts HSV to RGB (or BGR, right). To see this image (which is really cool, by the way), run the code with the “- rgb” or “ — rgb” argument.
Well, that’s it! Optical Flow is too cool right ?! How many things can you do with it? Don’t forget to look under the hood, because knowing how the magic happens allows you to optimally parameterize the algorithm (not just this one, but everyone) for your application. Remembering that: not only for hobbyists but YES, ALSO IN INDUSTRY !! I hope you enjoyed! If you want to see more about artificial intelligence in industries, take a look in our website: MVISIA. To the next!