StereoPi
Published in

StereoPi

OpenCV and Depth Map on StereoPi tutorial

OpenCV and Depth Map on StereoPi tutorial

Introduction

Hardware setup

Software used:

  • Raspbian Stretch (kernel 4.14.98-v7+)
  • Python 3.5.3
  • OpenCV 3.4.4 (pre-compiled, ‘pip’ from Python Wheels)
  • Picamera 1.13
  • StereoVision lib 1.0.3 (https://github.com/erget/StereoVision)

Notice

Step 1: Image Capture

cd stereopi-tutorial
python 1_test.py

Step 2: Collecting Images For calibration

python 2_chess_cycle.py

Step 3: Image Separation

python 3_pairs_cut.py

Step 4: Calibration

python 4_calibration.py

Step 5: Depth Map Tuning

python 5_dm_tune.py

Step 6: Real Time Depth Map Using Video

python 6_dm_video.py

Notes for advanced users:

  • Once started, the first script will display the average time between captured frames and, once stopped, it will show the average FPS. This can be used as a simple tool to determine the camera and capture parameters at which Python can capture imagery without dropping frames. Using this script, we’ve found that setting the camera to 1280x480 at 20 FPS provided stable results, with no lost frames.
  • You may have noticed that we capture images from camera at 1280x480 resolution and then scale it down to 640x240. You may ask, “why don’t we just capture 640x240, and avoid the Python processing to scale them?”
    The reason is that PiCamera has some bugs related to capturing images at low resolutions, which results in the image appearing broken (photo mixed with green squares). So, to avoid this, we use a method of acquiring a large image and then scaling it down via the GPU. In this way, there is no additional load to our ARM CPU!
  • Why do we use BGRA capture, but not BGR?
    As we mentioned before, we resize the images using the GPU, and a native GPU format is BGRA (or RGBA, as you wish). If you use BGR you will find two issues; the first is a lower FPS (about 10–20% in our case) and the second is a PiCamera warning «PiCameraAlfaStripping: using alpha-stripping to convert to non-alpha format; you may find equivalent alpha format faster”. Googling this error led us to PiCamera documentation where we found the BGRA method.
  • Where is PiRGBArray in your code?
    This is a native class of PiCamera for working with cameras. In our experiments, using a hand-made Numpy array instead of PiRGBArray gives us about 50% more FPS. We don’t think PiRGBArray is a poor choice and it may be we made some mistakes in implementation. We’ll look into this further at a later time.
  • What is the CPU load while building the Depth Map with video?
    Let me answer this with a screenshot:
  • What is the maximal Depth Map FPS we can have with these settings?
    We were able to achieve ~17 FPS, while the camera captures 20 FPS. FPS appears to be most sensitive for a number of parameters. MinDisparity and NumOfDisparities are obvious as these define the number of “steps” for running a window while comparing two images. Another parameter appears to be preFilterCap.
  • What is the CPU temperature?
    Using a Compute Module 3+ Lite, we’re seeing a temperature of 63.4’C
  • How can we use GPU?
    It can be used for undistortion and rectification in a real time. There are some similar approaches, for example using WebGL, or Python-based Pi3d, and also Processing for Raspberry Pi.
    There is also an interesting project called py-videocore by Koichi Nakamura. In our communications Koichi told me to use his kernel and StereoBM sources for CUDA to accelerate the depth map. So, there are lot of optimization options.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store