Pythonic Eyes: Unveiling the Art of Stereo Vision Calibration with OpenCV

Introduction

Stereo vision is the process of extracting 3D information from multiple 2D views of a scene. Stereo vision is used in applications such as advanced driver assistance systems (ADAS) and robot navigation where stereo vision is used to estimate the actual distance or range of objects of interest from the camera. — MathWorks

But to put it simply, Stereo vision, in essence, is closely tied to the number of cameras employed for visualizing a scene. A single-camera calibration results in monocular vision, while the use of two cameras defines stereovision. Having two cameras proves more advantageous in converting 2D space into true-to-life 3D scenes — an exploration we delve into in this vision sub-series.

The role of the calibration parameters in the reconstruction of the scene is two-fold. On one side they are needed to match the images across the cameras, hence to pair the points belonging to different cameras, images of the same 3D target. On the other side they are needed in the actual 3D reconstruction process, to define the geometry of the system with respect to the 3D reference frame. — Beschi et.al.

Now that we are excited enough to learn about the topic, lets see how to bring to reality. While stereovision calibration might seem daunting to many, I’ll try and make sure to simplify the process and ensure the code’s reproducibility.

Setting Up Your Environment

For setting up the environment, I prefer using the Anaconda distribution due to its stability in managing complex and isolated environments. However, you can use either Anaconda or pip for this project, dealing primarily with OpenCV and supporting libraries.

For Anaconda

conda install -c conda-forge opencv
conda install -c anaconda numpy

For Pip

pip install opencv-python
pip install numpy

A few hardware considerations

Before proceeding with calibration, consider the following hardware points:

• Ensure both cameras have similar configurations.
• While not strictly necessary, using identical cameras simplifies the process.
• Experimenting with different cameras is possible but may require more calibration iterations.

Distance and Angle Between Cameras

Stereo calibration is an iterative process. Adjustments in hardware parameters, such as the distance and angle between cameras, may be necessary. Remember that a lower distance requires a lower convergence angle.

Stereo Camera Calibration Process

The calibration process involves capturing enough chessboard images and running a script with precise parameter configurations. This process helps find intrinsic and extrinsic parameters for the camera setup.

• Extrinsic Parameters: Correspond to rotation and translation vectors, representing the camera’s location in the 3D scene.
• Intrinsic Parameters: Represent the optical center and focal length of the camera.

Using sample images of a well-defined pattern (e.g., a chessboard), we find specific points with known relative positions. At least 10 test patterns are recommended for accurate results (Ref. OpenCV docs).

Remember, the extrinsic parameters signify the camera’s location in the 3D scene, while the intrinsic parameters represent the optical center and focal length.

Chessboard Pattern Size

When using a chessboard pattern in code, consider using (x-1, y-1) pattern dimensions instead of (x, y). This adjustment ensures the machine finds internal corners, not external ones. So, the input for the below pattern would be (7, 4) not (8, 5).

Capture Images

Capture images in real life using the chessboard pattern on both cameras. Ensure you have at least 10 different images in different planes for more accurate calibration. Organize images in separate folders titled “stereoLeft” and “stereoRight” for reproducibility.

With these preparations, you’re ready to dive into the calibration script.

Import libraries

import numpy as np
import cv2 as cv

Set configurations

chessboardSize = (7, 4) # Use (x-1, y-1) to the actual size of chessboad pattern
frameSize = (640,480) # Resolution of both cameras.

criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # To be used in sub pixel finder code.

Set initial variables and arrays

Prepare the object points for our calibration

objp = np.zeros((chessboardSize[0] * chessboardSize[1], 3), np.float32)
objp[:,:2] = np.mgrid[0:chessboardSize[0],0:chessboardSize[1]].T.reshape(-1,2)

objpoints = [] # 3d point in real world space
imgpointsL = [] # 2d points in image plane.
imgpointsR = [] # 2d points in image plane.


imagesLeft = glob.glob('images/stereoLeft/*.png')
imagesRight = glob.glob('images/stereoRight/*.png')

Main loop

for imgLeft, imgRight in zip(imagesLeft, imagesRight):

    imgL = cv.imread(imgLeft)
    imgR = cv.imread(imgRight)
    grayL = cv.cvtColor(imgL, cv.COLOR_BGR2GRAY)
    grayR = cv.cvtColor(imgR, cv.COLOR_BGR2GRAY)

    # Find the chess board corners
    retL, cornersL = cv.findChessboardCorners(grayL, chessboardSize, None)
    retR, cornersR = cv.findChessboardCorners(grayR, chessboardSize, None)

    # If found, add object points, image points (after refining them)
    if retL and retR == True:

        objpoints.append(objp)

        cornersL = cv.cornerSubPix(grayL, cornersL, (11,11), (-1,-1), criteria)
        imgpointsL.append(cornersL)

        cornersR = cv.cornerSubPix(grayR, cornersR, (11,11), (-1,-1), criteria)
        imgpointsR.append(cornersR)

        # Draw and display the corners
        cv.drawChessboardCorners(imgL, chessboardSize, cornersL, retL)
        cv.imshow('img left', imgL)
        cv.drawChessboardCorners(imgR, chessboardSize, cornersR, retR)
        cv.imshow('img right', imgR)
        cv.waitKey(1000)


cv.destroyAllWindows()

Individual calibration

For more accuracy we will calibrate both cameras separately first and then do stereo calibration.

retL, cameraMatrixL, distL, rvecsL, tvecsL = cv.calibrateCamera(objpoints, imgpointsL, frameSize, None, None)
heightL, widthL, channelsL = imgL.shape
newCameraMatrixL, roi_L = cv.getOptimalNewCameraMatrix(cameraMatrixL, distL, (widthL, heightL), 1, (widthL, heightL))

retR, cameraMatrixR, distR, rvecsR, tvecsR = cv.calibrateCamera(objpoints, imgpointsR, frameSize, None, None)
heightR, widthR, channelsR = imgR.shape
newCameraMatrixR, roi_R = cv.getOptimalNewCameraMatrix(cameraMatrixR, distR, (widthR, heightR), 1, (widthR, heightR))

Stereo Vision calibration

flags = 0
flags |= cv.CALIB_FIX_INTRINSIC
# Here we fix the intrinsic camara matrixes so that only Rot, Trns, Emat and Fmat are calculated.
# Hence intrinsic parameters are the same 

criteria_stereo= (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# This step is performed to transformation between the two cameras and calculate Essential and Fundamental matrix
retStereo, newCameraMatrixL, distL, newCameraMatrixR, distR, rot, trans, essentialMatrix, fundamentalMatrix = cv.stereoCalibrate(objpoints, imgpointsL, imgpointsR, newCameraMatrixL, distL, newCameraMatrixR, distR, grayL.shape[::-1], criteria_stereo, flags)

print(newCameraMatrixL)
print(newCameraMatrixR)

Rectification

After finding the new camera matrix and distortion for both cameras, use rectify map and create the new stereo map for the setup.

rectifyScale= 1
rectL, rectR, projMatrixL, projMatrixR, Q, roi_L, roi_R= cv.stereoRectify(newCameraMatrixL, distL, newCameraMatrixR, distR, grayL.shape[::-1], rot, trans, rectifyScale,(0,0))

stereoMapL = cv.initUndistortRectifyMap(newCameraMatrixL, distL, rectL, projMatrixL, grayL.shape[::-1], cv.CV_16SC2)
stereoMapR = cv.initUndistortRectifyMap(newCameraMatrixR, distR, rectR, projMatrixR, grayR.shape[::-1], cv.CV_16SC2)

print("Saving parameters!")
cv_file = cv.FileStorage('stereoMap.xml', cv.FILE_STORAGE_WRITE)

cv_file.write('stereoMapL_x',stereoMapL[0])
cv_file.write('stereoMapL_y',stereoMapL[1])
cv_file.write('stereoMapR_x',stereoMapR[0])
cv_file.write('stereoMapR_y',stereoMapR[1])

cv_file.release()

Done!

Now your calibration is complete and you can use this stereomap for rectifying your frames by using the following function

# Camera parameters to undistort and rectify images
cv_file = cv2.FileStorage()
cv_file.open('stereoMap.xml', cv2.FileStorage_READ)

stereoMapL_x = cv_file.getNode('stereoMapL_x').mat()
stereoMapL_y = cv_file.getNode('stereoMapL_y').mat()
stereoMapR_x = cv_file.getNode('stereoMapR_x').mat()
stereoMapR_y = cv_file.getNode('stereoMapR_y').mat()


def undistortRectify(frameR, frameL):

    # Undistort and rectify images
    undistortedL= cv2.remap(frameL, stereoMapL_x, stereoMapL_y, cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT, 0)
    undistortedR= cv2.remap(frameR, stereoMapR_x, stereoMapR_y, cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT, 0)


    return undistortedR, undistortedL

Common Issues and Solutions

Encountering calibration challenges? Here are some common issues and their solutions:

Inadequate Frame Overlap

• Issue: Improper calibration? Ensure both cameras have sufficient frame overlap.
• Solution: Increase the overlap of camera frames for better calibration results.

Insufficient Calibration Images

• Issue: Still facing calibration issues? Try capturing more calibration images and vary the chessboard’s position in different planes.
• Solution: Increase the number of calibration images, ensuring diverse perspectives for accurate calibration.

Chessboard Corners Not Detected

• Issue: Unable to find chessboard corners in the images?
• Solution: Verify that the chessboard pattern is in focus, and ensure correct dimension input in the code.

And there you have it! With these solutions, your stereo vision setup should be good to go. In the upcoming article, we’ll explore how to leverage this setup to measure depth and transition from 2D to 3D vision. Stay tuned!