Using ChArUco boards in OpenCV

Camera calibration and pose estimation.

TLDR: Much of the example Charuco code online at the moment is out of date, and will put you wrong. If ChatGPT or Stack Exchange is recommending you use: cv2.aruco.CharucoBoard_create(length, width, ...) you’ll find the correct syntax is now cv2.aruco.CharucoBoard((length, width), ...). Hopefully the example functions below will help.

If you’ve used OpenCV before, you’ll be familiar with the classic chessboard we use to calibrate cameras. You may also be familiar with Aruco markers, which are used for pose estimation in robotic applications. These markers carry a unique identifier and — similar to the chessboard — we can use its corners to figure out its coordinate frame. A Charuco board is a helpful way of combining them into one, as illustrated.

This allows us to uniquely identify which corner of the chessboard is which. Identifying unique corners is useful for two reasons:

• A chessboard won’t be detected if its partly blocked. This is not the case for Charuco boards.
• Identifying corners can help us more accurately describe the board’s coordinate frame.

Since some of the more recent OpenCV releases (the latest being 4.8.0 at the time of writing) there have been some pretty significant changes to the Aruco library — large parts of the code you will find floating around online at the moment have become obsolete. Hopefully this guide will provide a more up to date walkthrough, and show you how I also use Charuco boards for image rectification and pose estimation (a difficult task with the standard checkerboard).

Ensure you’re running the latest version of Python, and also the contrib version of OpenCV. This step is very important, as it gives us access to the correct library:

pip install opencv-contrib-python

It’s important you use the `contrib` version to get access to the library — and make sure you don’t install the headless version (which blocks GUIs from appearing). If you’re getting errors trying to run any of the Aruco or ChAruco library functions, it is likely an error with your installation.

Next, you can start writing your script.

Create the Charuco Board

The best place to start is creating yourself a Charuco board. The following script will help you create a board according you your requirements. The image will display for two seconds before saving into your current folder.

import os
import numpy as np
import cv2

# ------------------------------
# ENTER YOUR PARAMETERS HERE:
ARUCO_DICT = cv2.aruco.DICT_6X6_250
SQUARES_VERTICALLY = 7
SQUARES_HORIZONTALLY = 5
SQUARE_LENGTH = 0.03
MARKER_LENGTH = 0.015
LENGTH_PX = 640   # total length of the page in pixels
MARGIN_PX = 20    # size of the margin in pixels
SAVE_NAME = 'ChArUco_Marker.png'
# ------------------------------

def create_and_save_new_board():
    dictionary = cv2.aruco.getPredefinedDictionary(ARUCO_DICT)
    board = cv2.aruco.CharucoBoard((SQUARES_VERTICALLY, SQUARES_HORIZONTALLY), SQUARE_LENGTH, MARKER_LENGTH, dictionary)
    size_ratio = SQUARES_HORIZONTALLY / SQUARES_VERTICALLY
    img = cv2.aruco.CharucoBoard.generateImage(board, (LENGTH_PX, int(LENGTH_PX*size_ratio)), marginSize=MARGIN_PX)
    cv2.imshow("img", img)
    cv2.waitKey(2000)
    cv2.imwrite(SAVE_NAME, img)

create_and_save_new_board()

We can specify how many rows and columns we want in the chessboard, as well as the size of the squares and markers (the documentation specified that it should be in meters, although in this instance, we’re really only interested in the ratio of the two, since we specify the image size). In the function above:

• Dictionary represents the dictionary of Aruco markers which are used,
• Board is the Charuco object,
• img is the drawing of the board (cv Image object).

The board from the above function.

Calibrating with the Board

For this step, you will have printed out the image you’ve just saved, and taken a photo of it (maybe attached to some surface). For camera calibration, you’ll want at least 10 different images of the board from a few angles. This is because calibration is correcting for radial and tangential lens distortion and transforming our image to the “ideal pinhole” camera model. It does this by solving an optimisation problem to minimise the least squares projection error between the image points and the expected world points — so more images means more datapoints for the optimisation.

If you have all your images in a folder, you can try the script below to run a calibration on your images and save out the result:

# ------------------------------
# ENTER YOUR REQUIREMENTS HERE:
ARUCO_DICT = cv2.aruco.DICT_6X6_250
SQUARES_VERTICALLY = 7
SQUARES_HORIZONTALLY = 5
SQUARE_LENGTH = 0.03
MARKER_LENGTH = 0.015
# ...
PATH_TO_YOUR_IMAGES = '/Users/Ed/Downloads/Calibration_Images'
# ------------------------------

def calibrate_and_save_parameters():
    # Define the aruco dictionary and charuco board
    dictionary = cv2.aruco.getPredefinedDictionary(ARUCO_DICT)
    board = cv2.aruco.CharucoBoard((SQUARES_VERTICALLY, SQUARES_HORIZONTALLY), SQUARE_LENGTH, MARKER_LENGTH, dictionary)
    params = cv2.aruco.DetectorParameters()

    # Load PNG images from folder
    image_files = [os.path.join(PATH_TO_YOUR_IMAGES, f) for f in os.listdir(PATH_TO_YOUR_IMAGES) if f.endswith(".png")]
    image_files.sort()  # Ensure files are in order

    all_charuco_corners = []
    all_charuco_ids = []

    for image_file in image_files:
        image = cv2.imread(image_file)
        image_copy = image.copy()
        marker_corners, marker_ids, _ = cv2.aruco.detectMarkers(image, dictionary, parameters=params)
        
        # If at least one marker is detected
        if len(marker_ids) > 0:
            cv2.aruco.drawDetectedMarkers(image_copy, marker_corners, marker_ids)
            charuco_retval, charuco_corners, charuco_ids = cv2.aruco.interpolateCornersCharuco(marker_corners, marker_ids, image, board)
            if charuco_retval:
                all_charuco_corners.append(charuco_corners)
                all_charuco_ids.append(charuco_ids)

    # Calibrate camera
    retval, camera_matrix, dist_coeffs, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco(all_charuco_corners, all_charuco_ids, board, image.shape[:2], None, None)

    # Save calibration data
    np.save('camera_matrix.npy', camera_matrix)
    np.save('dist_coeffs.npy', dist_coeffs)

    # Iterate through displaying all the images
    for image_file in image_files:
        image = cv2.imread(image_file)
        undistorted_image = cv2.undistort(image, camera_matrix, dist_coeffs)
        cv2.imshow('Undistorted Image', undistorted_image)
        cv2.waitKey(0)

    cv2.destroyAllWindows()

calibrate_and_save_parameters()

What’s going on in the function:

• You still create your dictionary (for Aruco markers) and board (ideal board), but also params an object which can be modified to change the way markers are detected (you could, for instance, modify the thresholding).
• For each image, you start by detecting markers, given the expected markers and search parameters.
• The key step, cv2.aruco.interpolateCornersCharuco finds the checkerboard corners, and matches the corresponding Aruco ID to them.
• Given that corners were found and identified, these are collected and loaded into a function.

You can see the corners of the image are now warped to account for the lens distortion.

Obtaining Pose with the Board

This is the really useful part of a Charuco board — we can leverage both the calibration ability of the Chekerboard and the pose estimation of the Aruco markers.

Of course, for pose estimation, we only need a single image (or frame — if your source is a video). The function detect_pose below takes a single image, but I’ve added a main function to iterate through all our calibration images. Hopefully the code is fairly self explanitory.

def detect_pose(image, camera_matrix, dist_coeffs):
    # Undistort the image
    undistorted_image = cv2.undistort(image, camera_matrix, dist_coeffs)

    # Define the aruco dictionary and charuco board
    dictionary = cv2.aruco.getPredefinedDictionary(ARUCO_DICT)
    board = cv2.aruco.CharucoBoard((SQUARES_VERTICALLY, SQUARES_HORIZONTALLY), SQUARE_LENGTH, MARKER_LENGTH, dictionary)
    params = cv2.aruco.DetectorParameters()

    # Detect markers in the undistorted image
    marker_corners, marker_ids, _ = cv2.aruco.detectMarkers(undistorted_image, dictionary, parameters=params)

    # If at least one marker is detected
    if len(marker_ids) > 0:
        # Interpolate CharUco corners
        charuco_retval, charuco_corners, charuco_ids = cv2.aruco.interpolateCornersCharuco(marker_corners, marker_ids, undistorted_image, board)

        # If enough corners are found, estimate the pose
        if charuco_retval:
            retval, rvec, tvec = cv2.aruco.estimatePoseCharucoBoard(charuco_corners, charuco_ids, board, camera_matrix, dist_coeffs, None, None)

            # If pose estimation is successful, draw the axis
            if retval:
                cv2.drawFrameAxes(undistorted_image, camera_matrix, dist_coeffs, rvec, tvec, length=0.1, thickness=15)
    return undistorted_image


def main():
    # Load calibration data
    camera_matrix = np.load['camera_matrix.npy']
    dist_coeffs = np.load['dist_coeffs.npy']

    # Iterate through PNG images in the folder
    image_files = [os.path.join(PATH_TO_YOUR_IMAGES, f) for f in os.listdir(PATH_TO_YOUR_IMAGES) if f.endswith(".png")]
    image_files.sort()  # Ensure files are in order

    for image_file in image_files:
        # Load an image
        image = cv2.imread(image_file)

        # Detect pose and draw axis
        pose_image = detect_pose(image, camera_matrix, dist_coeffs)

        # Show the image
        cv2.imshow('Pose Image', pose_image)
        cv2.waitKey(0)

main()

This should allow you to get the pose of the board (which you may use to localise a robot, detect a plane coordinate system, etc).

The coordinate system is relative to the “top left” corner of the original image.