Harmonic Aliasing (Wiggle Error) in Indirect Time-of-Flight Depth Cameras
Indirect Time-of-Flight (iToF) depth cameras measure distance for each pixel. Ideally the measured distance is linear with the actual distance, however this is often not the case. This article explains one of the causes of non-linear distance measurements in iToF cameras, why this error occurs, and the possible fixes. We call this error harmonic aliasing, but it is also known as wiggle error, or circular error.
The figure below shows simulated distance error results for an iToF camera configured using 4 raw frames and running at 50MHz. The maximum distance before phase wrapping occurs at 50MHz is 2998mm, therefore the peak to peak error of 58mm is an accuracy error of approximately 2% (58/2998).
An error of 58mm ill cause issues for applications such as picking up objects with a robotic arm.
The Cause of Harmonic Aliasing
iToF sensors measure the cross-correlation between the laser signal and the sensor signal. The phase and amplitude of the correlation signal correspond to the distance travelled and the amount of reflected light.
To measure the phase and amplitude of the correlation signal it is sampled multiple times, and a Fourier transform across the samples recovers the phase and amplitude. The animation below show where the samples on the correlation signal change (using four samples 90 degrees apart) as the laser signal changes phase. With a iToF camera we call each of these samples raw frames. In an iToF camera each sample of the correlation signal is one raw frame from the image sensor, therefore it takes 4 raw frames from the image sensor to make one depth frame.
The mathematical theory presented about iToF [1] uses sinusoidal signals, not square waves that are used in practice.
The cross-correlation of two square waves is a triangle. A cross-correlation (or convolution) in time domain is a multiplication in frequency domain. The Fourier series of a square wave is odd frequencies with amplitudes of 1/n. Therefore the multiplication of two square waves with odd amplitudes results in a triangular wave with odd amplitudes of 1/n², in the frequency domain.
Another way to visualize this is the path on the real, imaginary plane. In the ideal case there is one frequency that rotates from 0 to 2pi, but with a triangular signal it is the sum of odd frequencies rotating, as plotted in the animation below. This representation is borrowed from explaining Fourier transforms by drawing circles.
In the case where there are four raw frames there are not enough samples to fully reconstruct the cross-correlation signal. In signal processing terms the Nyquist sampling frequency is two (4/2), therefore the higher frequencies alias around the Nyquist sampling frequency. This is illustrated in the animation below
This means the higher frequencies introduce a wiggle onto the first frequency, causing the error observed in figure 1. This is the cause of the harmonic error, the higher frequencies aliasing onto the primary harmonic and causing non-linearities in the phase measurements.
Reducing Harmonic Aliasing
There are four broad approaches to solving/mitigating this error in iToF cameras:
- Optimizing the Nyquist frequency
- Adjusting the duty cycle of the laser
- Calibrating away the error with a look up table or polynomial fit
- Other techniques to cancel the error
Optimize the Nyquist frequency
The simplest method to reduce error caused by harmonic aliasing is to use double the number of samples, than the highest frequency in the cross-correlation signal. This means 10 or 14 frames would be required, depending on highest frequency present and desired error. 14 samples is unreasonable because of the decrease in depth frame rate for most applications. The better approach is to use an odd number of samples as this avoids directly aliasing the higher frequencies, using 3 or 5 samples is best to minimize the error.
Duty cycle Adjustment
Cross-correlation in time is a multiplication in frequency (Fourier) space. If we can modify one of the signals to have a zero at a given location in its Fourier transform the resulting cross-correlation will also have a zero at that location (zero times anything is always zero). The animation below shows the Fourier transform change with duty cycle. At 29% there is a zero for the 3rd harmonic. By setting the duty cycle of the laser we can reduce the error due to harmonic aliasing, but not completely eliminate it. This concept is further explored by Payne et al. [2]
Look Up Table (LUT) or Polynomial Fitting
A common approach is to calibrate the non-linear phase by using a Look-up-table or curve fitting to model the error and know what offset to apply. The use of a LUT was described by Kahlmann et al [3], and spline fitting by Lindner et al [4]. to use calibration to remove this error.
This calibration method works when it is the only source of error in iToF measurements. However other measurement errors caused by motion blur, multi-path interference introduce their own offsets/errors. When the harmonic calibration is then applied on this data the calibration can increase error instead of decreasing it!
Other Techniques
There have been a few novel techniques to reduce or remove this error in ToF cameras by changing the modulation signal and/or depth calculation process. Payne et al. [5] changed the phase of the sensor modulation signal by +/- 45 degrees during the integration time of each raw frame. This cancelled out the contribution by the 3rd harmonic but reduced the signal power by 10% which decreases precision. This was expanded by Peters et al. [7] by using more phase steps.
Streeter [6] changed the phase steps from [0, 90, 180, 270] to [0, 90, 120, 210] and kept the calculation of phase the same. This has the affect of cancelling out the 3rd harmonic (so higher accuracy) but at a slightly reduced precision.
Feigin et al. [8] took a different approach and modelled the harmonic error as multi-path interference. By taking measurements at multiple modulation frequencies the harmonic contribution could be computed and the correct phase recovered.
About Chronoptics
Chronoptics are experts in designing indirect ToF depth sensing modules, focusing on the processing from the image sensor to the point cloud, and tailoring each module to output the best point cloud for a give application. For advice on correcting harmonic aliasing in your ToF sensor contact us at hello@chronoptics.com