Harmonic Aliasing (Wiggle Error) in Indirect Time-of-Flight Depth Cameras

Indirect Time-of-Flight (iToF) depth cameras measure distance for each pixel. Ideally the measured distance is linear with the actual distance, however this is often not the case. This article explains one of the causes of non-linear distance measurements in iToF cameras, why this error occurs, and the possible fixes. We call this error harmonic aliasing, but it is also known as wiggle error, or circular error.

The figure below shows simulated distance error results for an iToF camera configured using 4 raw frames and running at 50MHz. The maximum distance before phase wrapping occurs at 50MHz is 2998mm, therefore the peak to peak error of 58mm is an accuracy error of approximately 2% (58/2998).

Figure 1: The measured cross-correlation signal using square waves (blue) compared to the ideal sine wave (red)on the left, and the resulting measurement error on the right.

An error of 58mm ill cause issues for applications such as picking up objects with a robotic arm.

The Cause of Harmonic Aliasing

iToF sensors measure the cross-correlation between the laser signal and the sensor signal. The phase and amplitude of the correlation signal correspond to the distance travelled and the amount of reflected light.

Figure 2: Cross-correlation of two square waves

To measure the phase and amplitude of the correlation signal it is sampled multiple times, and a Fourier transform across the samples recovers the phase and amplitude. The animation below show where the samples on the correlation signal change (using four samples 90 degrees apart) as the laser signal changes phase. With a iToF camera we call each of these samples raw frames. In an iToF camera each sample of the correlation signal is one raw frame from the image sensor, therefore it takes 4 raw frames from the image sensor to make one depth frame.

Figure 3: Four phase step sampling of cross correlation signal. By changing the phase of the sensor signals four times to [0, 90, 180, 270] degrees four samples are measured on the cross-correlation signal. The phase of these measurements correspond directly to the phase of the laser, which is how far the light has travelled.

The mathematical theory presented about iToF [1] uses sinusoidal signals, not square waves that are used in practice.

The cross-correlation of two square waves is a triangle. A cross-correlation (or convolution) in time domain is a multiplication in frequency domain. The Fourier series of a square wave is odd frequencies with amplitudes of 1/n. Therefore the multiplication of two square waves with odd amplitudes results in a triangular wave with odd amplitudes of 1/n², in the frequency domain.

Another way to visualize this is the path on the real, imaginary plane. In the ideal case there is one frequency that rotates from 0 to 2pi, but with a triangular signal it is the sum of odd frequencies rotating, as plotted in the animation below. This representation is borrowed from explaining Fourier transforms by drawing circles.

Figure 4: An animation of the correlation signal as a sum of circles. The first harmonic rotates once, while the third 3 times, and so on. This builds up to a triangular correlation signal, as seen in figure 1.

In the case where there are four raw frames there are not enough samples to fully reconstruct the cross-correlation signal. In signal processing terms the Nyquist sampling frequency is two (4/2), therefore the higher frequencies alias around the Nyquist sampling frequency. This is illustrated in the animation below

Figure 5: A triangular wave, and its Fourier transform. The Nyquist frequency is half the sample rate, so with 4 samples the Nyquist frequency is 2. The higher order harmonics alias around the Nyquist frequency, leading to non-linearities in phase.

This means the higher frequencies introduce a wiggle onto the first frequency, causing the error observed in figure 1. This is the cause of the harmonic error, the higher frequencies aliasing onto the primary harmonic and causing non-linearities in the phase measurements.

Reducing Harmonic Aliasing

There are four broad approaches to solving/mitigating this error in iToF cameras:

• Optimizing the Nyquist frequency
• Adjusting the duty cycle of the laser
• Calibrating away the error with a look up table or polynomial fit
• Other techniques to cancel the error

Optimize the Nyquist frequency

The simplest method to reduce error caused by harmonic aliasing is to use double the number of samples, than the highest frequency in the cross-correlation signal. This means 10 or 14 frames would be required, depending on highest frequency present and desired error. 14 samples is unreasonable because of the decrease in depth frame rate for most applications. The better approach is to use an odd number of samples as this avoids directly aliasing the higher frequencies, using 3 or 5 samples is best to minimize the error.

Duty cycle Adjustment

Cross-correlation in time is a multiplication in frequency (Fourier) space. If we can modify one of the signals to have a zero at a given location in its Fourier transform the resulting cross-correlation will also have a zero at that location (zero times anything is always zero). The animation below shows the Fourier transform change with duty cycle. At 29% there is a zero for the 3rd harmonic. By setting the duty cycle of the laser we can reduce the error due to harmonic aliasing, but not completely eliminate it. This concept is further explored by Payne et al. [2]

Figure 6: The Fourier transform of a square wave changes with duty cycle, eliminating some of the harmonics.

Look Up Table (LUT) or Polynomial Fitting

A common approach is to calibrate the non-linear phase by using a Look-up-table or curve fitting to model the error and know what offset to apply. The use of a LUT was described by Kahlmann et al [3], and spline fitting by Lindner et al [4]. to use calibration to remove this error.

This calibration method works when it is the only source of error in iToF measurements. However other measurement errors caused by motion blur, multi-path interference introduce their own offsets/errors. When the harmonic calibration is then applied on this data the calibration can increase error instead of decreasing it!

Other Techniques

There have been a few novel techniques to reduce or remove this error in ToF cameras by changing the modulation signal and/or depth calculation process. Payne et al. [5] changed the phase of the sensor modulation signal by +/- 45 degrees during the integration time of each raw frame. This cancelled out the contribution by the 3rd harmonic but reduced the signal power by 10% which decreases precision. This was expanded by Peters et al. [7] by using more phase steps.

Streeter [6] changed the phase steps from [0, 90, 180, 270] to [0, 90, 120, 210] and kept the calculation of phase the same. This has the affect of cancelling out the 3rd harmonic (so higher accuracy) but at a slightly reduced precision.

Feigin et al. [8] took a different approach and modelled the harmonic error as multi-path interference. By taking measurements at multiple modulation frequencies the harmonic contribution could be computed and the correct phase recovered.

About Chronoptics

Chronoptics are experts in designing indirect ToF depth sensing modules, focusing on the processing from the image sensor to the point cloud, and tailoring each module to output the best point cloud for a give application. For advice on correcting harmonic aliasing in your ToF sensor contact us at hello@chronoptics.com

References

[1] Lange, R., & Seitz, P. (2001). Solid-state time-of-flight range camera. IEEE Journal of quantum electronics, 37(3), 390–397

[2] Payne, A. D., Dorrington, A. A., & Cree, M. J. (2011, June). Illumination waveform optimization for time-of-flight range imaging cameras. In Videometrics, Range Imaging, and Applications XI (Vol. 8085, p. 80850D). International Society for Optics and Photonics.

[3] Kahlmann, T., Remondino, F., & Ingensand, H. (2006). Calibration for increased accuracy of the range imaging camera swissranger. In Proceedings of the ISPRS Commission V Symposium’ Image Engineering and Vision Metrology’ (Vol. 36, pp. 136–141). Isprs.

[4] Lindner, M., Schiller, I., Kolb, A., & Koch, R. (2010). Time-of-flight sensor calibration for accurate range sensing. Computer Vision and Image Understanding, 114(12), 1318–1328.

[5] Payne, A. D., Dorrington, A. A., Cree, M. J., & Carnegie, D. A. (2010). Improved measurement linearity and precision for AMCW time-of-flight range imaging cameras. Applied optics, 49(23), 4392–4403.

[6] Streeter, L., & Dorrington, A. A. (2015). Simple harmonic error cancellation in time of flight range imaging. Optics letters, 40(22), 5391–5394.

[7] Peters, C., Klein, J., Hullin, M. B., & Klein, R. (2015). Solving trigonometric moment problems for fast transient imaging. ACM Transactions on Graphics (TOG), 34(6), 1–11.

[8] Feigin, M., Whyte, R., Bhandari, A., Dorrington, A., & Raskar, R. (2015). Modeling “wiggling” as a multi-path interference problem in AMCW ToF imaging. Optics express, 23(15), 19213–19225.