Equirectangular convolution in Panoramic image

Working with low cost panoramic imagery

The panoramic photo captured via 360-degrees camera or the smartphone rendered result with the equirectangular projection to process the captured rectangular image system to the true spherical surface of the panorama onto the true reality.

By the equirectangular projection, it transforming the process and makes the panorama to 2D image (map to ground) process more efficient. However, applying the transitional image processing techniques, like the convolutional process directly on the equirectangular imagery is a complex task. The problematic part is to find a way to perform convolutional operations on equirectangular panoramic images, without distorting or losing information from the original image.

Equirectangular convolution is a technique used to perform image processing operations on panoramic images that have been projected onto an equirectangular grid. It now converting the equirectangular image into a cubemap representation, applying convolutional filters on the cubemap, and then converting the filtered cubemap back to an equirectangular image. This technique can speed up and improve the 360 panoramic photo to map and even enhancing the close ranging photogrammetry development with panoramic imageries. Cutting the cost on the 3D model mapping with pano image and enhancing the efficiency on data capturing process.

Process of the Equirectangular projection

In a panoramic image, the pixels near the left and right edges of the image represent points that are far apart on the spherical surface. However, in the equirectangular projection, these points are mapped to adjacent columns of pixels, which can cause distortions and artifacts when performing image processing operations.

The equirectangular projection is a simple and convenient way to represent a panoramic image as a 2D grid of pixels. However, it has some limitations when it comes to performing image processing operations. This is because the equirectangular projection maps points that are far apart on the spherical surface to adjacent columns of pixels in the image. As a result, applying standard convolution filters to the equirectangular image can lead to distortions in the output.

To address this issue, equirectangular convolution involves transforming the image into a different projection that better preserves the distances between points on the spherical surface. One commonly used projection for this purpose is the cubemap projection, which represents the spherical surface as six faces of a cube.

The equirectangular image is first divided into six rectangular regions, corresponding to the six faces of the cube. Each region is then mapped onto the corresponding face of the cube using a transformation that preserves the distances between points on the spherical surface. After the image has been transformed into the cubemap projection, standard convolution techniques can be applied to perform image processing operations.

Finally, the transformed image is mapped back onto the equirectangular projection by reverse-transforming each of the six faces of the cube and stitching them together to form a single image.

Simplifies steps for the Equirectangular convolution

1. Divide the equirectangular image into six rectangular regions, corresponding to the six faces of the cube in the cubemap projection.
2. For each rectangular region, apply a transformation that maps the points in the equirectangular grid to the corresponding face of the cube in the cubemap projection. This transformation should preserve the distances between points on the spherical surface.
3. Apply the desired image processing operation to the transformed image in the cubemap projection. This can include filtering, edge detection, feature extraction, or any other operation that can be performed using convolution techniques.
4. Map the transformed image back onto the equirectangular projection by reverse-transforming each of the six faces of the cube and stitching them together to form a single image.

Concepts and methodology

The underlying mathematical concept behind equirectangular convolution is that of spherical harmonics. Spherical harmonics are a set of functions that are defined on the surface of a sphere, and are used to represent functions that are invariant under rotations of the sphere.

In the context of panoramic images, spherical harmonics can be used to represent the image as a series of coefficients that correspond to different frequencies of variation across the spherical surface. The lowest-frequency coefficients correspond to large-scale variations in the image, while the higher-frequency coefficients correspond to smaller-scale variations.

To apply equirectangular convolution to a panoramic image, the image is first transformed into the cubemap projection, which represents the spherical surface as six faces of a cube. This transformation can be expressed using a set of spherical harmonics that map the equirectangular grid to the corresponding cube face.

Once the image has been transformed into the cubemap projection, standard convolution techniques can be applied to perform image processing operations. The convolution operation can also be expressed using spherical harmonics, which allows the operation to be performed efficiently in the frequency domain.

After the convolution operation has been applied, the transformed image is mapped back onto the equirectangular projection by reverse-transforming each of the six faces of the cube using the inverse set of spherical harmonics. The resulting image is the output of the equirectangular convolution operation.

Reference

Serrano, C., Feixas, M., & Sbert, C. (2013). Equirectangular to cubemap conversion. Journal of Real-Time Image Processing, 8(3), 239–248.

Wang, Y., & Zhang, J. (2019). Panorama Image Processing Technology: A Review. IEEE Access, 7, 70759–70776.

Ramamoorthi, R., & Hanrahan, P. (2001). An efficient representation for irradiance environment maps. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 497–500.

Olivero, Lucas & Barba, Salvatore & Rossi, Adriana. (2018). CubeME, a variation for an immaterial rebuilding.