Fourier Feature Mapping Enables MLPs to Learn High-Frequency Functions in Low-Dimensional Domains

A multilayer perceptron (MLP) is a class of feed-forward neural networks with many perceptrons organized into layers. MLPs can solve problems stochastically and provide approximate solutions for complex problems such as fitness approximation.

A recent trend in computer vision is to replace the traditional discrete representations of objects, scene geometry, meshes and voxel grids with continuous functions parameterized by coordinate-based MLPs, which take low-dimensional coordinates as inputs and are trained to represent high-dimensional outputs (i.e. shape, density, colour, etc.). Coordinate-based MLPs are now widely used to represent a variety of visual signals, including images and 3D scenes.

Google Research and University of California researchers however found that standard MLPs are poorly suited for low-dimensional coordinate-based vision and graphics tasks and have difficulty learning high-frequency functions. To overcome these issues, they have proposed the use of Fourier feature mapping with MLPs in order to learn high-frequency functions in low-dimensional problem domains.

The researchers first leverage kernel regression with a neural tangent kernel (NTK) — which can describe the evolution of deep artificial neural networks during their training by gradient descent — to prove the deficiency of standard MLPs for low-dimensional coordinate-based vision and graphics tasks. Because the training points are distributed with uniform density, they then attempt to make the composed NTK shift-invariant. Finally, as a “wider” kernel with a slower spectral achieves faster training convergence for high-frequency component, their efforts to control the bandwidth of the NTK lead to the improvement of training speed and generalization.

Figure (a) visualizes the NTK function for a 4-layer ReLU MLP with one scalar input, where the kernel does not have a strong diagonal, making it perform poorly for kernel regression in low-dimensional problems. A basic input mapping makes the composed NTK stationary (b). A Fourier feature input mapping tunes the composed kernels’ width © and higher-frequency mappings, resulting in wider spectra of the composed kernels, enabling faster convergence for high-frequency components.

Effects of Fourier features on network convergence

Fourier features improve the results of coordinate-based MLPs for a variety of high-frequency low-dimensional regression tasks, i.e. image regression, 3D shape regression, MRI reconstruction and inverse rendering.

In experiments, the proposed Fourier feature mapping approach dramatically improves coordinate-based MLP performance across all tasks, with random Gaussian features performing best. The results illustrate the increasingly popular technique of using coordinate-based MLPs to represent 3D shapes in computer vision and graphics by using a simple mapping strategy.

The paper Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains is on arXiv.

Author: Hecate He | Editor: Michael Sarazen

We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

Thinking of contributing to Synced Review? Synced’s new column Share My Research welcomes scholars to share their own research breakthroughs with global AI enthusiasts.

Need a comprehensive review of the past, present and future of modern AI research development? Trends of AI Technology Development Report is out!

2018 Fortune Global 500 Public Company AI Adaptivity Report is out!
Purchase a Kindle-formatted report on Amazon.
Apply for Insight Partner Program to get a complimentary full PDF report.