Fourier Image Lab
Published in

Fourier Image Lab

Enhanced Dynamic Range

In this article I’d like to briefly describe the work that I did at OmniVision.

This patent describes the system:

HDR quarter resolution image captured using a sensor with variable pixel exposure was processed through HDR ISP pipeline and upscaled to full resolution and further enhanced by overlaying high frequency components from a full resolution image sequentially captured using same sensor configured to linear mode. The HDR image was merged on the sensor, tone-mapped in ISP and aligned and fused with non-HDR (or LDR) image using on GPU.

Image sensor has a Bayer pattern and a variable exposure pattern. Each of 4 blocks of 2x2 size inside a larger 4x4 block were assigned different integration times and combined using exposure merge algorithm on sensor.

2 steps of acquiring HDR and LDR images

Step 1 of the diagram above shows acquisition process of HDR image with using sensor with 4 spatially varying exposures. After merging the HDR image was processed and tone-mapped in ISP and saved to memory.

Since in merging stage 4 blocks of 2x2 blocks ere combined into single 2x2 block a lot of spatial resolution was lost and aliasing appeared.

To restore the resolution of HDR image computational photography approach was introduced in step 2, where the sensor pixels were integrated with same exposure time and full resolution image was processed through ISP and also saved to memory for further fusion with HDR image.

Frequency guided fusion of HDR and LDR images

Fusion method calculates high frequency component of the LDR full resolution image and rendering it above the up-scaled LDR image.

The exposure merge was done on the sensor chip and fusion was done on GPU.

system component diagram

It is also possible to do exposure merge on ISP chip, but it will increase data through-put between sensor and ISP by 4x

The system showed quite amazing results over-performing iphone and pixel phones of same generations on both dynamic range and detail reproduction also featuring no ghost artifacts even if fast moving metronome was placed against right window.

I have presented the system demo at poster and demo sessions at EI2017 and CES2017.

Electronic Imaging comity put placed the invention on the same page as Boyd Fowler’s and Brian Cabral’s talks!

demo session whole setup live, ready to demonstrate
The invention attracted has a lot of interest to work at omnivision
result

The concept was developed at with OmniVision Singapore team in collaboration with A research institute and involved cross-team collaboration and efforts from teams in US and China.

Alignment of the tone-mapped and non-tone-mapped image was done on GPU and was developed by OmniVision team in Shanghai:

Speed of ransac was reasonably increase by adding constraints on the set of control points and removing outliers based on user experience and memory limitations. The method is described in the following invention:

Management of ghosting artifacts was greatly improved by OVT US team.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store