AI Transforms RGB-D Images Into an Impressive 3D Format
This is an updated version.
In 2018, Facebook introduced a machine learning-based 3D photo feature which enabled users to generate an immersive 3D image from any ordinary photo. This was an “almost perfect” 3D image generator — yes it would grab your friends’ attention, but the background renderings were pretty blurry. Now, a research group from Virginia Tech, National Tsing Hua University and Facebook has introduced a game-changing algorithm that generates impressive 3D photos from a single RGB-D (colour and depth) image.
Unlike depth-based warping techniques that produce gaps or stretch existing image content or the Facebook 3D photo approach which can produce unrealistic surface textures, the proposed method leads to much more photorealistic results. Taking an RGB-D image as input, researchers use a Layered Depth Image (LDI) technique with explicit pixel connectivity as the underlying representation. The learning-based inpainting model synthesizes new colour or depth textures and structures into the occluded regions of the image in a spatial context-aware manner. The generated 3D photos can be rendered with motion parallax using standard graphics engines.
Researchers compared the new approach with MPI based methods on the RealEstate10K dataset and quantified performance using the common SSIM and PSNR image similarity tests on the synthesized target views and the ground truth. The LPIPS (Learned Perceptual Image Patch Similarity) metric was also included to quantify the performance of the generated view compared to human perception. The proposed method showed similar performance on SSIM and PSNR metrics, while LPIS scores indicated the synthesis views exhibit better perceptual quality. Researchers validated the method on a wide variety of everyday scenes, where it produces considerably fewer visual artifacts compared with state-of-the-art novel view synthesis techniques.
The paper 3D Photography using Context-aware Layered Depth Inpainting is on arXiv. This research’s GitHub page is here. The research group has also introduced a Chrome extension that can add depth parallax on images from Instagram profile pages.
Author: Yuqing Li | Editor: Michael Sarazen
Thinking of contributing to Synced Review? Synced’s new column Share My Research welcomes scholars to share their own research breakthroughs with global AI enthusiasts.
We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.
Need a comprehensive review of the past, present and future of modern AI research development? Trends of AI Technology Development Report is out!
2018 Fortune Global 500 Public Company AI Adaptivity Report is out!
Purchase a Kindle-formatted report on Amazon.
Apply for Insight Partner Program to get a complimentary full PDF report.
