This AI Could be Used To Design The Next Assassin’s Creed Game!
Overview of the paper “Next-Best View Policy for 3D Reconstruction” by D Peralta et al.
Games like Assassin’s Creed that try to accurately recreate certain types of real-life architecture require a lot of design work. In order to create a 3D replica model for such architecture, we need to start by collecting thousands of images or 3D scans of real-life historical monuments.
Usually, drones are used to capture this information using LiDAR scanning or depth cameras by manually flying them around the monument capturing this information from different viewpoints. Turns out, deciding the path of this drone is something that we can actually optimize using Machine Learning. This is important because we want to capture as much surface of the monument as possible using very few images and in a short amount of time, so that the 3D reconstruction process is simpler and the replica is very accurate.
Today’s paper, titled “Next-Best View Policy for 3D Reconstruction” explores this method where they propose a Reinforcement Learning method to decide the path of the drone.
Scan-RL Model Architecture
In this work, synthetic data is used for training the RL agent using Unreal Engine. A 3D house designed manually is rendered as a 2D RGB image from the viewpoint of a drone. This image is fed to a deep neural network as state observation of this environment. The neural network has to learn a policy to decide what the next best view is, so we can say it is essentially learning to control the path of the drone capturing these images. In order to give rewards to this policy network, a 3D reconstruction module is used to reconstruct a point cloud from currently captured depth images.
The accuracy of this 3D point cloud reconstruction decides whether the reward will be positive or negative. Hence, using any off-the-shelf Reinforcement Learning algorithm, this network can be trained to optimize this scanning process.
This method is better than simply scanning a house naively using a circular path because it is also able to account for occluded or hidden parts of the monument. For example, in this case the drone is able to account for the hidden parts of the house that are occluded by the roof. As shown in the paper, this RL based scanning method covers 97% of the house compared to just 87% for manual scanning.
Let’s just take a moment to appreciate how much research work needs to go behind the development of such big AAA games just so that we can solve very specific problems like the one I have covered here. It’s not something we appreciate enough when we simply look at the end-product, so it’s truly incredible to come across this paper.
Useful Links
Thank you for reading. If you liked this article, you may follow more of my work on Medium, GitHub, or subscribe to my YouTube channel.
