AutoSweep: Recovering 3D Editable Objects from a Single Photograph
Overview of the paper “AutoSweep: Recovering 3D Editable Objects from a Single Photograph” by X. Chen et al.
Designing simple 3D models of basic everyday objects in games can be quite tedious and time-consuming. There can be thousands of such interactive objects in VR games like Half Life Alyx and this can quickly add up the time and cost of developing this game.
So today I want to cover a paper that looks to use AI for automating some aspects of this design process in order to save time and other resources. The authors have introduced a novel method named AutoSweep that can recover 3D Editable Objects from a Single Photograph.
Their Deep Learning framework integrates with Unity engine to produce 3D objects from a single input image of that object. As you can see below, this method not only recovers the shape of the objects, but also the surface texture, making it very convenient to add a replica of a real-life object in a game.
Their framework includes using a Mask-RCNN to perform instance segmentation on images to focus on the object of interest. They combine this information with an edge map to obtain the body and profile information of that object, which is then labeled according to generalized primitives, which means basic building blocks like cylinders and cuboids. These building blocks can then be transformed to 3D shapes with a sweeping procedure commonly used in CAD design.
They show their method is also useful in converting hand-drawn sketches to 3D editable objects, which can be another useful tool in game development. If you want to see more results from this work, I have linked the full paper and the code in the description below.
Useful Links
Thank you for reading. If you liked this article, you may follow more of my work on Medium, GitHub, or subscribe to my YouTube channel.
