Build 3D From Anything! VR & Metaverse World Building
In this article, we’ll be discussing a novel implementation of 3D object building using AI. We’ll talk about how this may affect other technologies, like virtual reality and video animations, so keep reading. Before we dive into this topic, if you would like to check the video version of this article instead, then check out the link below.
Traditionally, 3D modelling requires tedious tasks involving artistic skills and technical know-how. Expert Modellers would usually use software like Blender to build out 3D models for use in video animations or game design, which usually takes hours or even days. Popular 3D examples can be seen in your favourite movies. Remember the first time you watched the incredible graphics of The Avatar, or realistic movements of the Hulk in The Avengers ? These results usually require costly budgets and dedicated time as mentioned before.
We’ll begin with one company that’s at the forefront of this space, Nvidia. Nvidia is a multinational company mainly known for designing graphics processing units and applications for high performance computing. The company released an AI solution for quickly modelling 3D objects in just a matter of seconds! That’s mind-blowing, right ? The AI product which they call InstantNerf, is a machine learning neural network model that takes in as input, either a 2D image or video of a real world object or landscape. This is then turned into a 3D representation in just seconds! The specialized neural network model used for this is called Neural Radiance Field or Nerf for short. The first release of Nerf was slow and could take hours to train, InstantNerf as the name suggests, does the training almost instantly. This is definitely a game changer! The neural network works by filling out missing parts of the 2D image used as input to produce a complete 3D version. It’s even more incredible when you think about all the other features like lighting, texture, shadows and viewpoints the AI model has to learn. In the movie industry, complex 3D scenes usually have huge memory utilizations and may impact work efficiency. On the other hand, Nerf models store the entire scene with much smaller memory consumption.
InstantNerf makes 3D modelling more efficient for game designers and animation artists in scenarios where designing a digital version or modification of a real object is required. The solution is also appealing to people with little to no design skills who want to create digital objects. This type of fast 3D modelling for virtual reality use cases can be very handy and at a lesser cost than the status quo. Unlike other 3D models, InstantNerf can also generate 3D replicas of the environment within which an object was captured. There are endless possibilities with this. A practical VR example of InstantNerf implementation can be done at Museums or tourist attractions where parts or whole of the venue is transferred digitally. This will ensure that these establishments can reach a wider audience. The great thing about digitizing real objects is that we’re still in the early stages, and over time, results are evidently going to improve. You can imagine 5 years from now, when 3D world building and virtual reality technology are producing high quality models at an instantaneous rate, that would be amazing, wouldn’t it? Another interesting implementation could be for custom game modelling where players could decide to use models of their living rooms in a pluggable way. Clearly big game companies like Rockstar Games who produce the GTA series or Maxis, the creators of the Sims games would have to accommodate such use cases in the future. Most importantly, you can finally live your Ready Player One fantasies in a few years so hold tight.
Aside from InstantNerf, there are other AI-based 3D modelling solutions that might not be as fast, however they produce state of the art 3D model results in certain areas. We’ll just rapidly go through a few of them here so you are aware of other AI developments in the 3D generation space. First in the list is BANMo, a neural network solution for rendering movements of animals or people. It captures exact movements in 3D space which are meant to be as realistic as possible compared to previous attempts. Certain poses or movements tend to be difficult for AI models to replicate and BANMo reaches state of the art results on most of these. Another AI model is I M Avatar. This model is for 3D reconstruction of the head producing quality texture and realistic facial movements. Most difficult parts to model are mouth interiors or hair texture which I M Avatar improves upon. Third in the list is SHAPY, an AI-based solution for estimating actual body shapes of a person. This could perhaps be used in the fashion industry for virtual clothes fitting. Finally, HumanNerf which is a Nerf-based model for human movements also improves on 3D rendering of complex human movements. It also fully models parts of the body that might not be visible in the real world version used as input. With all these exciting innovations coming up in 3D modeling, it’s hard to contain excitement for what the future holds.
What other implementations do you think later versions of these models would have ? Let me know your thoughts in the comments.
Thank You!
Resources:
InstantNerf: https://developer.nvidia.com/blog/getting-started-with-nvidia-instant-nerfs/
BANMo: https://banmo-www.github.io/
I M Avatar: https://ait.ethz.ch/projects/2022/IMavatar/
SHAPY: https://shapy.is.tue.mpg.de/
HumanNerf: https://grail.cs.washington.edu/projects/humannerf/
