Rebuild Your Home in 3D with Gaussian Splatting

Artificial Intelligence is probably one of the major fields in our days, it is being used for basically anything from driving cars to healthcare being able to even prevent blindness (by Teresa Lobo), recently new 3D reconstruction method has been presented. SNGULAR and its Artificial Intelligence team wanted to learn more about the last updates on 3D reconstruction technologies.

Many of the SOTA methods available nowadays for 3D reconstruction require a high amount of CPU/GPU usage to process the scene or to render it, some of them even both. The new method presented on SIGGRAPH 2023 Gaussian Splatting comes to the rescue for some affordable times of both, inference and render.

The last few years we have seen a good amount of articles about 3D reconstruction like the method NeRF (Neural Radiance Fields), which achieved a really good 3D reconstruction with highly accurate scenes and models that can be used on a wide amount of fields, like virtual tourism, immersive gaming, and architectural visualization. This however comes at a price, NeRF is time consuming, requires a high amount of memory and storage and it is limited to static scenes. Gaussian Splatting solves some of these problems making it probably the best 3D reconstruction method nowadays.

This article intends to apply this method to the specific area of interior reconstruction our main objectives are:

• Understanding how 3D reconstruction and specifically Gaussian Splatting works.
• Running a small demonstration to reconstruct your own scene without leaving your home.

So dive in with me and find how to get a 3D reconstruction of your home.

Understanding 3D reconstruction

Reconstruction begins with a few photos taken with a camera seen from different viewpoints of the scene. These images will serve as the input data for the reconstruction. The reconstruction’s main purpose is to recover its spatial structure of the scene, and make us able to view it from any angle.

One of the most important key concepts in 3D reconstruction is feature matching, where distinctive points or features in the 2D images are identified and matched across multiple views. Using the correspondences between two images it is possible to estimate the relative position of the cameras, with which the triangulation of the 3D position of these features will be calculated. Gaussian Splatting uses cloud points from a different method called Structure from Motion (SfM) which estimates camera poses and 3D structure by analyzing the movement of a camera over time.

3D reconstruction finds applications in a wide range of fields. In archaeology, it helps researchers document and analyze historical sites and artifacts. In architecture, it aids in creating accurate building models for design and renovation. Medical imaging relies on 3D reconstruction to visualize complex anatomical structures. Entertainment industries use it for realistic CGI and video game development. Moreover, 3D reconstruction plays a crucial role in autonomous robotics, enabling robots and drones to perceive and navigate their environments. As technology continues to advance, 3D reconstruction is poised to revolutionize how we interact with the physical and digital worlds, opening up new possibilities in virtual and augmented reality, manufacturing, and beyond.

Introducing Gaussian Splatting

In SNGULAR we have decided to give people a small glimpse of this new and revolutionary technology.

Gaussian Splatting is a new revolutionary method of 3D reconstruction which tries to solve some of the problems other methods have like the big processing times. For this method instead of representing the scene as a volume or a mesh, their method uses the world as a 3D canvas, drawing small strokes until the scene looks like the input images. One of the main advantages of this method is that it allows to reconstruct not only still images but also moving scenes

Image from the paper

Starting with a cloud of points, it creates a small stroke for each point, then, it slowly changes the size, position, rotation and color of each stroke, optimizing those parameters each iteration using Stochastic Gradient Descent techniques to reduce the error between the strokes and the input images. That is not enough though, as some areas can require more, or less strokes to be drawn correctly so every 100 iterations a densification process will be applied, splitting strokes in zones that require more detail, and removing them in zones where they are no longer required.

Image from the paper

Once the optimization reaches 7000 and 30000 iterations in the code obtained in the official repository by default a new ply file will be created containing all the strokes required to represent the scene, we just need to load the file in a compatible visualizer and we are finished!

Advantages of Gaussian Splatting

Now that we have a general idea of how gaussian splatting works we can summarize the best points of this method.

• Low training times: A scene can be trained in barely one hour of computation time.
• High render fps: Gaussian Splatting is able to render scenes in real time up to 100 fps.
• Dynamic scenes: Not only static scenes like other methods, Gaussian Splatting is able to reconstruct moving scenes.

Tutorial: How to recreate you home

For this small tutorial we will be using Ubuntu 20.04. A CUDA capable graphic card is required, we used a GTX 1080.

The first step is to get your images. The easiest way is to make a video of the area you want to reconstruct.

Try to:

• Get a video not too large or short, about 1 minute can work.
• Try to make smooth movements and from different angles.
• Use a resolution of around 1920x1080.
• Focus on an object and turn around it making sure the background also appears.
• Do it with good illumination conditions, otherwise it can be difficult to get matches for the software.

Once we have our video we can extract the frames using ffmpeg. For that in a terminal we will execute:

ffmpeg -i <Path to video file> -r X Example/input/image%03d.jpg

You will need to replace the X in -r for a number depending on the length of the video, you can play with this parameter until you get ideally between 100 and 300 images. The more images the more the quality of the point cloud, but also you will need a Graphic Card with more VRAM. For our video we used an X of 2, so the final command would be:

ffmpeg -i video.mp4 -r 2 Example/input/image%03d.jpg

After a few seconds we will have our images prepared to be processed.

As mentioned before, Gaussian Splatting takes as input a cloud point calculated with SfM along with the images; they recommend using either Colmap or NeRF to achieve this. Using Colmap is a very straightforward method. For that in ubuntu we can simply download Colmap using apt.

sudo apt-get install colmap

If you are running in a server (with no screen) it may be needed to build colmap from scratch though, you can find how to do it here.

With colmap installed we can proceed to generate the cloud of points, Gaussian Splatting repository provides a script to do that easily, so firstly we will download the repo using git.

If github is not installed in your system you can do it with:

sudo apt-get install git

Then we can run

git clone https://github.com/graphdeco-inria/gaussian-splatting - recursive

Once downloaded we proceed to prepare the environment, we will be using conda, in case you dont have it installed you can find it here.

We install the requirements for Gaussian Splatting:

cd gaussian-splatting
conda env create - file environment.yml

When it’s finished we can activate the environment

conda activate gaussian_splatting

Then we start to process the cloud of points

python convert.py -s <PATH>/<Foldername>/

Once it is finished we will have a directory tree similar to this one:

<location>
| — -input
| | — -<image 0>
| | — -<image 1>
| | — -…
| — -distorted
| — -database.db
| — -sparse
| — -0
| — -…

Then we can start the training with the following command!

python train.py -s <path to COLMAP or NeRF Synthetic dataset>

This process will take a while, when it is done, we will have the results inside the folder output, there will be two folders with the results of iterating through it 7000 times and 30000 times. We can proceed then to visualize the results using a visualizer.

The easiest way to do that is with a colab made by “camenduru”:

https://colab.research.google.com/github/camenduru/gaussian-splatting-colab/blob/main/gaussian_splatting_viewer_colab.ipynb#scrollTo=QIUkLk6_lnFp

You just need to run the cell, open the http link looking something like this “https://<Random-String>.trycloudflare.com” and drag the file point_cloud.ply file found in the output folder. And there it is! Now you can see your reconstructed home!