Crafting Realistic Renderings with PyTorch3D
Why do we need to render 3D models, you ask🙃? Imagine a world where architectural designs remain trapped within blueprints, where characters in a video game lack depth and dimension, and where scientific simulations fail to offer a visual understanding of complex phenomena. Rendering steps in as the technological magician that transforms these static representations into dynamic, vibrant visuals. It provides us with the means to simulate lighting, shadows, materials, and intricate details, creating an immersive experience that engages our senses and sparks our curiosity.
How?? 🏗️
Imagine rendering as a magical transformation that turns digital 3D models of the things we see into pictures. It’s like when you take a photo with your camera, but it’s done inside a computer. Instead of using a camera, we use 3D models, which are like digital sculptures. To make this happen, we often use a cool technique called ray tracing. It’s like sending beams of light into the digital world to create super realistic images.
Check out Figure above for a clear picture of how ray tracing works. In this example, we have a digital world with a single 3D sphere — think of it like a digital ball. Now, to create the image of this sphere, we play a cool game: for every tiny dot in the picture, we imagine a special beam of light shooting out from the camera, through that dot. If this light beam hits the sphere, the sphere’s color shows up on that dot. But wait, there’s more — we’re not just thinking about the surface, we’re also considering how far inside the sphere this beam goes. This helps us make sure things in the back don’t cover up things in the front. It’s like creating a colorful 3D puzzle with light!
Now, lets go deeper 🤔, Idea and Code
Think of rendering like preparing a delicious pizza. There are two main steps: making the crust and adding the toppings. Similarly, rendering has two important stages: rasterization and shading.
First, we have rasterization — this is like laying out the pizza crust. In this step, we figure out which shapes from our digital world match up with each tiny spot in the picture. It’s like fitting puzzle pieces together.
Next, we move on to shading, which is like putting the tasty toppings on the pizza. Here, we take the shapes we found in the previous step and figure out the exact colors they should be on the picture. It’s like giving each part of the pizza the perfect flavor.
So, just like making a yummy pizza involves both the crust and the toppings, rendering involves rasterization and shading to create awesome images!
Get ready for a coding adventure using PyTorch3D! We’re turning a mesh model into a masterpiece with realistic rendering.
- Setup environment with anaconda, pytorch and open3D
conda create -n python3d python=3.7
source activate python3d
# https://pytorch.org/
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
conda install pytorch3d -c pytorch3d
pip install open3d jupyterlab
It gonna take some while, be patient🤒
2. First, we need to import all the Python modules that we need:
import open3d
import os
import sys
import torch
import matplotlib.pyplot as plt
# Util function for loading meshes
from pytorch3d.io import load_objs_as_meshes
# Data structures and functions for rendering
from pytorch3d.renderer import (
look_at_view_transform,
FoVPerspectiveCameras,
PerspectiveCameras,
PointLights,
Materials,
RasterizationSettings,
MeshRenderer,
MeshRasterizer
)
from pytorch3d.renderer.mesh.shader import HardPhongShadersys.path.append(os.path.abspath(‘’))
3. We need to load the mesh that we are going to use. The cow.obj file contains a mesh model for a toy cow object, can be downloaded from here:
!wget https://dl.fbaipublicfiles.com/pytorch3d/data/cow_mesh/cow.obj
!wget https://dl.fbaipublicfiles.com/pytorch3d/data/cow_mesh/cow.mtl
!wget https://dl.fbaipublicfiles.com/pytorch3d/data/cow_mesh/cow_texture.png
#Load meshes and visualize it with Open3D
mesh_file = “./da/cow_mesh/cow.obj”
print(‘visualizing the mesh using open3D’)
mesh = open3d.io.read_triangle_mesh(mesh_file)
open3d.visualization.draw_geometries([mesh],
mesh_show_wireframe = True,
mesh_show_back_face = True,
)
# Set paths
DATA_DIR = “./data”
obj_filename = os.path.join(DATA_DIR, “cow_mesh/cow.obj”)
device = torch.device(‘cuda’)
# Load obj file
mesh = load_objs_as_meshes([obj_filename], device=device)
4. Time to Dive In: Cameras and Lights Take Center Stage! We work our magic with look_at_view_transform, turning camera distance, angles, and more into R and T matrices. These gems define where our camera goes. And don’t miss the ‘lights’ at [0.0, 0.0, -3.0]!
R, T = look_at_view_transform(2.7, 0, 120)
cameras = PerspectiveCameras(device=device, R=R, T=T)
raster_settings = RasterizationSettings(
image_size=512,
blur_radius=0.0,
faces_per_pixel=1,
)
lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])
renderer = MeshRenderer(
rasterizer=MeshRasterizer(
cameras=cameras,
raster_settings=raster_settings
),
shader = HardPhongShader(
device = device,
cameras = cameras,
lights = lights
)
)
images = renderer(mesh)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, …, :3].cpu().numpy())
5. Here, we tweak things — dialing down ambient components to almost zero (specifically, 0.01). With the point light behind and ambient off, our rendered object goes incognito, soaking in the shadows.
lights.location = torch.tensor([0.0, 0.0, +1.0], device=device)[None]
materials = Materials(
device=device,
specular_color=[[1.0, 2.0, 0.0]],
shininess=10.0,
ambient_color=((0.01, 0.01, 0.01),),
)
images = renderer(mesh, lights=lights, materials = materials)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, …, :3].cpu().numpy())
6. Stepping into Our Next Experiment: Camera Dance and Illumination Remix! We spin the camera and relocate the light for a spotlight on the cow’s face. Keep an eye on the ‘shininess’ set at 10.0 — a key player in the Phong lighting model. Plus, the ‘specular_color’ [0.0, 1.0, 0.0] spills a touch of dazzling green shine!
R, T = look_at_view_transform(dist=2.7, elev=10, azim=-180)
cameras = FoVPerspectiveCameras(device=device, R=R, T=T)
lights.location = torch.tensor([[2.0, 2.0, -2.0]], device=device)
materials = Materials(
device=device,
specular_color=[[0.0, 1.0, 0.0]],
shininess=10.0
)
images = renderer(mesh, lights=lights, materials=materials, cameras=cameras)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, …, :3].cpu().numpy())
7. We are going to change specular_color to red and increase the shininess value
materials = Materials(
device=device,
specular_color=[[1.0, 0.0, 0.0]],
shininess=20.0
)
images = renderer(mesh, lights=lights, materials=materials, cameras=cameras)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, …, :3].cpu().numpy())
At here, You might be inclined to adjust numerical values throughout the tutorial for seamless 3D rendering using PyTorch3D. Until our next encounter, stay tuned for more exciting AI tutorials!
