How We Turn Physical Products into Realistic 3D Models for AR

Sam Cribbie
Dec 6, 2017 · 7 min read
Magnolia Market’s “Preview in your home” functionality.
Press play to interact with the 3D model.

Available Methods

There are a couple different options to consider when it comes to creating a 3D model from a real world object.

Option One: Photogrammetry

Using a series of photos taken of a real world object, photogrammetry software can create an accurate high-density mesh of most objects. The “mesh” is a group triangles that define the shape of the object. Along with the mesh, it will also create texture images that define the colour of the object.

Bad Candidates

  • smooth surface
  • transparent
  • reflective/ shiny
  • featureless ( i.e. one solid colour; no visual patterns to detect)

Good Candidates

  • rough surface
  • opaque
  • lots of visual patterns on surface (i.e. colour changes, texture, depth)

Option Two: 3D Scanning

3D scanning is similar to photogrammetry, but uses more specialized hardware.This technology shares a lot of the pitfalls that photogrammetry does. Though it can be highly effective when it comes to accuracy, it provides a non-optimized model and texture set. This means the file size will be larger than necessary and will potentially require manual work to make it ready for use. Furthermore, it can be expensive to buy a good 3D scanner or have your object 3D scanned elsewhere.

Photo credit: Makerbot.com

Option Three: 3D Modeling Programs

In this option an artist starts with a blank digital space and creates the model from scratch. This can be a time consuming process and needs the skills of an experienced modeler to get right. However, the results can be visually accurate and fully optimized for purposes such as ours.

Our toolbox: Autodesk Maya and Mudbox, Algorithmic’s Substance Painter, and RealityCapture

Our Process

Step One: Taking Reference Photos and Measurements

The first step for each product is taking good reference photos. We were lucky enough to have Magnolia send us each product, which was a huge help during the entire process.

Just a few of the products sent to us by Magnolia.
Photo credit: Stephen Eastwood
Photo credit: Andrei Serghiuta

Step Two: Modeling

For the full modeling and painting process check out this link: https://youtu.be/t68hb1alb7g.
Reference Image compared to the created mesh.

Step Three: Painting the Textures

The high-density mesh will be used to generate smooth texture maps. Substance Painter will use these initial texture maps for the generation of various effects, like edge wear, scratches and rust.

Texturing in Substance Painter
Substance Painter can generate realistic imperfections
Some of the layers that make up the watering can texture
For an object like this watering can, a nicely polished surface would look out of place

Prepping Models for AR

Before the models can be used, they must be exported in a format used by your 3D engine of choice. In our case, we used the Collada (DAE) format as we were building a native Swift app using XCode, and the textures were exported separately as JPEGs.

Texture maps for the watering can from left to right: diffuse, roughness, occlusion, normal, metalness
3D model with contact shadow (left), 3D model without contact shadow (right)

Moving Forward

While we found a good workflow for creating high quality 3D models of products, the process is still time-consuming and manual. The challenge is going to be scaling this to our merchants. Our next step is to look at ways of adding a 3D artists to our Partner program, and to find affordable techniques that merchants can do themselves.


Shopify AR/VR

Experiments, thoughts, and lessons learned from Shopify's AR / VR team.

Thanks to Jon Wade.

Sam Cribbie

Written by

Shopify AR/VR

Experiments, thoughts, and lessons learned from Shopify's AR / VR team.