Ethics and augmented reality: How much is too much when editing 3D models for journalism?

Download American Food and What They Carried, available on on Actual Reality for Android.

We were in the thick of post-production for the first episode of our AR series American Food, when I saw this screenshot pop up on Slack:

McClatchy New Ventures Lab 3D artist David Martinez was showing the team his work in progress. He was patching holes in the 3D dishes that would be the centerpiece of our story about Sacramento’s Little Saigon, the first stop on our tour of cultural food enclaves around the United States.

We had all been discussing how, in this new medium, we could maintain the ethical standards that McClatchy journalists have upheld for more than 160 years. In particular, we were asking ourselves: How much editing is too much?

One challenge of experimenting in 3D — especially with the affordable, quick-turn production methods we are testing — is that the tech has shortcomings.

In this case, Eric Howard and Stan Okumura, the producers of American Food, had returned from the field after capturing an array of Vietnamese dishes using a process called photogrammetry. Photogrammetry involves taking many photos from several angles and using special software to combine them into a photo-realistic, 3D model.

In post-production, we found that the complex light information from shiny, white and transparent surfaces confuses the photogrammetry software (in this case, we’re using Reality Capture), leaving holes in our 3D images.

A screenshot from the Reality Capture software shows photos of the dish (left) and the 3D object created from the photos (right). You can see the holes in the dish that need to be repaired.

(There are ways to reduce the shine in the field—which we’ll discuss in another post—but they aren’t always practical or possible.)

So, with plates and bowls filled with holes, we were faced with a decision: We can either leave the rough spots or make edits to ensure that the surfaces are as faithful to the original as possible.

For guidance, we looked at industry precedent for visual media, in particular these two rules:

• From the NPPA: “Editing should maintain the integrity of the photographic images’ content and context. Do not manipulate images or add or alter sound in any way that can mislead viewers or misrepresent subjects.”
• And from NPR’s visual ethics code: “When reporting on news events, the photographs we take and use depict them truthfully, honestly and without bias. They are only enhanced for technical clarity — to correct color or improve contrast, for example. We are careful in how we crop them to ensure that the scene is in proper context.”

For 2D photos, this means limiting edits to modifications like adjusting exposure in the darkroom or color correction in Photoshop. Editors shouldn’t take liberties to add Instagram-like filters that might suggest the photo was taken at a different time of day, or go even further by removing or adding elements in a scene.

With this spirit in mind, we came up with this policy for editing 3D images:

• Captured objects may be edited for technical and visual clarity if the edits are as faithfully representative as possible.

So what did that mean for David? How did he put this into practice?

Since photogrammetry requires lots of photos, he had a ton of visual information to reference as he edited the chewed-up dishes, like this one of Vietnamese soup, for visual clarity. He looked closely at the photographs as he used “digital clay” in Zbrush 3D modeling software to fill in the missing parts of the shape.

This screenshot shows the “digital clay” (in green) David added to the bowl in Zbrush.

Then, to get the color and texture right, he used a combination of the clone stamp tool and hard and soft brushes in Photoshop to copy the surface.

With those two techniques working together (within a limited time frame), the final image is as faithfully representative as possible:

While the patchwork is noticeable, this was an early iteration of our photogrammetric models. You can see other, more polished models on our Sketchfab channel where the evidence of editing is harder to see. Or download the entire American Food series.

In part because viewers may not be able to tell that models were edited, we decided that it’s not enough to hold ourselves to these standards: We want to be transparent and let our audience know what went into the visual experience before them. In our app, we run an editor’s note explaining the edits and defining this type of object as a “photo-realistic model.”

The glass bowl

Another test of our ethics code came up when we wanted to feature a model of the dessert dish bouza in the episode taking viewers to Little Arabia in Anaheim, Calif.

In this case, the stretchy, Middle Eastern “ice cream” was served in a glass bowl, and the model of the dish had more than a few small holes:

A screenshot from the Reality Capture software shows photos of the dish (left) and the 3D object created from the photos (right). You can see the dents in the dish that need to be repaired.

David had his work cut out for him. To fix this, he would have to build the bowl from scratch. To make the bowl, David produced a realistic re-creation, referencing the original scan. He had information on the circumference, for example, and photographs of the surface’s texture.

Then, he put the photogrammetric model of the bouza inside the re-creation of the glass bowl.

Although he easily could have spent months making this look even more realistic, we were also testing how much we could do in a short production timeline.

While getting as close to the original object as possible was still the goal, we think this goes beyond simply editing for technical and visual clarity. We also know it’s important to give ourselves limits to how and when to use a realistic re-creation, so we came up with these guidelines:

When a photo-real, 3D object cannot be reconstructed, we may use an artist’s re-creation in its place. We have done this only if:

• The object is integral to the story,
• Using a re-creation is the only option for representing the object in the story and
• The re-creation is as faithful to the original object as possible.

Any time we’ve used a highly realistic artist’s recreation to take the place of an object, it is noted at the beginning of the story you’re viewing.

We know as technology improves and we get better at applying these new production techniques to nonfiction storytelling, these guidelines will evolve.

For example, at the same time we were finishing the Little Arabia episode, David was experimenting with ways to light the glass texture to get a more realistic look (including one technique that uses a 360 image to recreate the lighting conditions where the object was found).

Here’s an idea of what the lit bowl might look like, using the lighting features available in Sketchfab:

This raises a new question: If you’re able to recreate the lighting conditions from the environment where the object was found, shouldn’t that be the standard? And if we can’t meet that standard, should we let our audience know?

We will continue to have these conversations — in the field, on Slack, sitting together in front of Zbrush and Reality Capture, and here on Medium. And we will be transparent with our audience about what they see and how it was produced.

How are you approaching 3D production and ethics in your newsroom? We’d love to hear from you in the comments below, or email us at newventureslab@mcclatchy.com