Sainsburys Tech — Argos 3D and AR on Web
So I don’t confuse anyone, Argos is part of the wider Sainsbury’s group, just thought I’d mention it!
Support for 3D and AR in devices have significantly increased, which provides many opportunities for retailers.
Native apps already have these features, Argos launched AR on iOS in early 2019, we decided to create an in-house solution for AR on the web. There is also a shift taking place of people moving towards web applications instead of just using native apps, particularly powered by Progressive Web Apps (an initiative from Google that we are also working on, more about that in the future). So we decided to investigate and take the challenge.
We did some research into the different technologies which are currently available, which led to us taking a split approach to delivery. At the same time, we do not want to affect the performance of users who cannot use or are not interested in exploring these new features.
We believe these new capabilities can offer a great new user experience, which is being backed by the stats/results further down in this article. They get access to 3D and AR features through the website without compromising the performance or installing native apps.
This has created three phases:
· Phase 1: AR for iOS12/Safari users and also improve the solution used by Apps to retrieve models. This first phase allows us to assess the number of users interacting with the new feature and the conversion rate when interacting.
· Phase 2: Add the 3D model viewer for desktop and all other devices
· Phase 3: Extend the AR models to the remaining devices
Phase 1 — AR in Safari/iOS 12+
The Safari implementation is simple. iOS devices provide Quick Look framework that supports AR/3D handling UI and UX, which means that only this HTML code is required:
<a rel=”ar” href=”model.usdz”>
<img src=”model-preview.jpg”> //Thumbnail with default AR badge
</a>
Notes:
rel=”ar” tells webkit that is AR content
href=”model.usdz” location of the usdz file
Pros: Very easy to use
Cons: No customisation, no option to modify the textures (you need to generate a new .usdz file per different texture) and it’s only available for iOS 12+
The Model Process
In order to provide this new assets to work with the code we need to be able to:
Identify and present the functionality only for Safari users.
Generate the models and make them available in our asset/media manager — creating the link between the product SKU and the new assets
Change our Product API to support this new functionality
USDZ models are archives containing all the data needed to render the 3D image; including the model and all associated texture files. At the time of building these files, the most popular method was to use the usdz_converter bundled with Xcode 10. This is a command line tool that takes in the model name, followed by the mappings of what textures apply to each object within the model.
We had a list of 200 models that we needed to convert to usdz models, totalling just under 300 products. With other file types that have the textures applied separately, the models can be used across multiple products and the colour textures swapped out. However, this cannot be done for usdz files and so for each original model we will need to generate several usdz files. It would have meant converting 200 models would be an extremely long and tedious process, instead we looked at how this could be automated.
By writing a Python script which integrated with Blender (a tool to edit 3D models), it would convert the original model to one compatible with the usdz_converter tool and also generate a list of the objects within the model. The script then iterates through the products associated with the model, downloading the textures and running the command to trigger the conversion. Allowing us to go from a folder of 200 models to 300+ udsz files in just a few minutes, ready to be uploaded to our content delivery provider.
Once this has completed, we then ingest the data, in the same way we do for other types of media. The main change being, we separate it from other assets, working closely with Amplience (our provider — hosting of the assets). This was due to the difference in object structure and the fact some clients may only want standard media types.
The workflow is as follows:
· Convert the collada file to a filetype compatible with the usdz_converter tool
· Generate a list of the objects within this model
· Look up which products this model applies to
· Download the associated texture files with for the products
· Write the command to build the usdz file
Thanks to the team for putting it all together:
Richard Lindsay, Jon Kepa, Buwa Okome, Robin McSkelly
References:
https://webkit.org/blog/8421/viewing-augmented-reality-assets-in-safari-for-ios/
https://developer.apple.com/videos/play/wwdc2018/603/
Following posts will look at phase 2 & 3. We will describe in more detail: 3D approach, file formats, Three.js, Model-Viewer, Bundlesize…
Examples:
Try it yourself:
