Updates on glTF from Mozilla, Adobe, Facebook, Google & Microsoft at GDC 2019
Ever since glTF as a format was announced back in 2015, there has been a lot of development and support from companies and the community which has led to the growth of the format eventually leading up to its current version which is 2.0. In this period there has been a large rate of adoption from various industry leaders and companies in the immersive space who have built platforms, tools, exporters and engines which all support glTF in some form or the other.
At the Game Developers Conference 2019, the world’s largest professional game industry event which lasted from March 18th to the 22nd, the whole community came together to explore how the format has grown and how it has been integrated into everyday workflows of existing products in the industry. There were 5 talks given by active contributors to glTF from Mozilla, Adobe, Facebook, Google & Microsoft which vastly covered their usage of the format.
What you will find below is a short breakdown of these talks and their slides to give you a better understanding of how each one of them is contributing to glTF. Do note that these companies are contributing in larger ways to the format and what has been presented are the parts which are actively being used by various users around the world. As these talks aren’t out yet, the information below is based on my reading and understanding of the slides. Once out, any missing information will be updated along with the links to the various talks given. Let’s Dive In!
Mozilla Hubs and Spoke
Publishing Virtual Worlds with glTF
Traditionally, video conferencing has been limited to the bounds of a rectangular screen where the extent of collaboration goes as far as several people sharing their screen in real-time; endlessly asking if they can be heard.
Mozilla Hubs changes that and bring a new meaning to collaboration by giving you an immersive virtual space right on your browser through a link which can be opened on not just a desktop but even in Virtual Reality or in Magic Window mode. Once you join in, you can interact with others invited to the space and collaborate on a whole lot more than just shared screens.
Hubs does provide you with certain pre-built spaces for you to join in but if you would like to go create your own, you can use Spoke by Mozilla, a web based tool which lets you take your 3D scenes or use existing ones from Sketchfab and compose them by adding in more 3D objects or even tweaking certain properties such as background, colour, light etc. Once you’re done with your space, you can hit publish and use it for your social interactions across realities.
What Was Showcased
Mozilla Hubs
- All scenes and objects loaded on Hubs use glTF / GLB as a file format
- Upload of files such as Images, Videos, Audio and 3D models (GLB / glTF) with a way to make them stay permanently if signed in
- Addition of assets from across the web such as Youtube videos, 3D models from Sketchfab & Google Poly, GIFs, Images and even scenes created by other creators using Spoke.
- Custom avatars built using glTF can be used or even custom Avatars can be uploaded as glTF / GLB files.
- Live chat across desktop, mobile and HMD with screen share support of your window or webcam
- Video Live streaming support (Twitch, Crunchyroll, Youtube Live, Twitter, and more)
- A camera tool for taking selfies and screenshots in the virtual space and a pen tool for 3D spatial drawing and annotations
- Optimisation of materials of all models by switching to MobileStandardMaterial when on low quality mode (mobile and standalone).
Spoke By Mozilla
- Import and addition of models as a glTF /GLB file including URLs of Sketchfab or Google Poly models
- Functionality to add Media such as Images and Videos
- Addition of other essential 3D components such as Lighting, Skybox, Groups and Colliders.
What’s Coming Soon
- Custom Avatar Skins
- Resources for exporting avatars with the Blender glTF Exporter
- Hosted Spoke Editor
- Hubs Discord Integration
Work and Talk by Robert Long, Mozilla
Recording Of Presentation:
https://youtu.be/mCJoBGOCTwk?t=2241Slides From Presentation: https://www.khronos.org/assets/uploads/developers/library/2019-gdc/gllTF-Mozilla-Publishing-Virtual-Worlds-GDC-Mar19.pdf
Adobe Dimension and glTF
Giving graphic designers the power to design with 3D
Adobe Dimension is a 3D rendering and design software developed and published by Adobe for macOS and Windows. It started out as Project Felix but soon took shape into what it is today. Unlike other modelling tools such as SketchUp or Blender, models are not created in Dimension.
Dimension primarily deals with the staging of the 3D scene, lighting it up and finally exporting it out. All their updates were aligned around these three parts of the workflow.
What Was Showcased
Most of what Adobe has been doing as a part of the glTF Working Group has been to enable their Web Based Viewer which allows creators to see in exact replication, their scenes which are created on Adobe Dimension.
Mike Bond covers it in detail on a blog post that talks about how they built the publish to web feature but I’ll be covering the main points from the blog and the talk.
Lighting
- Dimension supports export of lights for a 1:1 replication of scene created using the KHR_lights_punctual extension
- Image based lighting is supported along with ability to export it from Dimension using the EXT_lights_image_based extension
Textures
They are currently optimising resource usage for the 3d scene by minimising the number of textures the GPU must load using the KHR_texture_transform extension
Materials and Transparency
Providing absorption, scattering and refraction properties to materials exported from Dimension by introducing the ADOBE_materials_thin_transparency extension to support more realistic properties of materials such as plastics and glass than just the default transparency alpha value provided by glTF.
Compression Using Draco
As scenes in Dimension can get quite heavy, Dimension converts and uses the KHR_draco_mesh_compression extension to export out the model which uses the Draco compression libraries in glTF format allowing for easy and faster transmission.
By using all these extensions, Dimension is able to export out scenes with a closer replication to what the creator intended it to be. Check out the following scenes created using Dimension on their Web Viewer.
What’s Coming Soon
The following updates are what Adobe plans to push to Dimension 2.2 which will be coming out in April
Substance Materials Support
Allegorithmic who always stood for making 3D creation accessible to artists of every stripe was recently acquired by Adobe and we can now start to see how this deal is starting to influence the other products within the Creative Suite.
- Dimension 2.2 will allow importing of materials exported from Allegorithmic Substance Designer, a tool used to create materials for use in 3D content, such as animations, special effects, and video games
- They will even support exporting of these materials with the glTF file.
Importing Of glTF / GLB files
As glTF starts to become a widely accepted transmission format across various tools and the creators who use them, Dimension will not just support exporting of models, but even allow them to be imported as a glTF / GLB file.
Future 2019 Releases
- Web Viewer will be moved out of beta
- Animation pass-through support
- More export/publish routes
- Compressed Texture Transmission Format (CTTF) texture support
- New extension(s) to support advanced material features such as Clearcoat, Sheen, Displacement, Transparency — thin film and volume, Anisotropy, etc.
Work and Talk by Mike Bond, Adobe
Recording Of Presentation:
https://youtu.be/mCJoBGOCTwk?t=1933Slides From Presentation: https://www.khronos.org/assets/uploads/developers/library/2019-gdc/glTF-and-Adobe-Dimension-GDC-Mar19.pdf
Facebook and glTF
An on-demand, optimising glTF backend
With the advent immersive technologies and easy access to content creation tools, posts on the newsfeed are starting to see a shift from to 2D to 3D based content. With the introduction of Facebook 3D Posts and Facebook 3D photos Facebook introduced a new way to engage users leading to greater conversions.
Facebook 3D Posts
Facebook 3D posts were introduced as a way for content creators to distribute rich, high-quality 3D content on the Facebook newsfeed. This benefits various companies especially in the entertainment and retail sectors as they already have access to a huge repository of 3D models of their products. Models can be uploaded in the GLB file format and should stay within the 3MB size limit.
Facebook 3D Photos
Not to be confused with Facebook 3D posts, Facebook 3D photos gives users a way to share their moments with an extra layer of depth added in. Giving 2D photos another dimension makes them way more interactive and has been made possible with the introduction of depth sensors in modern smartphones. Although one may think that 3D photos are simple images with depth information, Facebook has an interesting implementation by using 3D objects as GLBs which has been covered extensively by Yuri akella Artiukh in the blog below.
What Was Showcased
Texture compression service
When 3D posts were first introduced, uploading one would have Facebook break down the model along with it’s textures to it’s bare bones and perform certain client-specific customisations and optimisations to ensure that the model is compatible across the Web and other Native Platforms.
As this scaled up, and 3D photos were introduced, users caught on to this new immersive medium and started uploading large amounts of content onto the feed which resulted in millions of textures being uploaded and consumed. As this content would have to work on older phones, which had a requirement of compressed textures, Facebook built an in-house texture compression service and scaled it up to compress hundreds of textures per second.
Texture Caching
A step was introduced right before the creation of variants for the web and the native Android and iOS platforms, which reused the compressed texture which was created as a part of texture compression service.
Be The Backend For All glTF Products
Once this infrastructure was built and scaled up it was generalised and extended to cover several apps and products across the company which included Facebook 3D Posts, Facebook 3D photos and the models used in Oculus Home.
What’s Coming Soon
- Introduce “Level of Detail” as models are computationally expensive to generate on demand
- Streaming textures and geometries for the models uploaded
- Configurable asset specification to support assets which hold variations of different material/colour etc.
Talk by Pär Winzell and Susie Su, Facebook
Recording Of Presentation:
https://youtu.be/mCJoBGOCTwk?t=2635Slides From Presentation: https://www.khronos.org/assets/uploads/developers/library/2019-gdc/glTF-and-Facebook-GDC-Mar19.pdf
Google <model-viewer>
Easily display interactive 3D models on the web and in AR!
Adding 3D models to a website is not an easy task considering that it usually requires to have a good understanding of web based 3D technologies. What if adding a 3D model could be made as simple as adding a link with an <a> tag?
This is exactly what <model-viewer> does by easily allowing you to display interactive 3D models on the web and even in Augmented Reality. All you have to do is define a simple tag and give it the path to your model file.
<model-viewer src="path/to/model.gltf"></model-viewer>What’s Was Showcased
Model Formats
glTF and GLB are the two formats which work right out of the box with the <model-viewer>. Although it does let you link a USDZ model, you would see just an image of the model unless opened on an iOS device and placed in AR.
Lighting & Environment
<model-viewer> supports image based lighting where you can define a background image or an environment image and have that light up the model you are showcasing.
Animation
Objects with pre-built animations can be loaded into <model-viewer> and played. You even have the choice to specify which animation you would like to use if there are more than one built into the object.
Lazy Loading Of Models
To ensure that you have control over how you would like the user to view your model, <model-viewer> lets you have your own choice of UX flow when it comes to loading the models.
Augmented Reality
Models used on the <model-viewer> can be viewed on Magic Leap’s Helio browser, on iOS devices with the USDZ model and on devices which have the experimental WebXR Hit test flag turned on with the same glTF / GLB model used.
Fidelity Testing
When it comes to viewing 3D models, there are multiple renderers and maintaining consistency across them becomes a cumbersome task as each one have their own way of rendering the model. As <model-viewer> is another such renderer which renders the model, it becomes essential to ensure that the render stays closer to the source or how the creator of the model intended it to be.
Since one of the main goals of <model-viewer> is consistency, Google chose to focus on viewers that have significant levels of distribution and/or reputation as a reference and automate the testing and comparison process. The viewers used for reference are :
Once these tests are performed, they are compared and any discrepancies between the viewers can be addressed. All these tests can be viewed by heading over to the Results Viewer Page.
What’s Coming Soon
- Khronos WebGL Reference Viewer
- Solution for iOS Quick Look (As models don’t show up and Safari needs to support WebXR Hit test API)
- Interactive Progress Report Dashboard
- Standalone Fidelity Testing Effort
Work and Talk by Chris Joel, Google
Recording Of Presentation:
https://youtu.be/mCJoBGOCTwk?t=1563Slides From Presentation: https://www.khronos.org/assets/uploads/developers/library/2019-gdc/glTF-Google-Model-Viewer-Fidelity-Testing-GDC-Mar19.pdf
Microsoft and glTF
glTF update — GDC 2019
Microsoft as a part of the glTF Working group has shown strong support by integrating glTF as a file format across their Product Line right from their Office suite where 3D models can be integrated into Powerpoint, Word and Excel to creating custom models and exporting them from Paint 3D. They have even had several Open Source Contributions made for glTF across the web and native libraries.
What Was Showcased
3D Animations in Office
3D models can now be inserted across the main products under the Office Suite such as Word, Powerpoint and Excel. These models can either be imported as glTF / GLB models locally or from the Remix 3D store where models are created and uploaded from tools such as Paint 3D.
SharePoint Spaces
SharePoint Spaces takes the existing functionality of SharePoint and adds in a level of 3D and immersive technologies making it simpler to create and visualise mixed reality spaces using the intranet data you already have. Other than existing Web Parts, there are newer ones such as 360° video, 3D object and 3D data visualisation which all use glTF as the primary model format letting you create anything from a virtual training simulation to an immersive data visualisation.
3D Bing Ads
Microsoft recently worked on and tested out an ad campaign on Bing in collaboration with Samsung for the launch of their latest line of Galaxy Phones. Consumers who searched for the phone got an ad to see the phone in 3D (glTF) enhancing their experience and allowing them to see the phone in all its glory.
OneDrive Support For glTF
Along with the various kinds of file types which can be uploaded to OneDrive, glTF is one of the new formats which can not just be uploaded but even viewed right on OneDrive. This is made possible using the Babylon Viewer which has vast support and allows for glTF, GLB, OBJ or Babylon files to be easily viewed.
Microsoft 3D Viewer
3D Viewer lets you view 3D models and their animations in real-time. Model formats such as glTF and GLB are supported out of the box along with support for other formats such as FBX, STL, OBJ, PLY and 3MF. Besides being a viewer, Microsoft 3D viewer lets you play around with lighting controls, inspect model data and visualize different shading modes.
AltspaceVR SDK | Mixed Reality Extension SDK
The Mixed Reality Extension SDK enables developers to build 3D world extensions for AltspaceVR, a social platform for virtual reality where users can have conversations, attend live events, watch videos, play interactive games, and browse the Internet. You can add custom glTF / GLB objects to extend the existing Altspace VR worlds and build multi-user games and other dynamic experiences.
Babylon.JS
Microsofts very own 3D engine — Babylon.js can be used as a viewer for multiple 3D formats including glTF and GLB. One of the projects created with The Khronos group is a glTF Asset generator to validate multiple importers which exist and Babylon is one which provides a 100% coverage of all models. As a part of their Open Source Contribution, they have built exporters for 3DS Max and Maya allowing files to be first exported to their .babylon format and later to glTF. Files exported can be directly validated by users on their VS Code plugin allowing them to be debugged and viewed in real-time if any issues are present.
Work and Talk by Saurabh Bhatia, Microsoft
Recording Of Presentation:
https://youtu.be/mCJoBGOCTwk?t=1271Slides From Presentation: https://www.khronos.org/assets/uploads/developers/library/2019-gdc/glTF-and-Microsoft-GDC-Mar19.pdf
Looking Forward
As glTF continues to grow every day with huge support from the community, it will start to enable a whole new set of workflows which were previously not possible or even heard of across various sectors in the industry.
Evangelizing content creation comes to the fore as these immersive technologies evolve and new Ecosystems form. I currently lead product at Scapic, where it’s our mission is to simplify creation and distribution of immersive content for the next generation internet with our suite of easy to use tools.
Do let me know if I’ve missed out anything in the comments and feel free to reach out to me @akashkuttappa
