glTF Debugging in Visual Studio Code

What happens when you combine glTF with an extensible code editor like Visual Studio Code?

The glTF Tools extension of course! Shout-out to Ed Mackey for creating the extension originally. This tool does a great job with validating glTF assets via the glTF-Validator, previewing assets in multiple WebGL rendering engines, inspecting data from referenced URIs, and more!

Recently, I started implementing some improvements to this extension as part of a Hackathon event at work. I wanted to add some features that support visual debugging of glTF issues. Whether I am answering questions on GitHub or debugging glTF assets from an exporter, I often want to be able to see certain aspects of the glTF rendered in the preview. To this end, I added two main features to the extension. First, I added a way to visually inspect mesh primitive data. Second, I integrated the shiny new Babylon.js inspector. In this article, I walk through my experience working on this project.

Inspect Mesh Primitive Data

One feature that I have been wanting when debugging glTF assets is the ability to select a vertex or a triangle and then see it in the 3D model preview. Seeing the data as raw numbers is often much more difficult compared to the visual representation of the data.

The first thing that I need is a way to select a vertex or a triangle. The glTF Tools extension supports inspecting accessors, images, and shaders which all references raw data from an external URI or a Base64 encoded string. This opens a text view of the data like this.

While this shows a vertex via its position, it does not allow for selecting a vertex. The indices accessor view has a similar issue for selecting a triangle.

I need a new view for mesh primitive data to enable item selection. First, I try using pure HTML to construct a view for this, but it has performance issues. I can implement a virtualized view of some kind, but after some more experiments, I find that VS Code’s tree view displayed in a view contribution is a better fit. Here is what the new view looks like.

This solves the performance issues and allows selecting items including selecting multiple items. Hooray! 🎉

Now comes the interesting part. There are a few more items to tackle before I can visually show something.

First, the when context value needs to stay true when switching between the glTF text editor and preview window. Without this, switching to the preview window causes the mesh primitive tree view to hide. This is not straightforward due to how the VS Code API works. I end up creating a GltfWindow class and corresponding GltfPreview.onDidChangeActivePanel to handle the logic. Getting multiple glTF text editor and preview windows with the correct context value is rather tricky with the existing API. Ideally, I think this kind of logic should be handled by VS Code directly.

Second, I need to communicate to the preview window that the user has selected items in the mesh primitive view. At first glance, window.postMessage seems like a solution to communicate across the iframe, but it also seems cumbersome. Fortunately, newer versions of VS Code introduce the Webview API that supports message passing directly. This means I have to do a bit of plumbing to convert the existing preview to a webview before I can pass messages, but it’ll be worth it.

Finally, I want to render widgets indicating the position of the vertex and its tangent space if available. I also want to render a triangle when selecting a triangle in the mesh primitive view. I can implement this quickly with Babylon.js using sphere meshes for the positions, axes viewers for the tangent space widgets, and lines meshes for the triangles.

Woohoo! I can now visually inspect mesh primitive data. 😅

Babylon.js Inspector Integration

I also want to integrate the Babylon.js inspector into the extension. The inspector is a valuable addition as it has plenty of useful features and is great for debugging issues. Combining the Babylon.js inspector along with the glTF VS Code text editor can also provide interesting integration features. For example, picking a mesh now instantly select the part of the glTF in the text editor that corresponds to it.

Awesome, right? 😎

While David Catuhe does most of the hard work for the inspector itself (e.g. integrating React into Babylon.js), integrating the inspector to the glTF Tools extension surfaces its own issues.

Issue 1

The first issue stems from the inspector being hosted in VS Code instead of a web browser. For example, the inspector includes a couple of buttons. The left button undocks the window and the right button closes the inspector. Both of these make little sense in this context. Undocking a window is not supported in a node.js environment. I want a toolbar button in the VS Code window to show and hide the inspector.

Inspector has these buttons by default
glTF Preview with toggle button to show/hide the inspector

Both of these buttons are now excluded when hosted in VS Code thanks to some options passed in when showing the inspector. Though this particular issue is easy to fix, there are a few more of these types of issues that still need to be addressed. For example, in the tools tab, these features need to output files.

How should the code save files when hosted in VS Code?

I’m not sure how to handle these yet, but we will think of something.

Issue 2

The second issue is an odd layout weirdness. For some reason, the webview inside VS Code causes the HTML auto width calculation to go bonkers. 😕

After finally narrowing it down, I file an issue on VS Code. It turns out this is likely a bug in Chromium since the insiders build of VS Code with a new version of Chromium no longer exhibits the bad behavior. This will be resolved once the new version of Chromium makes it to a stable build of VS Code.

Conclusion

It has been a fun experience implementing these improvements to VS Code. I hope people will find the new features helpful when working with glTF.

This is also just the beginning.

There are a few issues to iron out. One example I mentioned earlier is that the inspector needs to handle certain browser features that are not available in a node environment. Another example is that the vertex and triangle widgets don’t work with skinning and morph targets.

There are new features we can add. For example, the preview window can visually show certain validation errors, as suggested by Ed. I want to be able to visually see widgets for the tangent space of individual pixels as opposed to just vertices.

The possibilities are endless!

Gary Hsu — Babylon.js Team