This week we unveil our first prototype from the R&D team at Jam3. We will be working in rapid sprints to produce new prototypes, libraries, tools, and experiments every few weeks. The first week is research, the second is prototype development, and the third is writing blog posts and other tasks. These prototypes will lay the groundwork for larger client projects and help to explore new avenues of development.
For our first R&D experiment, we built MTCHMV (pronounced Match Move). To see it, check out the link below in desktop Chrome or FireFox.
MTCHMV explores how After Effects 3D camera tracking can be exported and used on the web. The findings from this prototype will open new ways of manipulating and interacting with video content in real-time.
After Effects is used to analyze a video segment and determine camera movements in 3D space. The camera parameters and its keyframes are exported to JSON, which allows us to composite real-time and interactive elements on top of the video using WebGL and ThreeJS. The 3D content reacts to the user’s interactions, elements in the video, and the audio track.
Our MTCHMV prototype demonstrates four different applications of this technique. Use the arrows in the demo to cycle through effects.
The first example shows some abstract, colourful 3D shapes bouncing in real-time to the music with WebAudio. The shapes are rendered with a transparent WebGL canvas, and composited on top of the video. A trail of particles stream out from the back of the biker, and everything is tracked to the movement of the video. For the bike trail, we keyframed a solid 3D layer in After Effects that roughly matches the position of the biker in 3D world space.
Physics and Ground Shadows
The next example shows how we can match move the ground and walls in our video to integrate real-time physics for the 3D composited elements. For the prototype, we used flat planes to define the areas the balls should bounce (the road and building walls). A more complex project might build 3D models to match the topology of the ground and other colliders, such as the lamp posts. This would help collisions, depth and shadows behave more realistically. Using the biker position from the last example, we were also able to repel the balls as the biker moves near them.
This also demonstrates real-time shadow casting, which helps integrate the 3D balls into the video footage. For this, a transparent ThreeJS plane has a custom shader which only renders the result of the shadow casting pass.
Real-Time 3D Text
This example shows real-time 3D text being composited on top of the video footage. The text is anchored to artist-defined positions in 3D space, and animated in with After Effects keyframes. While the content is playing, you can change the text fields to update the text.
To achieve this, we are using Jam3/three-bmfont-text to render scalable text with signed distance fields.
This could be used, for example, to connect with Facebook for user-customized text, or with a weather API for real-time updates.
Painting in 3D
The last demo allows the user to paint colourful 3D strokes as the camera moves through the scene. The brush strokes react to the music and cycle through colour palettes as the video loops.
To render the strokes, screen-space projected quad strips are used with a custom shader for line dashes and textured edges. See this blog post for more details:
To build this project, we used ae-to-json and its CLI. This is an in-development project that tries to export generic JSON data from the currently open After Effects session. For details, see here:
The JSON is optimized and stripped down to only the data we need (keyframes, solids, and camera parameters) and this is fed into our ThreeJS compositor. For convenience, we are using a module to handle the interpolation of the After Effects timeline data:
During development, we ran into some issues. The first issue was matching the ThreeJS camera movement to the After Effects compositions. Our early tests rendered the camera paths, but with some occasional glitches due to Euler angles “wrapping around” in degrees. To resolve this, we converted all orientation keyframes to quaternions and used spherical interpolation for the camera rotation. Our final result still has some slight jitter, where ThreeJS does not match each After Effects frame exactly, so this is one area which requires more work.
We found that not all video footage would track automatically in After Effects. For a larger production using this approach, it would be worth collaborating with a professional match move artist. Further — for the best quality track it’s important to lock everything to the same frame rate. Your source video, 3D Camera Track filter, web-encoded video, and the frequency at which you update ThreeJS camera position and orientation should all be locked to the same rate (in our case 23.976 FPS).
Another issue is with depth testing. If we render a 3D element ahead of the biker, it would not get depth tested correctly, and the immersion in 3D would be lost. This is visible when objects appear atop the lamp post on the left, or when the biker moves through the 3D balls. A match move artist might be able to resolve this, by rotoscoping or modeling the biker, and we could render a transparent mesh with depth testing enabled. The problem is shown in the image below.
This prototype was very quick — built in a week by two developers. There is a lot more work that could have gone into the design and overall experience, as well as more interesting ways of compositing elements.
Something that did not make it into the final demo is matching the lighting with a composited mesh. This would be great to explore in a future project, perhaps even using spherical photos shot on-location to match the lighting exactly as it appears in the video. See below for a rough test using physically based materials on a ThreeJS torus knot:
Another feature we hope to explore is 3D depth buffer elements. For example, rotoscoping or modeling the biker. These meshes — invisible to the user — would render to the z-buffer so that 3D elements cull correctly as they pass behind real elements in the video scene.
Lastly, there is more work that could go into the interpolation and camera movement. Our 3D data doesn’t yet match the After Effects data exactly, and we did not attempt to handle easings equations in the keyframe data.
Exporting the camera data from After Effects paves the way for a lot of interesting new use cases on the web. For example, an Olympics website which allows the user to play 3D tennis composited over a real video of a court, or allowing the user to discover athlete data with real-time hot spots tracked to the video content. Or drone footage of a city that adds real-time points of interest, such as current events and live music. The great news is that the opportunity for overlaying dynamic interactions in conjunction with camera tracking has become a viable creative output.