The Motion Waveform: A Proposal for a Useful Video Editing Tool

A feature common to most video and audio editing software is the ability to display audio waveforms. Waveforms are also the dominant user interface feature of the popular music sharing platform SoundCloud. Take a moment to listen to this track by Moderat and notice how certain changes in the music can be clearly anticipated by looking at the waveform. You know something is going to get louder at 0:35 and you know there’s going to be a transition at 2:53. If you look carefully, you’ll see rests at 3:27 and 4:23. The long fade out starting at 5:11 should be obvious.

If you are an experienced audio engineer or digital music composer its possible to identify interesting audio information purely by looking at the graphic representation of the waveform. The waveform visualization can reveal lots of useful information such as the attack, sustain, decay and release (also referred to as the ASDR envelope) as well as visualization of rhythms, silences, changes in dynamics, tempo. It’s possible to make a first pass through a lengthy recording in complete silence to select interesting segments using only the graphic information from the waveform if you know what you’re looking for and how it looks.

It would be great to create a similar level of abstraction for video without having to build our own specialized editing and annotation tools from scratch.

My proposal is to create an application (ideally a plugin for video editing software) which analyzes the movement in a selected video clip and translates this movement into a graphic representation over time — a motion waveform. To make it easy to integrate into existing video editing applications I would propose to map this movement analysis data to the audio spectrum and create a stereo .wav file. The left channel can be used for movement in the left half of the frame and the right channel for movement in the right half of the frame. The idea would be to analyze the video clips, render the data as an audio waveform and then place the resulting audio clips in sync on the editing timeline with the original video. If successfully executed this would allow the video editor the ability to see movement in the timeline at a glance and save time when selecting clips from raw material. Whether this will be possible will be dependent on making meaningful mappings between the movement and its graphic representation as a waveform. The ideal translation of the movement into audio would create clear graphic contrasts between stillness and movement, enable the identification of rhythmic movement, movement in a given direction, movement within the frame and movement of the entire frame, etc.

I believe a motion waveform analysis tool would be a useful supplement to the video editing process and I would be interested to collaborate with software developers to create it.

Update #1–20 January 2014:

Christian “Mio” Loclair has coded a simple proof of concept by building on some ofxCV code from Kyle McDonald. His code is available on GitHub and built using OpenFrameworks.

Christian was also an attendee at last November’s Motion Bank Lab in Frankfurt where we got to know one another. I’m very excited about this new development and looking forward to seeing how this progresses!

Update #2–19 February 2014:

I think it would be worthwhile to the ongoing conversation around the motion waveform to consider the value of the data which is available in a typical color correction tool called a vectorscope. In the following color correction tutorial for Adobe Premiere, we can see clearly how luminance information displayed in a vectorscope can give an abstract sense of the composition in the video frame. What is particularly intriguing is how this reduction can be used to create an abstraction of movement. Jump ahead to 7:07 where the author of the video tutorial demonstrates the visibility of motion by scrubbing through the timeline.

It would be great to find a way to translate the luminance information in the vectorscope to an audio waveform display which would retain these motion characteristics.