Annotating music in IIIF

The original Variations Audio Timeliner was initially developed in 2002 by the Indiana University Digital Library Program. It is a desktop application for Windows and macOS that students and teachers could download and use to create visual representations of the structure of music.

Since then, after many versions and iterations, the Indiana Timeliner made it’s way onto the web as an open source, rich browser-based application. It draws from the original design and usability while also embracing the open standards set forth by the IIIF community. The Timeliner project creates both visually interesting representations of music and an output that is interoperable with other tools in the IIIF ecosystem.

What can it do?

The building block for creating a representation of the structure of music in the Timeliner is a bubble. Each bubble starts at a point in time, and ends at another point of time in the future. Bubbles are, in this interface, continuous and as a result you cannot have gaps between bubbles. Bubbles can have bubbles inside of them, which itself can have bubbles nested inside. This bubble structure creates a continuous hierarchy.

Creating bubbles

There is no concept of creating a new bubble in Timeliner, instead you split existing bubbles. Any bubble can be split multiple times at different points. Those bubbles can then be grouped together, creating more bubbles. This inside-out format works well for musical notation, where you can begin by identifying variations in the piece and then take a step back to find the groupings.

When you first start a new project you will always be presented with one bubble, from the start of the musical piece to the end.

Along the bottom of the bubble viewer is a variety of controls. These controls, combined with the current time to the left, allow you to split, group and remove bubbles and add non-bubble markers, which denote a point in time opposed to a range when adding a bubble.

To split the bubble simply chose on the timeline where you want to split and either hit the add new marker “+” button, or double click.

With this first interaction alone, you can start to break down your bubbles into smaller and smaller units. Control or shift clicking lets you select multiple bubbles, which you can group together, shown as another larger bubble.When you start to group these units into larger concepts, with colours there to help distinguish each level you can really start to see how the bubble interface helps to visualise the music.

With these building blocks, you can start to create complex representations of musical structure and associate metadata, labels and meaning to each section.

What we’re doing here is creating a hierarchical structure, shown best when viewing the annotations (more on that soon) attached to each of the bubbles under the main viewer.

You can see how we can sit back and watch the information about the piece of music flow past, making the Timeliner as much about creating and notating musical structures, as learning about the music itself.

Editing bubbles

When you need fine grain control and want to get the timings perfect, you will want to zoom in to refine your bubbles and get them exactly synced up with the music. The 3 buttons to the right of the audio controls control the zoom, allowing you to zoom in, zoom out and reset the zoom back to normal.

Each bubble is more than just a span or time and a colour. You can add a lot of descriptive information to go along with your selected time segment. In the Timeliner these are annotations.

These annotations are where you can delve into the finest details of the piece of music.

Each annotation has a label that will appear on the bubble itself, and a longer description which appears in the second half of the user interface. They are colour matched with the bubbles so you can easily spot them in your timeline.

Hitting the pencil icon on any of the annotations will bring up the editor for them, where you can create or edit each annotation, including changing the colour of the associated bubble, and manually enter the exact time. Especially useful if the timestamp you want to use is known.

Markers

Sometimes you may want to highlight an event, or a single point in time and not a range. Markers fill this need, and can be dropped anywhere on the timeline. They’ll appear under your annotations, and become part of the bubble, so you’ll see it for the duration of the bubble. You can click on the arrow on the marker to jump straight back to that point.

When you start a project you can either load in an existing project or provide a link to an audio file, such as an MP3. If you start with an audio file, you’ll be able to manually add a label and description for your entire timeline, found on the right hand side.

This, along with all of your bubbles will be available to anyone who you share your project with.

The user experience

In addition to being a tool for recording information about a piece of music, the Timeliner is made to be expressive, both through the information inside of the annotations and also the colours used.

You can use colour to denote extra information for bubbles, an extra semantic connection for bubbles on the page.

This also serves a dual purpose as this experience should be able to be viewed by all. The Timeliner has complete control over the colours used to build up a timeline, but also allows people who come to view the timeline the option to override this choice.

The purpose of the bubble UI is to be able to differentiate the regions covered by the bubbles. However, not all users perceive colour in the same way. Colour blindness affects almost one in twelve males and one in a hundred females, additionally almost half of recorded types of visual impairment are contrast sensitivity. It’s imperative that users experiencing or learning using Timeliner have the choice to change these colours for their needs. The Deuteranopia, High contrast, Grayscale colour schemes are available to those who need it, keeping the Timeliner as accessible as possible.

Wrapping up

Once a project is finished, you can download it as a file. This small file will contain all of the annotations, markers and descriptions you’ve written, along with the link to the audio file. You can share it with others or come back to edit it another day. The format is IIIF, so it will be possible to import into viewers that support audio manifests. The Avalon Media System, used by Indiana University, has also integrated the bubble editor, further improving the editing and saving process.

The development of the Timeliner project was a collaborative open source project between Digirati and Indiana University. Be sure to check out the project on GitHub and also the demo featured in the images throughout this article.

If you would like to get involved, reach out on GitHub or contact us directly.