Introduction to Annotation as a Research Practice in Dance
Since 2010, the Motion Bank research project has been developing software and methods for the annotation of time-based media, primarily video, in the field of contemporary dance. This article introduces the specifics of Motion Bank’s annotation tool and some of the practical approaches I developed over the last three years working with it.
Origins and specifics of Motion Bank’s annotation tool
Motion Bank’s web systems for annotating time-based media have their origins in dance practice. The first version was created by the renowned The Forsythe Company for internal use and named “Piecemaker”. It was developed by the long-time company member David Kern to support the rehearsal process through collaborative documentation. This focus on the necessities of a dance company and artistic creation processes distinguishes it from other video annotation tools to this day. From the very beginning, it had to be open to processes whose outcome is uncertain.
Furthermore, by the time of its creation choreographer William Forsythe had developed his own movement vocabulary, mostly cutting ties with classical and neoclassical ballet and its more or less “universal” terminology. A strictly systematic approach as it is known from linguistic annotation tools like ELAN just wasn’t applicable and still isn’t for many practitioners from contemporary dance and qualitative dance research, where pre-given, universal categories do not really exist or are viewed suspiciously. Thus, a core principle of the tool has always been the possibility to annotate “freely”, descriptively. Last but not least, people do not want to fiddle with software that has a steep learning curve, keeping them from doing their actual work: dance, choreography, dramaturgy, teaching. Especially newer versions of the Piecemaker annotation tool thus aim to make the basic method of annotating videos as convenient and low-threshold as possible.
There is no software “product” by Motion Bank
However, the fact that the software development focuses on those specific needs doesn’t mean that similar things can’t be achieved with other tools and most of the approaches described below can be applied using other annotation software. It is also important to know that the Web Systems are not a (commercial) software product. Motion Bank is a research project with a small team and a core interest in methods, practices and theory around the documentation, analysis and transmission of dance and movement practices using digital media and technology. You can instead think of the Web Systems by Motion Bank as an offer to explore and research what annotation can add to your (artistic) research or teaching practice and develop your own methods, strategies, practices and theory. On the other hand, this also means that you should feel free to use any other annotation tool that might suit your or your institution’s needs. Yet, if you feel comfortable with it and you are working for an institution like a university, dance company, museum or an archive there are possibilities to adopt the software and to run it independently from Motion Bank’s infrastructure and development since the software itself is open source.
If you want to use the current Motion Bank Web Systems to explore annotation, just drop us a line: office@motionbank.org
Learning by doing
I first encountered the possibilities of using annotation when I started working with Motion Bank in early 2017. Back then Motion Bank was established at Hochschule Mainz University of Applied Sciences by Florian Jenett and Scott deLahunta, three years after the first phase with the Forsythe Company ended. The tool that was available at that time was a second, rewritten version of David Kern’s original “Piecemaker”, developed during Motion Bank’s first working phase (2010–2014) to support the documentation of choreographies by renowned artists and the creation of online publications about the documented works.
While Piecemaker 2 opened up beyond the particular use of the Forsythe Company, it was put together in a rather quick way, and it was a bit bumpy to use. However, I was able to get into it pretty easily, and even though it didn’t do so much more than linking text comments to specific moments in a video, it opened up an entirely new approach to studying dance for me. Being a dance and theatre scholar with a humanities background, it helped me to look at the actual movement and to differentiate its qualities in much more detail. Until today typical movement analysis is not my primary interest, but annotation was and is a perfect way for me to study dance closely.
I started giving Motion Bank workshops for people from the dance field quite soon, explaining to them how to use the software. A lot of exchange happened with educational programs, such as the Master’s program in dance education MACoDE at HfMDK Frankfurt and the dance departments of Codarts Rotterdam. In addition, I was introducing it to individuals who document the work and creation processes of choreographers. Recently, Scott deLahunta and I have given workshops for people from anthropology and ethnography at the British Museum tasked with documenting endangered immaterial knowledge, as well as the Tate London, where a team is collecting choreographic and performative works.
Piecemaker today: there is not one “how to”, and the software does not impose anything on you either
The Piecemaker software was rewritten again from 2017 on and also merged with Motion Bank’s web publication tool into what is now called Motion Bank Web Systems (a standalone, “offline” version of the annotation tool is going to be released soon). The current version of it is stable and easy to use, as well as built on up-to-date tech and data standards. Its principal function remains simple: you add text comments to specific moments in a video or other time-based media.
If you want to learn how to technically use the software, check our tutorials: tutorials.motionbank.org
As a result, in workshops and collaborations, our central purpose is increasingly achieved. We do not talk about technology and its challenges anymore, but about the methods and strategies of annotating dance or other bodily practices. Here the question of how to annotate comes up constantly, especially since the software is not suggesting any predefined application scheme. The answer to the question of “how to” is often the same: there is no one way, no right way, no singular method. It is learning by doing, developing your own method, depending on your project, objective and interest. However, through my own use and exposure to others using it in workshops and teaching situations, as well as being closely connected to the actual development of the software over time, I have come up with a few general approaches.
In the following section, I will share what I think might be helpful for a first orientation and as a loose guideline, especially if you haven’t used annotation before. The approaches in this article neither claim completeness, nor are they scientifically grounded. Additionally, they are not limited to dance, and I have tried to phrase them openly so they might be applied to other areas of bodily practice.
A Practical Introduction to Annotation — Starting points
In the following, I have divided the collection of approaches into five different sections. These can still be combined as required, as in the practice of annotation, they are often overlapping anyway:
1. Indexing and segmenting
2. Developing a research question, deciding for a central point of interest
3. Adding background information, feedback and (self-)reflections
4. Interview and “video recall”
5. Developing categories, terms or names and applying them
1. Indexing and segmenting
It can feel a bit awkward to write down what you can already see in a video, but you might want to consider that the information is locked in a container as long as it stays only within the video image. You can’t search for information or keywords the way you would in a digital text or on the web. If you look at the video player, you can see a visual representation of the start point and the endpoint, but you don’t have any indication for what’s in-between. Annotating the most significant events in the video can give you an overview. At the same time, you get to know the material better, especially in case you’re not familiar with it at all. It can also give you a visual overview of the structure of the content or the occurrence of events that seem notable to you. Later, even a rough segmentation makes it so much easier to find and come back to specific moments quickly.
Of course, you can continue to do this on a more and more granular level, describing “everything” to the smallest details, which can be a method of getting to know and to understand your material very well. Therefore, annotation can, above all, be an excellent training in close observation through repeated viewing. What is happening in dance and other bodily practices is not as obvious as it might first appear, and I think you’ll be surprised how quickly you go beyond simply describing what you first see in the video. This approach is also a lot about making things explicit by finding terms to name them and words to describe them. Suddenly, you’re in the heart of what annotating is about from my point of view. Furthermore, you might also recognise that you quickly go beyond naming and describing, but start differentiating qualities, interpreting and judging. It can be a good idea to remind yourself from time to time that even a simple segmentation is based on subjective perception and assessment anyway.
2. Developing a research question, deciding for a central point of interest
It does not matter if you start with a general segmentation of a media file or not, it can be beneficial to be clear in advance what your primary interest or research question is. Since dance and bodily movement can contain so much information, you can easily get lost in the details, especially if you’re getting more and more granular. The more you know what you are looking for, the more focused your annotation process will be. You can then index or segment the video regarding your interest or immediately look for essential moments, qualities and details.
What to focus on depends on your research subject, your artistic project and your interest. This means that whether you are coming from academic, artistic or didactic areas, you probably know what you are looking for. In general, it can be anything: quality of arm movements, feet relation, use of verbal language, discursive references, use of music, costumes, skill-related aspects, nonverbal communication between choreographer/instructor and dancers/movers, etc. If you’re working on a video collaboratively, it can be interesting to bring together different perspectives. You might combine a dance scholar’s viewpoint and an anthropologist’s, or split the work, so one person can focus on the use of language, the other on the use of music. In a movement analysis, different people can look at different qualities or body parts.
3. Adding background information, feedback and (self-)reflections
The strategies I described so far assume, more or less, that you deal with recordings and videos of something you haven’t been part of and that you encounter more as an outside observer. But of course, it may well be the case, that you annotate video recordings of yourself or a practice where you are the expert, e.g. a choreography you created, a dance class you developed or a technique you teach. Then you’re probably the ideal person to mark the critical moments in a recording and to use annotations to add background information that cannot be obtained from the video. That is where a crucial aspect of annotation can come into play, namely adding a discursive, narrative, linguistic level to the visual information level of the video image. In dance, for example, you can also refer to somatic methods, mental images and numerous other aspects that are simply not visible in a video. They may not be relevant for watching a dance performance (or a recording of it), but they are relevant when it comes to knowledge transmission and education.
In training, you can use annotation to give feedback to students based on a video. The combination of video image and a concise written commentary can be very instructive and help to focus on the key points. Annotation can also be a useful tool for demonstrating how things should be done correctly, again with the possibility to point at the key moments and actions. For students or people who would like to educate themselves and improve their skills, this is also a possibility for self-reflection. The analytical way of working associated with annotation and the change of media can also be a helpful distancing.
4. Interview and “video recall”
Let’s consider the contrasting situation. If you have unfamiliar what you are dealing with, it can be a good idea to simply ask those who are familiar with or have direct involvement in what is documented on the video (as long as this is possible). You might just chat with the experts a little bit, and this can already have an impact on your understanding of what you encounter. Especially if you are pursuing a scholarly project, it makes sense to get information in a more targeted and systematic way. A good possibility is to watch a video together with an expert or an involved person and to immediately record their comments as annotations, whether in the context of a formal interview or in a more casual exchange.
Especially in dance and bodily practices it is possible to document, using the technique of ‘video recall’, some of the “inner view” of the event that has been recorded, what it felt like, what the decision-making processes were, etc, at least from the perspective and memory of the persons involved. It is important to indicate in the annotations who is making the statements (whether the annotator or interviewee). You can also invite the interviewee to annotate the video themselves, but this sets up a different situation, especially if they have no previous experience with video annotation. Asking several people to annotate the same material can nevertheless be an interesting option to document and later represent different perspectives. If people use separate accounts, it will always remain clear who wrote what.
As you might have recognised, there are parallels showing up to video-based ethnographic methods, and it is a good idea to do more reading on these methodologies if you are not familiar with it.
5. Developing categories, terms or names and applying them
This approach can begin with developing a research question (Nr. 2) which might give rise to a collection of terms to start with in the annotation process and to help index your video (Nr. 1). You might also just begin by describing and naming what you can see in the video, which can produce a terminology to use for further annotation. These terms may be useful for other videos of the same choreography, artist, company or practice. This way, you are developing a “bottom-up” vocabulary from within the material instead of imposing one on it, so you get a vocabulary emerging from and adapting to the recorded event. A good start could be to look for terms that show up in your descriptive (longer text form) annotations. If you are an outside observer of what is happening in the recording, it may well be best practice to simply pay attention to the use of the language of the people involved. Most bodily practices come along with some terminology or a way of naming things, even if it is often idiosyncratic and fragmentary. You might want to pick up and make use of these terms, because they take into account the perspective and knowledge of those involved. This can be achieved also through a video recall approach (Nr. 4). On the other side, if you are directly involved with what the video documents, you might already know about the terminology and as an expert, make use of it for annotating the video (Nr. 3). However, if you are part of a team, company or institution, it can be helpful to discuss the vocabulary you are using related to the particular practice. This way, you can make sure you have the same or a similar understanding before you start using it for a systematic way of annotating videos.
Although I have been critical of imposing a predefined scheme, this does not mean that any recourse to a pre-established vocabulary imposes something on the material in a wrong way. The structure of choreography or the order of scenes in a piece can, for example, form a stable and somehow unquestionable temporal scheme that can be recorded through annotation and help a lot in terms of indexing. Adapting systems like Laban Movement Analysis (LMA) can also work or labelling different aesthetic orders that come into effect. Just think about how you deal with it if the assignment of terms or categories is not clear, which is quite often the case within the arts, especially in bodily practices. At least from a humanities perspective, it does not make sense to favour one over another term if the core content is persistently ambiguous. One way to deal with it could be a combination of formalised annotations with descriptive and explanatory ones. A benefit from the use of formalised annotations is undoubtedly the computational processing that it enables and that allows for better indexing, filtering and searching.
