Dance, Cells and Grids—The Story of MoSys
How our choreographic publishing system evolved from a simple site builder to a multimodal annotation tool
TL;DR — MoSys is the publication element of the current Motion Bank System (2018 – now) but also a multimodal annotation tool in its own regard. This article traces MoSys’ story, current state, and future.
When we talk about our research at Motion Bank we often phrase it as “working on methods and tools for the documentation, enhancement and publication of knowledge inherent in contemporary dance”. Here “tools” refers to two building blocks of our work which we just recently (in early 2018) integrated into one/the “Motion Bank System”, and “methods” refers to the way we (and others) think about and use these. One part of the new joint system is the well known Piecemaker tool for annotation of time-based media. The other is what we call MoSys and in this article, we’d like to draw a clearer picture of it.
The Origins
MoSys was originally started as the backbone of our “Online Score” web publication format. In dance, a “score” is commonly understood to be a document that provides instructions for action and can be translated into a performance. An “Online Score,” on the other hand, got established as a label for web documents that brings together various formats and can serve various purposes … as you will learn below.
In 2012, slowly finishing our “phase one” (2010–2014) we realised, looking at our results from the work with Deborah Hay, Jonathan Burrows & Matteo Fargion, Bebe Miller, and Thomas Hauert, that we would need a tool to not only publish these diverse results but also enable experimentation with and discussion about the content of these “web pages”. Content here refers to both the materials, that is text, image, video, animated and interactive elements, and also the perspectives formulated with them.
The “Online Score” format, a way of communicating and enabling the reading of dance knowledge on the web, originates from the “Synchronous Objects” project (The Forsythe Company with Advanced Computing Center for the Arts & Design at The Ohio State University, 2009) which is considered to be the pilot project to Motion Bank. In Synchronous Objects, following questions like “What else might dance look like?” (William Forsythe) the team at ACCAD OSU (Maria Palazzi, Norah Zuniga Shaw, Matthew Lewis) created a web format that would bring various results from the experimentation, visualisation and documentation around Forsythe’s piece One Flat Thing, reproduced into a form of interconnected tableaus that each allow for deeper exploration.
One project goal for the first phase of Motion Bank was to create more Online Scores with content from other choreographies and hence, extend the spectrum started in Synchronous Objects. Although we were clearly setting out to continue that heritage, we eventually arrived at creating MoSys from another angle.
The first results to be published originated from the work with Deborah Hay on her solo “No Time To Fly” between 2010 and 2012. Initially, in 2013, we were working on a classic page-based website to publish these. Prototyping it we ran into a very interesting problem. The highly interconnected information around the many recordings of the solo made it difficult to come up with a reasonable way for uninformed users to navigate that information space.
There were the aspects of the recordings and how to visualise them, the piece and Deborah Hay’s overall pool of work and choreographic practice. The recordings to be published consisted of 21 solos performed by three different dancers and recorded with multiple camera angles and sound. In addition there was more data: 3D traces and annotations for each of the recordings. The information around the piece consisted of its history, structure, and additional documents like the underlying score booklet. Also, we expected people to arrive at the site with different perspectives. Some might have a relation to dance others might arrive with an interest in data or its visualisation. After multiple iterations, we arrived at the idea of letting the material itself become the means to navigate all these aspects. Everybody seemed fine to play a video online, so why not structure the other information around that?
With time-based annotations (done in Piecemaker) at hand which link to most of the other aforementioned aspects, we were able to put this connection at the heart of the navigation mechanism. Choosing and playing videos would change other content of the page to display the corresponding bits of information or adapt form and visualisations. We instantly liked the outcome as it felt very intuitive to navigate content like this, and it also emphasised the depth of information embedded in something as “simple” as a video of a performance. Clicking annotations or changing interactive visualisations would make the videos and other content catch up and hence connected the many dimensions (time, space, data) in a natural way. This is why we originally referred to MoSys as a choreographic publishing system. The choreography became the structuring element. The possibility to define and use interaction between page elements transformed compiling these pages into a task of organising and structuring … which can be a way of working choreographically.
Another early thought guided our design decisions for MoSys. Addressing two very different audiences we were worried that emphasising either one perspective, “documentation” or “data”, might drive the other away. Putting the performances at front might make “data people” feel lost looking for visualisations and how to access and work with the captured information. Putting that quantitative and also technical perspective at front, on the other hand, would not invite a dance-oriented audience to explore the results we wanted to give access to. Looking at this problem we came up with the picture of an exhibition space with two entrances: one that would allow visitors to start from the performance and be guided to other forms of looking at and thinking about what they’ve seen. Another entrance on the opposite side would start from the data. From there it guides the audience to arrive at the performances making the underlying artistic work visible to them. This thinking led to the later layout of the MoSys pages (we call them grids) as a horizontally scrolling arrangement of elements, giving the impression of following along with the walls of an exhibition space.
Versions: v1, v2, v3
MoSys, similar to Piecemaker, has gone through three main iterations. An early version (v1, 2012) was developed by the Frankfurt score team, a first release was crafted at Meso and later continued by Motion Bank (v2, 2013–2017) and a third version (v3) was created by the current team at Hochschule Mainz. This last version is now part of the public Motion Bank system and under constant development since early 2018. It is a full re-write as we decided to streamline and synchronise our technology stack. Since version 2, the system has been open source and based on a decoupled website paradigm meaning that the back end (database, application) and front end (public site, editor) are connected through an application interface (API) delivering only the data, but not the actual views.
Core concepts
The focus and space of this article do not provide enough room for a full-length documentation which we offer on our tutorials site already. The following description of how MoSys works is to introduce the terms and core functionalities so that you can understand what you are reading and seeing in this article.
There are two “content types” that make up a MoSys publication: cells and grids. The cell is a lightweight container or wrapper for all kinds of page contents like text, titles, video, images, animated or interactive content, etc. The idea is that diverse data sources can be attached to a MoSys installation and these provide the actual contents. The Piecemaker instance running side-by-side in our system is already set up to do this and it is easy to pick a video or annotations from there to use as cells. Video hosting platforms like YouTube or Vimeo can be used to provide additional video sources. Specific installations like the one at the Pina Bausch Archive can also make use of their private data collection. This openness for almost any kind of source allows the possibility for your own local data sets to be connected and grow into a rich network of information. It places local knowledge back into the broad fields of knowledge (and anything else) available on the internet.
A cell can be placed into a grid which is a horizontally expanding “page” with a flexible row and column setting. Within a grid, the cells can be placed, moved around, and expanded to consume more than one grid field. This forms a very flexible layout system.
Another aspect of the horizontal grid layout is that by navigating through it with a sliding (browser) window one wanders through the contents as if reading a book or strolling through a gallery. There is a beginning and an end and a story that builds up. The cropped (browser) view allows one to focus on small knowledge fragments, sometimes inviting the user to spend more time at a single “location” and explore. The horizontal scrolling has been critiqued quite often, but we still opted for it because it felt more natural to explore unknown terrain along this axis following the picture of turning your head sideways to look around.
Underneath the grids lives a messaging service that allows cells to communicate with each other. Similar to performers on stage they can give or respond to cues or adapt to other information that is being passed around, like the current time of a video that is playing or information from an annotation like who the performers are or what scene is visible. This layer allows for both simple interactions and the creation of quite complex constellations.
Being based on web technology it is easy to also use many other formats as cell contents. One could add a map or 3D viewer or even make their own cell content using <iframe>s. We have done that for some of our interactive visualisation cells or even to allow things like commenting and feedback inside a grid. Title cells can act as hyperlinks between the grids and allow for navigation and information organisation. Often we create an opening grid with introduction texts and links to other (sub-) grids with more extended views. A classic tree-like website structure.
With the most recent version from mid-2020, we added a way to collaboratively work on and share grids. This was long-planned for and then finished and published as we opened our systems for educational and professional use as a reaction to the COVID-pandemic. Not only can you now co-edit grids it also became possible to export a set of grids as a self-contained website package that can easily be hosted on almost any server.
On an even more technical side, the cells are spatial annotations (position and also size) onto a grid. They follow the W3C Web Annotation Data Model standard (as do our Piecemaker annotations) and, being linked data, are easy to export and work with in other archival contexts.
Using and exploring MoSys
Working on our initial Online Scores, as described above, we soon discussed what MoSys would also allow us to do. One thing that came up early was realising that we started to use it to collect contents as a starting point to think about them, often without really knowing why exactly they had to sit next to each other on a grid. MoSys became a sort of moodboard for conceptional bits and pieces and once further refined these grids often were used to share and discuss these ideas internally. This also turned out to be a good way of discussing perspectives with the artists involved to get feedback.
With our original Online Score, we were exploring different forms of putting things together. Some grids came out being structured in a very light and open way, others ended up being very dense with cells covering almost all of the grid surface.
One interesting use case for MoSys traces back to our first public presentation of Motion Bank results at Tanzkongress Düsseldorf in 2013. Back then, we set up an interactive installation in the central meeting space consisting of two iMacs running the very first version of MoSys. On view were interview snippets, recordings, visualisations, and text fragments from the work with Deborah Hay. Visitors were able to start anywhere in the layout of materials, and the grid was set up to then navigate and play automatically through all kinds of fragments forming a storyline. This allowed for both an explorative approach to the content but also to join a sort of playlist and simply follow along and observe. In a way this opened our perspective up for not only looking at MoSys as a publishing tool where editors would prepare content and users would consume it but also as a knowledge terminal that would enable browsing and exploring of diverse materials.
Some grids we built tell stories similar to an article (see here), some are tutorials or presentations, others are just collections of data (see here). Yet again others are like stages and have a life of their own (see here). The format has proven that it is flexible enough to be used for all kinds of applications around and for our partner’s communication needs.
After publishing our first round of Online Scores and finishing up phase one (2013/2014) the MoSys part of our (then still separate) systems was further used and developed only in specific projects.
In 2014/2015 we organised two “System Hackathons”, hosted at h_da – Hochschule Darmstadt (Campus Dieburg) with a group of select designers and developers. The hackathons also included members of the Pina Bausch Archive which was originally developed at h-da (Prof. Bernhard Thull and team). The one-week meetings consisted of a deep dive tour into aspects of everybody’s systems and then free time to explore, prototype and discuss new connections, ideas, and features. One thread that showed up began with a question from the Pina Bausch Archive about how to enable guest researchers to better dive in and work with the materials at their location in Wuppertal. The idea of a workstation, based on MoSys, that would not only give access to the data collection but would also allow for easy grouping of materials under their research questions came up. This nicely linked back to our experience with the 2013 installation in Düsseldorf as discussed above. To explore the idea of a workspace we later set up a cooperation with the archive and built a prototype that was attached to their immense linked data store and had specialised cells for selecting and showing their different data types and contents.
Working on this we realised that this use case would also allow for knowledge institutions to gain a better understanding of how their data is actually being used. And in addition, the connections made by guests might create completely new and unforeseen connections in the data set. This became the basis of a totally new perspective, “multimodal annotation”, which we will address in a later section of this article.
Since its beginning in 2010 Motion Bank was set up to also include perspectives on dance education. When we started to work on MoSys and seeing it being used we felt that it might well suit some needs of dance education: transmission of implicit knowledge and reflection on bodily practice, process and performance. In 2016/2017 we got invited to join the “Dance On, Pass On, Dream On” (DoPoDo) Creative Europe project which became the perfect setting to further explore these ideas. Motion Bank's role in the project was to work on an “Online Toolbox” to bring together, make visible and available what all the partners were working on. On the education side project partner Codarts Rotterdam was working on multiple aspects: transmission of knowledge of senior teaching staff to future generations, process and method documentation of dance therapy sessions, supporting students' personal progress through self-reflection and feedback and monitoring and support of health assessments. This gave us a broad basis to put both Piecemaker and MoSys to use, and it was in the realm of this project that we rewrote and united the two tools into the current Motion Bank system. In regards to MoSys, the most interesting results for us were seeing it being used as a publication format for final projects in the master's program, working on ideas of automatic and personalised grid generation based on health assessments and prototyping feedback cells that allowed in-grid collaboration on the user level (vs. in the editor-system level).
Following up on the discussions about a workstation with the Pina Bausch Archive, we came to the conclusion that MoSys could be used in the context of exhibitions as exploration terminals and also to plan and even steer whole interactive installations and rooms. This idea came to realisation in the “Between Us” project and exhibition that we worked on with the Staatstheater Mainz and Kunsthalle Mainz between 2018 – 2020. There, the question of exchange between artistic disciplines was at the heart of the collaboration, and the Motion Bank system was used to both collect data (Piecemaker) but also to communicate the process and give visitors to the exhibition the means to explore it more deeply (MoSys). One dedicated room of the exhibition at the Kunsthalle Mainz was set up as a physical MoSys grid with interactive components and four workstations that could be used by visitors to dig into and further explore the collected materials. It was great to see some visitors spending up to an hour just with these stations.
In 2020 we were seeing new Online Scores being released in parts stemming from the opening of our systems as reaction to the pandemic, but also as part of our own internal research. In September 2020, we published a “behind the scenes” look at the process of creating Threaded Fine, a 5 hour durational work by the choreographer Rosemary Lee. Commissioned by ŻfinMalta, Malta’s National Dance Company, the creation of Threaded Fine took place in January 2020 and involved the development of 23 separate solos by 10 members of the dance company and a cast of 13 other performers. The Online Score has recordings of the full performance, offers insights into how the solos were learned and practiced and includes a selection of interviews with the performers sharing their experience. This project shows the potential for an in-depth study of choreographic processes to be conducted by a single researcher equipped with a video camera, tripod, wireless microphone and computer running our systems.
Since the beginning of MoSys we’ve been curious about how it might be used by other researchers to study the published materials. To better understand this we invited media & performance scholar Danae Kleida to revisit our Online Score “Seven Duets” (2013) about the work of Jonathan Burrows and Matteo Fargion. In the article “Entering a dance performance through multimodal annotation: annotating with scores” (International Journal of Performance Arts and Digital Media, Special Issue 2021: “Digital Annotation and the Understanding of Bodily Practices”) she reports on how she went from arriving at the joint work of choreographer Burrows and musician Fargion with almost no prior knowledge to studying their work through the publication and bringing that together in her own Online Score.
Multimodal Annotation
As mentioned before this term refers to the aspect of creating annotations inside MoSys’ grids through placing (spatially) and hence (conceptually) relating various (local or remote) materials through cells. As Kleida puts it in her 2018 master’s thesis “On the Technological Conditions of Representation of Movement”:
What makes the MoSys presentation system so distinct is that it allows more than just a text-centric examination. In fact, through MoSys it is possible to annotate dance with images, sounds, links, videos, additional dance videos, other grids, and so forth.
This is a very interesting and fruitful perspective for us as it extends our current research in the methodology and application of text-based annotation (mainly through Piecemaker). On one hand, it links into current pop-cultural “annotation practices” like responding with emojis or gif animations. Looking at it more generally it makes use of the idea that data points (materials) themselves can become means of expression through the contextual relationships they are placed into. And in reverse also become a kind of taxonomy. This is also very common in research, just consider the listings of citations that a specific paper has received. On the other hand, multimodal annotation also dips into functionalities like making audio-notes in WhatsApp or similar chat software. This is interesting because of two aspects: the media form that is chosen for what type of message and also the ease of use that comes with voice recording. Just consider the difference between placing a person cell (linking to an artist for example) into a grid versus having to manually add a profile entry with name, picture, title, link, etc. The outcome might visually be similar, but besides the extra work invested also only provides very loose connections to other grids or external instances of that person being mentioned.
Next Steps
We have a long and curated list of feature requests and ideas that we would like to explore in the future. Systematising the attaching of data sources, being able to customise cells, or even add new types, easy configuration of the messaging system… to name just a few. Of course, there are also some bugs that need to get squashed.
During the last months, while everybody is working from home, we all have seen the rising demand for tools that help foster exchange. Most of the tools that we’ve seen being used – and some we tested – seem to fall in the categories of communication (chats, whiteboards, etc.), organisation (to-do lists, mind-maps, etc.) or collecting (image boards, link lists, cloud storage, etc.). But only few seem to focus on exchange around bodily practice, performative process or ephemeral content and this is why we opened up our systems last year. We are seeing a growing user group apply Piecemaker for reflection and feedback and it makes us wonder how more of this can be translated into MoSys especially in regards to being able to respond/exchange through other means than just text. This raises not only questions about types of content that can be added but also how connections between cells in the grids can be made and made visible. This is why multimodal annotation will be a future field of exploration.
Another field will be to look into how to capture and handle the state that describes the annotators perspective extending the aspect of time based annotations. The Piecemaker user interface now has an experimental 3D viewer as alternative to the video player. It allows to not only navigate time but also change the camera perspective. This raises the question of how to deal with the additional information of zoom level and point of view in annotations. These specific conditions might be important for understanding the annotation.
Motion Bank is fortunate to just have received funding from the ministry of science, education and culture (MWWK) of the state of Rhineland-Palatinate to work for 18 months (2021–2022) on making its system more mobile-friendly responding to needs that arise from remote collaboration in the body-based practices. In #cotanz, we will also work on integrating AI machine learning to support referencing and interaction in dedicated mobile interfaces for use during dance practice and in parallel to performing or in performance-observer scenarios. This also sheds new light on MoSys not only in terms of mobile-friendliness but also in terms of how to deal with these new forms of relations in regards to representation and interaction.
Online Scores
Here is a list of all Online Scores that we are currently aware of:
2009 _ Synchronous Objects / William Forsythe: “One Flat Thing, Reproduced” _ Palazzi, Zuniga Shaw / Ohio State University
2013 _ Using The Sky / Deborah Hay: “No Time To Fly” _ Motion Bank
2013 _ Seven Duets / Jonathan Burrows & Matteo Fargion _ Motion Bank
2013 _ TWO / Bebe Miller, Thomas Hauert _ OSU / Motion Bank
2018 _ Online Toolbox / Martin Nachbar, Jan Martens, CODARTS _ Motion Bank
2019 _ OFFprojects / Amos Ben-Tal _ Suzan Tunca / ICK Amsterdam
2019 _ Motion Bank Lab Brazil _ Scott deLahunta / Motion Bank
2019 _ Between Us / Taneli Törmä: “Effect” _ Motion Bank
2020 _ Somatic Sauce _ Amelia Uzategui Bonilla / Dance educator, researcher, alumni MA Contemporary Dance Education at Frankfurt University of Music and Performing Arts
2020 _ Motion Bank System tutorials _ David Rittershaus / Motion Bank
2020 _ Threaded Fine _ Scott deLahunta / Motion Bank
2020 _ Seven Duets, Revisited / Jonathan Burrows & Matteo Fargion _ Danae Kleida / Motion Bank
