code{strata}: the beautiful execution of ctrl-c ctrl-v
As a Professor in software technology, I spend most of my time with my group doing software research. This means, two essential things: software is the core object of my scientific work; I disseminate research results towards our scientific community and towards society. As part of this activity, I face one major challenge: software, this incredible object that we analyze and transform every day, is both extremely complex and completely invisible. Programs, algorithms, source code are all visible and can even be read out loud. But software, the thing that fuels the digital services that make our daily lives, from medium to instagram, wikipedia to Android, is much more than the code that we can read. It is the composition of this code with a large amount of libraries, runtimes, drivers, network. And, most importantly, software is the execution of tens of layers of code that run to deliver the service. Software execution, is intangible, invisible.
Software execution, is intangible, invisible.
We have worked with a group of music scholars to embody the execution of software through sound and images. Our goal is to explore different forms of artistic expressions to mediate software execution to a public who uses software all day long but has never experienced its inner workings. We aim at mediating the stratified aspect of software execution: the fact that when a user triggers a digital action, this eventually triggers the execution of a function that triggers another function, that triggers another function, etc. These functions are organized in different strata that are assembled by teams of developers to build various software services. Some of the very ‘deep’ strata are about the fundamental functions that deal with the interactions with the hardware chips; they are used by a large number of developers. Meanwhile, the ‘upper’ strata implement functions that are closer to the users and more specific to one service.
With the two pieces of code{strata} we wanted to let the user experience the rich and deep interactions between these strata, when performing what is considered to be a very basic and simple digital operation. We focus on the execution of an iconic software action, one of the unique features of digital media, ctrl-c ctrl-v.
execution, when rectangles are the functions (orange for
the application itself and blue for libraries).
Blue lines represent the way each function is calling
other functions.
Deep functions
code{strata} v1 conves a positive and tangible sound of software execution.
The idea behind this first piece is not to frighten by showing how complex software is, but to reveal the beauty of those great networks of organized code strata through a pure, spatial sound structure. Representing a computer system unit with a concrete block was a way for us to illustrate the opacity and the simplicity of high-level interfaces. A very strong contrast is made between what we may see, namely a fully closed and compact block, and what we can actually hear. One could consider this project as a zoom or a slow motion of what happens when a ‘copy and paste’ action is performed. Nevertheless, it is important to understand that this is not a realistic or exhaustive interpretation of data. Because as soon as there is interpretation, there are choices to be made. This project intends to convey a positive and tangible image of software execution. In that sense, this project aims at letting human users reclaim software complexity, as a genuine postdigital artistic installation [1].
Inspired by the work of composers like Jonathan Harvey in Mortuos Plango, Vivos Voco [2], we worked on the ambiguity between human voice and computer processing. We took a sample of human voice, whispering the sentence ‘copy and paste’. The whole sound is played back in as many fragments as there are nested functions called during the execution of the command. The length of each grain is indexed to the maximum depth of the function. If a method is called by many other methods, then we may hear a long grain (“cop”, “pa”, “and”). If it is called by a few other methods, then we shall hear very short grains (“t”, “k”, “p”). The position of the sample’s reading head is directly linked to the reading of the tree. In our case the copy and paste takes almost fifteen minutes to be fully heard. Using human voice as raw material for the sonification of the software reminds us of all the work that has been indirectly made by so many for us to be able to perform even such simple operations. We also spatialize the sound with a binaural encoding by IRCAM’s Spat [3].
Read our paper about code{strata} v1, presented at the conference on tangible and embeddable interfaces, .
Deep collaboration.
code{strata} v2 is a visual and sound tribute to the human effort invested in code strata
For this second piece, we use the same, deep tree of code execution of ctrl-c ctrl-v, and wish to insist on the human, collaborative aspect in the devlopment of each code strata. We also impose a design constraint: create a transportable artwork. That is why this project takes the form of a web application, brought up by an aesthetic so called plastic economy.
The austerity of the work, by application of the concept of plastic economy, appears legitimate. It evokes, on the one hand, the coldness of the machine and, on the other hand, our desire to decomplex and not scare the visitor, gradually giving way to wonder at the sight of all the functions called for the execution of an action as simple as the copy and paste. This second piece is designed as a visual and sound tribute to the essential role of human collaboration in the software universe.
In order to shine a light on those who have programmed all these functions, we draw through the work a kind of memorial in honor of those who participated in the creation of the ctrl-c ctrl-v. When we analyze the data collected, we distinguish two types of function calls: functions called by the text editor and those called by Javafx libraries; both represent distinct communities of developers, all working in the style of an ecosystem and this is what we try to find visually. Finally, to the visual exploration is added a sound exploration of what happens inside a computer, assuring in this way a total immersion.
The choice of breathing is not insignificant, obviously evoking the human. However, the division of the track in grains characterizes the couple tension / slackening specific to the execution of the ctrl-c ctrl-v. In addition, it is symbolic of the effort, effort of the machine that makes an incredible number of calculations for this action that seems so simple, but above all, effort of the developers who wrote these millions of lines of code.
Enjoy code{strata} v2 live!
The code{strata} team
code{strata} does not differ from any serious software and musical piece. It is the result of an exciting collaboration and of lot of work, passion, art and technology. The band members include Justine Fortier, Nicolas Harrand, Oscar Luis Vera Perez, Denez Thomas and Bruno Bossis.
If you would like to be notified with other cool results that we have, shot an email at software-research.subscribe@4open.science
References
Beyond the work cited in this post, the work was deeply inspired by the works of I. Arms [4] and W. B. Paley [5].
[1] Post-digital humanities: Computation and cultural critique in the arts and humanities. David M. Berry. Educause, 49(3): 22–26, 2014
[2] Mortuos Plango, Vivos Voco de Jonathan Harvey ou le miroir de la spiritualité. Bruno Bossis. Musurgia, XI (1–2): 119–144, 2004
[3] IRCAM. Spat. Retrieved November 4, 2017, from http://forumnet.ircam.fr/product/spat-en/
[4] Code as performative speech. Inke Arns. Artnodes 4, 2005.
[5] Code Profiles. W. Bradford Paley. CODeDOC exhibition, Whitney Museum, 2002. https://github.com/DIVERSIFY-project/CodeProfiles
