Visual Sound Instruments
This text is an extract of my thesis introduction from my doctoral studies at Moholy-Nagy University of Art & Design, multimedia department, written in 2014
When speaking of (music) instruments, some of the most problematic areas are dealing with direct attention, presence and the attitude of being in the here and now, making a healthy mix of curiosity and improvisation. These fields of course include the chance of making (and learning by) mistakes, failures and disappointments.
On the theoretical side, this situation makes me use the different forms of analogies and metafors when bringing forward new ideas, describing problems, understanding questions. New technologies, inventions are also based on metaphors when hitting the audience of the wider society. The analogy of musical instruments can be understood easily in cultural terms. While these instruments are globally present, they also reflect the needs and specialities of local communities. Their structure is very simple: making sounds with physical energy. This goal has a truly interesting, deep ergonomic tradition rooted in the past few centuries.
Software based interfaces are fresh, ephemeral tools that are lacking these type of long term cultural traditions. Therefore, comparative research of traditional instruments and digital tools are raising really interesting questions. And, last but not least, the conclusions of these investigations can be adapted into a more general approach of current creative interaction systems. Luckily, mostly I myself am making software based tools for performances, installations and different interfaces, therefore the research text and my final practical master work form a uniting whole.
Language, as a tool for manipulating abstract symbols is serving the basis for these studies on visual sound instruments. Language is a tool for interpreting the world, communicating ideas, describing thoughts and feelings on different levels of abstraction layers. Among natural languages, artificial languages have important roles in shaping the processes of the world. Language as a mimetic communication system has a nature that is generative, open, ever-evolving.
Mimesis is the first form of understanding social signs, which is mainly based on episodic fragments, thus creating an analog-like information transmission. Instead of mimetic (spoken, movie, photo) languages, the text focuses on digital, or cryptographic languages (such as musical notations, computer code, binary systems) which helps understanding the architecture of digital sound instruments. Designing with cryptographic languages needs a different understanding than the process applied in the design using mimetic languages.
The introduction of today’s widely used programming paradigms are helping to understand the different abstraction layers in interactive systems. Starting from one dimensional, text based representation, followed by a two dimensional, visual representation, it turns out that on the higher levels of abstractions, these systems are constructed of multidimensional interaction modalities, where behavior patterns, cognitive aspects and the nonlinear nature of time are also taking part in the experience.
To illustrate it accurately, the phenomenological dimension space is introduced as a method to process and categorize these dimensions.
Musical instruments naturally have these dimension spaces embedded in them, shaped by their ergonomic aspects, cultural functionalities and resonating bodies. In the case of software based interfaces, these dimensions are not originating a prior from the physical parameters of the object so they have to be built in explicitly into these systems.
This means, that the lack of a physical body induces a new type of inner coherence in software based visual instruments, which drives to unknown territories both designers and players.
If we look at the history of software based tools, there are identical shifts compared to what we find in the history of classical instruments with resonating bodies. The need of scalability, universality generates new paradigms in each lineup of the different tools. In classical music, such paradigm shift is the twelve tone tempered music from the early renaissance. This change made possible to tune the different types of instruments into a common pitch spectrum with the reduction of natural sounds, negotiating the importance of harmonics and natural overtone frequencies. A very similar approach related to digital tools is the invention of the MIDI protocol in the twentieth century, which was made for synchronizing different devices with 128 discrete values for each parameter. The invention of the tempered scale and the invention of the MIDI protocol were pretty similar in their time: they both constrained musical creativity because of the need for universality.
Reconfigurable, dynamic media systems bring forward new kind of creativities where the forementioned dychotomies are always present.
The concept of sound instrument is an extended one here. The interface includes the playground for the interaction while it is a representational surface, also. The role of the composer, performer, listener are now changed in mainstream culture such as with the rise of progressive contemporary communities of the fluxus and related art fields in the middle of the twentieth century. The composer includes notational elements in her instrument, which has been interpreted by the listener when consuming it. The prescriptive nature of notation and the descriptive nature of recording, visualization are merged, visual instruments became interactive notations at the same time.
The concept of composer, performer, audience is not valid anymore in these systems. The artworks are tools themselves the best analogy to describe them can be borrowed from the world of games. The relation of music and game are deeply rooted in our language also, think of the terms play the piano, play some music, when do you play? etc. The ecosystem of digital games are much more adaptable to the distribution of contemporary visual sound instruments than the heritage of musical production: instead of buying heavy vinyls or magnetic tapes, people get their tools and music for a few cents from online distribution channels. Since they get their material from these online entry points, realtime response from the audience is an essential component in the consuming of these artefacts. In the case of software based instruments, feedback from the audience has more and more components that works like the feedback in the consuming of games.
An immersive game flow is key element in the experience of a sound instrument: exploration, chance probabilities, playfulness and failure are all taking part in the whole. Game makers are making sonic instruments (game levels, if you prefer) where the players are navigating and the composition is brought forward during their personal journey.
