Self-surveillance and Pervasive Data

In a newly-emergent reality of ubiquitous data trails and automated surveillance, what kind of artistic strategies might be worth pursuing?

Originally presented as a talk at Scope Sessions #66, in Panke, Berlin, on Thursday, June 8th, 2017.

Hi. We are Kata Kovács and Tom O’Doherty. We are an artist duo, and we live and work in Berlin. We have worked together, since 2011, under the name Kovács/O’Doherty.

We’d like to talk through some ideas about technology — specifically, about the newly-emergent phenomenon of smart speakers, such as the Amazon Echo, Apple HomePod, or Google Home, and the strange overlap that these innovations have had with some of our own work, and the work of other artists.

We’ll get to that in a moment. But we’d like to start by briefly showing some of our previous work, to give something of a quick introduction.

We tend to work with artistic ideas related to time and duration, repetition and processes, the relationship between sound and architecture, and the combinations of time and movement. We have created works for indoor and outdoor spaces, and they have lasted anywhere from fifteen minutes to multiple years.

The first work we’d like to show is a performative sound installation, which we presented in 2013, called Echoes.

Echoes (2013). Photo: Trevor Good.

In this piece, five cyclists are guided through a series of singing scores by a sixth participant. This all occurs inside a large courtyard — one of the iconic Berlin Hinterhöfe — so that all the sound created by the participants reflects inside this architecturally reverberant space.

As the cyclists move, the sources of the audio — their voices — also moves. As this audio bounces back from the walls of the courtyard, it resonates in unpredictable ways, blending the original source of the audio and its acoustic reflection.

The second piece we’d like to show is a durational installation, which we first presented in 2014, called Increments.

Increments, as presented at Serralves Museum, Porto, in September 2015. Photo: Filipe Braga.

This piece uses a process of layered and repeated recordings of drum sounds and room tones. Over the five-hour-long course of the piece, in a series of slowly-repeating actions, two performers — us — record and play back sound on two drums. We switch between being active and contemplative, alternating between adding sound on the drums and sitting down to listen to the result.

With each switch, a new layer of recording is formed, of both the drums and their acoustic reflections in the room. Over the course of the five hours, the piece slowly transforms from sparse, clear sounds into a dense sound-field of rhythm and feedback.

Yes, five hours.

The third piece we want to mention, which we’re mainly busy working on right now, is a new sound installation, called Signal Tide, which combines live, real-time signals from an abandoned satellite with accompanying audio.

This satellite, the LES-1, was launched in 1965. It functioned normally for the following two years. It then ceased to function in 1967, and was abandoned as space-junk. However, in 2013, a signal from the LES-1 was unexpectedly received again, for the first time in 46 years, by an amateur radio enthusiast. The satellite — now tumbling slowly in orbit every four seconds — gives a unstable but steady signal, with distinctly-audible gaps due to this tumbling action.

Spectrogram image of an excerpt from the signal of the LES-1 satellite, showing the gaps in transmission approximately every four seconds.

We are fascinated by the poetic idea of this mysterious resurrected object, and our installation will combine the audio from the satellite’s signal with a musical ‘answering’ signal — the satellite will pull an earthbound ‘tide’ of sound toward it, in a fleeting accompaniment, as it passes overhead on its looping extraterrestrial pilgrimage.

This work will be presented later this year, in September. Here’s a picture of us gathering some satellite signals in the German countryside a few weeks ago, with one of our home-made antennas.

So, these were the works we wanted to mention, in order to give an idea of the kind of thing we do.

Along with these works, we have another, ongoing work, called Minute/Year, which we would like to mention in a bit more detail.

Minute/Year — Spectrograms, 2016, Day 1 — Day 366

Minute/Year is an automated, durational work, in which sound is automatically layered in a resonant space, for one minute each day.

Picture it like this: in an ordinary living room, there are two microphones installed on the wall on one side, and there are two speakers installed on the other side.

Every day, at the same time, the microphones record the sound that occurs in the space, during one specific minute. At the same time, the speakers play back the previous day’s recording — the sound that was recorded in the same space 24 hours before. So, each recording includes traces of its predecessor, which includes traces of its predecessor in turn, and so on, in a potentially infinite regression.

Each day’s minute-long recording is then converted into a spectrogram — an image of the frequencies of the audio — and the two parts are then published online. This entire process is automated — it happens without any human involvement. The minute runs each day at 20:17 (in 2017). In 2016, it ran each day at 20:16.

The grid above shows all the spectrogram images from the whole of 2016 in a single glance — 366 images, one for each day (2016 was a leap year).

Here are some individual images of the spectrograms generated by the installation, over the course of the last 18 months.

Minute/Year, Day 9 — Day 16, 2016
Minute/Year, Day 183— Day 190, 2016
Minute/Year, Day 14, 2016
Spectrogram screenprints, as presented at ‘Minute/Year (2016)’ at Lite-Haus Galerie, Berlin, January 2017

In January of this year, we had an exhibition, presenting hand-made screen-printed images of selected individual spectrograms generated by the installation over the course of 2016.

During the first year of the piece, it was installed in the living room of an ordinary private apartment.

In 2017 — the second year — the piece has been installed in a large open area in Lütticher 5, an apartment and shared living space in Wedding, Berlin, which is is also occasionally used as a gallery and performance venue.

With the new year, we changed the colour of the spectrograms.

Minute/Year, Day 2, 2017
Minute/Year, Day 1–32, 2017

So, practically speaking, the daily outcome of Minute/Year is a one-minute-long audio file, and accompanying spectrogram image, which are both automatically posted online each day. This means that they go on the website, and also on SoundCloud, Twitter, Tumblr, and our current favourite, Telegram, which gives us a notification on our phone each day as soon as that day’s minute has been posted online.

At this point, we should mention that this piece owes a very distinct debt to two previous works.

The first is Alvin Lucier’s 1969 work I am sitting in a room. In this work, a piece of spoken text is recorded, re-played, and re-recorded, over and over, in a room, as it slowly transforms into a fascinating and beautiful loop of room resonances. This work might be quite familiar to some, as it has been an influence for hundreds of other works, ours included.

The second obvious debt that we owe is to On Kawara’s Today series of paintings, which was begun in 1966.

These paintings are a conceptual investigation of duration and repetition, and they simply show the date on which they were painted, plainly rendered in white on a coloured background.

They were painted by the artist from January 1966 right up until his death in 2014.

We want to be clear about the influence of these two works in particular on Minute/Year. However, the piece is also motivated by many other overlapping themes.

Can surveillance be turned inside-out? What happens when public and private contexts are collapsed, for one minute each day? Is it possible to simultaneously be a listener and a performer? What ghosts lurk in the constant repetition and layering of sound in a single space? Who, and where, are the audience, in a work that is mainly experienced online?

These questions, and the discussions arising from them, frame the interests that have motivated the creation of the work.

Minute/Year has been created with the technical collaboration of Kris Slyka (who has also assisted with creating Signal Tide, the satellite-audio piece we mentioned earlier).

Minute/Year began on January 1st, 2016. It has run every single day since then, for one minute each time. It has no set end date.

Minute/Year — all minutes to date in 2017.
Minute/Year — all spectrograms from 2016 in series, in 49 seconds.

Ok, time for a little change of focus.

This is the Amazon Echo:

And this is the Google Home:

And the newly-announced Apple HomePod:

These objects, and similar ones from various other companies, are usually referred to by the generic category name of smart speakers.

They are internet-connected boxes, which contain both speakers and microphones, and which allow people to do everyday internet stuff by just saying it out loud — “hey, internet, what’s the weather like in Tokyo?” or “hey, Amazon, order me a new box of cornflakes” or “hey, magic tube, play the new Horse Lords album” — and having the box then announce the answer, or order the stuff, or play the music.

The future is tubular

In their implementation, smart speakers have a lot of practical parallels with an artwork like Minute/Year. Both involve recording what’s happening in a space, playing something back, and putting data online as a result.

However, they are also, obviously and practically speaking, extremely different to Minute/Year.

Part of what motivated us to begin Minute/Year was the possibility to create a sort of daily moment of absurdist self-surveillance. We were interested in the idea that we could deliberately turn a private space into a public one, for one minute every day.

When we began, we never thought that the whole notion could be turned on its head, and that people would start to embrace the idea of being permanently surveilled by corporations inside their own homes.

What is even stranger to think about is that, over the course of 2016, when Minute/Year began, this new innovation in consumerism has become much more widespread. At the beginning of 2016, these sorts of smart speakers were very much a tech-nerd thing. However, as time has passed, they seem to be slowly becoming ‘normal’ — and the big software companies are pushing them, hard.

Amazon Echo advertisement, Postdamer Platz, Berlin, June 8., 2017

Compare it to our phones. In 2007, only nerds and rich people had iPhones or other smartphones. However, by late-2008-ish, or early-2009-ish, more-or-less everyone had them. By 2010 or 2011, your grandparents had them.

These new smart-speaker boxes seem to be undergoing a similar process.

The ever-reliable, ever-snarky Maciej Cegłowski, owner of Pinboard (link)

Some numbers: the Amazon Echo was first released (to Amazon Prime members only, and in the United States only) in November of 2014. It was then generally released in the United States in June 2015. It was released in the UK in September 2016, and in Germany in October 2016. Amazon doesn’t disclose sales figures, but by most industry estimates, they have sold over ten million of these devices in just the last 12 months, from June 2016 to June 2017. Some estimates claim that the combined total of smart speakers of all kinds sold to date in the United States is 35 million.

In March of this year, just two months ago, audio that was recorded by an Amazon Echo was used, for the first time, as evidence in a murder case, in Bentonville, Arkansas.

The world is a strange place.

Clearly, this is an emerging mass product and social phenomenon, just like previous ones such as smartphones in the 2010s, or before that, television in the 1980s, or before that, private cars in the 1950s and 1960s.

We started Minute/Year at the beginning of 2016. In the time since then, one of the central ideas involved in it — the gentle absurdity of having a computer listening in to a domestic space — has undergone a sort of social transformation. At the beginning of 2016, humans were not being listened to by boxes in their homes, generally speaking. But humans in 2018 or 2019 will possibly feel like this is a perfectly normal way to exist. It feels a little strange, but it’s certainly one way to gain some durational perspective on your own work. There is an odd kind of irony in having the ocean of late-capitalist efficiency engulf your artwork in unexpected ways.

So we have this newly-emergent reality. However, we would like to hope that in this changed technological landscape, our little open-ended artistic project retains, or even strengthens, its social or aesthetic charge.

Here are some of the questions that we have found ourselves asking:

If everyone slowly becomes used to the idea that they are always being listened to, how does this change the act of listening?

If everyone is always acoustically observed, does that inevitably turn domestic spaces into a site of performed activity?

Can self-surveillance be a method of reacting to, or transforming, the habits and culture of this strange new era?

Any one of these questions could be the jumping-off point for a whole essay or talk — even a book-length investigation. But we would like to stay, for now, with our focus on the creative use of automated data and observation, and mention some related works by other artists.

There are certainly many other artistically-motivated projects that have been based around ideas of data and surveillance (indeed, in some ways, surveillance as a theme is an emerging artistic cliché). There are also many projects based around transforming certain unpredictable, machine-generated inputs into other, artistically-inclined outputs. However, there are a certain group of works that specifically attempt to combine self-surveillance with datasets, records, or indexes. We’d like to mention a couple of them here.

Many of these kinds of works have started from the perspective of using video or photography. One of the first of these efforts was undertaken by Steve Mann, who also coined the term ‘sousveillance’ to refer to the act of deliberately attempting to turn surveillance on its head.

Steve Mann, Visual Filter (1995)

Here’s a picture of Mann with his Visual Filter, used for continuous live webcast, in 1995. (We can now all do the same thing with the phones in our pockets.)

Mann’s daughter illustrated very well the difference between surveillance and sousveillance in a drawing, made when she was six years old.

Surveillance vs. sousveillance, as explained by Steve Mann’s six year-old daughter.

Closer to home, there are the works that have been created by Sophia New and Dan Belasco, an artist duo who live and work in Berlin. They work together under the name plan b performance.

All My Journeys in Berlin 2003–2012 (smaller scale) — Daniel Belasco Rogers

Dan and Sophia have both, since 2003 and 2007 respectively, been carrying GPS tracking devices, recording everywhere they go, so they have a permanent record of their movements in time and space. They have used this information as material to create various works.

All Our Journeys in Berlin 2007–2012 — Daniel Belasco Rogers and Sophia New (2013)

Ironically, Dan’s and Sophia’s work is another durational project that has been, in a sense, overtaken by capitalism. When they started doing this process, they were the only people in the world doing this. But we all carry GPS devices now.

More recently, there have been a series of works by the American artist Michael Mandiberg (who is probably best-known for his ongoing efforts to attempt to print out the entirety of Wikipedia).

Print Wikipedia — Michael Mandiberg

One of Michael’s more recent works is Quantified Self Portrait (Rhythms), which is part of his ‘Workflow’ series of works. Quantified Self Portrait (Rhythms) works by taking the sounds of a year of the artist’s heart rate, alongside the sound of a year of his email alerts, and playing them back in real-time, the following year. He describes the project as a way to use himself as “a proxy to hold a mirror to a pathologically overworked and increasingly quantified society, revealing a personal political economy of data.” His self-surveillance, of both his body and his work in parallel, results in a subtle, durationally-inclined outcome.

Video documentation of ‘Quantified Self Portrait (Rhythms)’, as installed at LACMA, Los Angeles

All of these undertakings — by Steve Mann, by Plan B Performance, by Michael Mandiberg, and our own Minute/Year installation — start with the implication, or the reality, of ambient surveillance and pervasive data, and stretch their artistic possibilities.

We’re curious to see where these projects will lead. Likewise, we’re curious to see what other similarly-motivated works by other artists will emerge in a new era when it slowly becomes normal to presume that you are being listened to, or otherwise observed.