The Making of myCitySounds

myCitySounds: an interactive visualization and soundmap of audio samples recorded in the dynamic and ever-changing Mission district of San Francisco. Each recording session occurred while walking between the Stamen Design studio and the Gray Area theater on Mission Street.

myCitySounds art installation at Gray Area theater, in San Francisco

Project Origins

Back in 2012, I was on a mid-career grad school adventure…in Computer Science. As part of a research project, I started exploring sound mapping and audio analysis. I was interested in how much noise pollution I experienced in my day to day life, and developed an Android app to continuously take one second recordings, every minute, of every day, for several weeks.

What were the noisiest times in my day? My data visualizations were simple summaries of my research results. They also suggested a fault in my data collection methods. I was using the phone’s built in microphone, which is optimized for filtering out background noise. I wasn’t recording the noise pollution in my life. I was actually tracking my most conversational times.

Heat map showing a week of data, with darker squares showing the noisier hours: late afternoons an evenings.

While researching prior work and similar projects, I discovered Stamen Design and their TenderNoise project. I’ve been a fan of their work ever since. Some time after finishing my degree, I found out about the Gray Area Creative Code Fellowship, and that Stamen was on the list of mentoring organizations. I had to apply!

It was a long shot. A lot of people would be applying. I proposed an ambitious project: expanding on my earlier research, adding geolocation, and developing new visualizations.

To my surprise and delight, Stamen selected my proposal.

Finding Story & Defining Scope

From the beginning, this project was about collecting data and finding the story. Going out and observing an urban environment in a new way, and visually expressing these observations.

The scope of my proposed project was way too big to accomplish in the ten short weeks of the fellowship.

How much data could I actually collect? Trying to get recordings from all over San Francisco would be very difficult. I had sparse data problems. Gaps in a map would stand out. I didn’t have microphones installed all over the city. It was just me, walking around.

Early sketches revealed potential sparse data issues. I could record on some streets, or some locations, but not all of them.

In some early sketches, I explored ways to show sampled data, rather than full coverage. I started playing with the idea of visualizing sound as I went on walks around the city, which showed some promise. But a narrower focus was needed. Instead of trying to tell a story about the whole city, this would be a much smaller story.

It seemed natural to focus on the area between the Stamen studio and the Gray Area theater, which are both on Mission street. I would be walking back and forth between the two locations throughout the fellowship, sometimes on Mission Street, and sometimes on Valencia Street (which is parallel to Mission Street). This constrained the sampling area to Mission and Valencia, between 16th and 22nd streets. It also focused the project on the Mission district, a neighborhood in San Francisco currently going through a great deal of change. My walking recording sessions would capture an audio ‘snapshot’ of the neighborhood in time.

Data Collection & Analysis

Since there wasn’t enough time to add geolocation features to the app I’d developed for my previous research, I looked for an existing app that came closest to what I needed. The OSM tracker had the essential features: audio recording, saving files with timestamp and geolocation.

It also displayed estimated GPS accuracy. This came in handy, as I learned about GPS and how signals bounce against buildings making positioning less accurate. In my walks, I tried to stay as far from buildings as I could. This helped with better accuracy, which ranged from 3–10 meters, but usually around 4 meters.

In my previous research, audio was sampled once per minute. As I started collecting data while walking, it became clear that this wasn’t nearly frequent enough. After trying several intervals, five seconds worked out best. Sampling, instead of continual recordings, was done for a statistical approach, and also to avoid recording entire conversations.

To avoid the issue with the built-in microphone in my previous research, I tested several external microphones and settled on an inexpensive omnidirectional external mic that I clipped to the back collar of my jacket or backpack.

Between the Stamen studio and the Gray Area theater, I’d walk on either Mission street or Valencia street. As I walked, I would count seconds, and press the record button when about five seconds had passed. It became meditative for me, and over the days, I became more aware of the specific sounds around me. I felt more present, in the space I was passing through.

Once I had the first audio samples, I could start working on the code for analyzing them. Data for each walk was transferred from the phone to laptop via USB cable. The Android platform stores audio files in a format that has to be converted to another format before it can be played back, so I used ffmpeg with a shell script to convert the files. Most of the code for analysis and some visualizations of the audio files was written with Python, using various scientific and FFT code libraries.

Technical description and source code, mostly written in python, is online in the myCitySoundsViz GitHub repo.

Visualizing Audio

My first visualizations for the project were of individual audio files. I found audio waveforms fascinating…how different types of sounds have different shapes. Incorporating these in the final project was on my mind from the beginning.

The early days of the fellowship were filled with brainstorming and sketching.

Thinking of ways to incorporate the waveforms, one of my early sketches reminds me of an electrical circuit.

I also experimented a bit with processing.js and played with the idea of audio bubbles bouncing around. This was a lot of fun, and reminded me of my days coding particle systems (before my grad-school adventure).

An early code-sketch with processing.js, showing wave forms in conceptual bubbles.

Identifying different types of sounds could have led to another level of detail in the visualization. Listening and classifying each recording took a long time, and only a few walks were analyzed this way. Doing the classification automatically would make this much faster, so developing a system to classify short sounds is on my list of things to work on next.

I did get a good start though, and while learning about audio classification and signal processing, I learned a bit about audio fingerprinting and generating spectrographs. These show much more detail than waveforms, and can be used to classify audio recordings. They can also be very beautiful.

Spectrographs of audio recordings.

Once I had more recorded walks to work with, I started experimenting with mapping them with d3.js and Leaflet. At first, it was just placing markers for samples in relative positions. The visualization took shape slowly, first for one walk, and then for several.

At first, I was thinking that each recording would be classified, and the dots could be colored according to what type of sound it was. But after experimenting with emphasizing individual walks, rather than aggregating the samples, I decided to go with using color to signify which samples were from the same walk.

This led to experimenting with showing “menu” of walks, instead of showing all the data at once. Offsetting the position of each walk looked interesting, but wasn’t very useful for looking at one walk at a time. Adding relative noise levels for each sample, created some interesting patterns.

Experimenting with showing a menu of recorded walks.

I decided to make a “menu” of walks, that could be examined in detail. Make them playable…like a song. Drawing a few streets and label them, to provide context.

I still wanted to show the waveforms of each audio sample, so included these in the details view for each sample. Details are visible when mousing over a mapped dot, and when the walk’s recordings are played back.

Not quite done with building the interface, but getting close.

Near the end, when everything was working and looked “OK”, I received some great feedback that helped me pinpoint what was so awkward about the version I was working on. It inspired me to rework the visualization, so that the menu was no longer off to the side.

I’d been struggling with the “menu” being too dominant, so the menu now became the visualization. By this time, my D3 skills had improved a bit, and I could implement an idea more quickly than when I started on the project.

The Installation version:

Once I started visualizing recordings on the map, I made an unexpected observation: I was walking on Valencia street much more than Mission. Why? Well, the block of Mission between 16th and 17th streets is …well, there’s a lot of crime on that block. My urban instincts subconsciously prevented me from walking on that block. I didn’t even realize I was avoiding it until I saw the data.

San Francisco crimespotting map of reported crimes, over a few weeks, near 16th street. I recorded while walking along Valencia to 16th, 16th to Mission, and on Mission.

Once I noticed I’d been leaving out a big area in my project, I changed my pattern and walked to Gray Area along Mission street. The one time I walked on the Western side of that block, I did not feel safe at all. I wasn’t even sure if I would get to the end of the block unharmed. But if I stayed on the Eastern side of the street, especially between 16th and 17th streets, it at least seemed safer.

Working with the team at Stamen was fantastic. Coming from a science and engineering background, I’d never had access to so much design talent before. The feedback I received helped shape the visualizations. It helped me step away from the details of the data and see the big picture, to get from “engineer art” to a much better design.

Challenges

For me, the greatest challenge was creative…about finding the story. I’m happy rooting around in a data set and finding new ways of exploring that data. But finding a way to share those discoveries with others, in a way that makes sense outside of my own brain? That was a challenge. It’s the age old challenge of good storytelling.

The time constraints were something I kept facing. Many ideas couldn’t be followed through on, because there wasn’t enough time. Sound classification being the one I most wish I’d had more time for.

Installation at Gray Area Theater

For the installation at the Gray Area theater, the visualization was projected against a wall and a podium stood in front of it, with headphones and a computer mouse. The idea being that people could come up, put on the headphones, and explore the sounds with my visualization.

They did! I first knew it might go well was when this guy walked up, put the headphones on and started exploring. Eventually he walked off to see the other pieces in the show. But he came back, with a friend!

myCitySounds art installation, at the Gray Area theater, in San Francisco.

The spectrographs didn’t end up in the interactive piece, but I still wanted to include them in the installation. I could take up as much space as I wanted, within reason, so I could show a few large printed pieces.

To decide which of the recordings to visualize and print, I listened to each of the 8,000+ recordings. Most of them were of background and traffic noise. But there were also interesting sounds, like sirens, voices, or music, and I created small spectrographs of these to see what they looked like. Many of them stood out as both recognizable sounds and were visually interesting.

Many of the sounds were pleasant, like laughing and music. Other sounds were painful, like sirens. So I created two sets of images and color themes. For the pleasant sounds, I used a blue-green pallet. For painful sounds, I used a grey-red pallet.

Normally, when I’m working on a data visualization, I’m focused on readability. As data art, when developing these visualizations, I was free to share my emotional response to the sounds. I wanted the laughter to look calm and happy. For the siren, I choose angry red colors to emphasize the noise painfully slicing through the cool grey mist of a foggy evening.

What does it all mean?

I don’t know what it all means. But I do know that I found it immensely satisfying to go out and collect my own data, and share my work with an audience full of curiosity.

I also discovered a, previously unknown, love of mapping. Since the fellowship, many of my favorite work projects have involved creating interactive and animated maps with data.

There was also discomfort for me, personally, as I worked on this project. I’m a coder, and I work in tech. But I’m also a long-time San Francisco local. In fact, my family roots in San Francisco and the Bay Area going back to the 1800’s. Walking around so much, I saw (and smelled) a lot of changes.

In a time of tech companies displacing and overwriting the vibrant nature of my beloved city, I walked past sidewalk homes made of cardboard and cafe’s filled with comfortable people who all look the same.

But it’s not all bad. I still run into long-time neighbors and old-timers. We stop and chat, and just enjoy each other’s company. I hear some wonderful sounds, like small children laughing, birds chirping, and live music drifting from the windows of a third floor Victorian apartment.

A big take away, for me, is the importance of resisting focusing purely on tech. Tech is only part of the equation or experience. This world that we scrape data from is full of living people. They thrive and they suffer. They…we…have bigger challenges than a messy git-merge.

What’s Next?

In the years since the fellowship, I’ve worked on a lot of great projects in startups, corporate offices, utilities, and indie games. Recently, I’ve decided to go independent as a consultant. This gives me a lot more flexibility with what projects I take on, and how I manage my time. I can go on a walk in the middle of the day, any time I wish. Which is really great!

I’ve been recording more frequently and in different parts of the city. I’ve also continued working on my sound classification system. It’s not quite done yet, but I’m happy with how it’s going and I keep coming back to it between work and other personal projects.

My goal is to have a new version of the project online later this year. For now, you can explore the online archived version I developed for Stamen.


Originally published at kristinhenry.github.io.