The Code of Music

by Stuti Mohgaonkar, Google Summer of Code 2021

mentored by Luisa Pereira

As of 2021, Processing Foundation has been participating in Google Summer of Code for a whole entire decade! Each year we’ve been honored to work with students on open-source projects that range from software development to community outreach, and this year’s cohort was no exception. Over the next couple weeks, we’ll post articles written by some of this year’s GSoC students, explaining their projects in detail and in their own words. The series will conclude with a wrap-up post of all the work done by this year’s GSoC students. Congrats, everyone, on a great summer!

In a course on music interaction design with Luisa Pereira, that I took during my spring 2021 semester at NYU’s Interactive Telecommunications Program (ITP), I was introduced to the convergence of technology and music. I learned how easy and accessible it could be for a music amateur like myself to create music with code, and it absolutely blew my mind. So I was thrilled to be selected for GSoC 2021 to work on a project with Luisa as my mentor, which was aligned with the intersection of code and music.

My project for the summer involved building an interactive web and PDF version of Luisa’s book, The Code of Music. The book aims to help readers gain a deeper understanding of the workings of music, addressing questions like: How does sound travel? How does this relate to pitch, melody, and rhythm? The book then gradually dives into explaining how anyone can create their own music — and musical systems — with code.

While the core of the project involved implementing interactive explanations that illustrate these concepts, we had a lot of work to do creating an infrastructure for the digital book. With Luisa and fellow ITP student Elias Jarzombek, who is a web developer, we worked on the following tasks:

• Building an infrastructure that would house and structure the book’s contents
• Thinking of a way to optimize performance and make the book light, as it contains many interactive illustrations and media
• Building an automated workflow that would make content creation and deployment streamlined
• Creating a style guide and generic reusable functions
• Finally, begin working on placing the book contents and building the interactive interactives

For me personally this was a great way to participate in a project that was more creatively inclined, while also being able to work on my development skills.

The “Melody” chapter in the book, The Code of Music. [image description: An animated GIF scrolls through the webpage “codeofmusic.com,” which shows animations of melody, and pitch and loudness, with accompanying text.]

The work

Infrastructure

The first iteration of the project was written in Jekyll, a site generator that takes in content in markup language and uses layouts to create a static website. For this iteration, one of the goals was to take the markup files and create not only a site but also a printable PDF version of the book.

We chose to use Magicbook as a platform because it provides a way to create web and PDF versions of the book from the same source files. Magicbook is an open-source project funded by the ITP program. Apart from providing web and PDF builds, Magicbook has a quick way to set the book structure with chapters and subchapters, and it generates a navigation menu without additional styling. It also gives the flexibility to add content in both html and markdown.

Another goal for this iteration was to improve performance. In the previous version, each interactive illustration was included on the page as a separate iframe, which slowed it down. We wanted to have a way to use inline canvases for our interactives. Naturally, the steps we followed were:

• Port the existing book from Jekyll to Magicbook
• Test out the libraries that were essential to the project and figure out how they could be bundled with Magicbook

Last, since p5.js is a very crucial library for the project—because all of our interactives would be built using it—we wanted to be able to use the p5 JavaScript files from other folders in our main content markdown files. To do this, Elias built out a separate plugin so that Magicbook could render our interactives as a separate canvas.

A diagram of the workflow for making the book. [image description: A diagram. On the left is the section called “Content Creation” on Notion. Text below this says, “Content written in plain text on Notion to allow for easy sharing and editing.” An arrow from this goes to the right side of the image; the arrow is marked with text that says, “Node.js script to pull data from notion, convert it to markdown, and push it to the github repo.” On the right side of the image is a section called, “The Book, Code of Music.” Beneath this is a diagram that shows the components of the book as they are arranged in Magicbook, including Chapters (in .md), Libraries, Interactives, and p5.js plugin.”

We spent a lot of time thinking about the best way to automate and streamline our workflows. After a lot of to and fro, we reached a solution that we were happy with.

The main thing we needed to automate was porting the content from notion to the website repository. Luisa writes all content for the book in Notion, as it is a great tool for sharing, getting comments, and collaboration. Because the content of the book would have very frequent updates, it could have been too time-consuming and error-prone of a process to copy/paste the Notion content to our repo each time an update happened. So, we wanted to automate this.

After exploring existing GitHub actions built for this we decided it would be best if we started by building our own NodeJS script. The script looks for all the chapter-related content on Notion, via the Notion API. Based on the type of content returned by the Notion API (heading, bulleted lists, links, etc.), we convert the Notion content to markdown content, and push the updated content to the GitHub repo. This script was then converted to a GitHub action that can be used in the future by our workflows. The code for this can be found here.

Additionally, each time a commit happens on the master branch, we build out GitHub workflows to deploy to our staging and production.

Interactives

An interactive, made in p5.js, to demonstrate how sound travels though the air. The user can record the audio through the microphone and see the sound particles change in real time. [image description: An animated GIF of an interactive animation. The mouse clicks a “start recording” button. An illustration of a speaking head is animated with red lines of sound particles that reach an ear on the other side of the image. The mouse then clicks the “stop recording” button, and the sound particle lines become black.]

We used p5.js to build all our interactives, which were supported by other libraries like ml5.js, tone.js, and Essential JS.

The book contains two types of interactives:

1. An interactive that shows musical concepts, such as melody, pitch, etc., which can be understood by interacting with a sound source, such as synthesized, pre-recorded, or mic input.
2. An interactive that shows how code affects parameters like pitch or tempo, where the user can play with the code and hear the change in real time.

Our first challenge was to find suitable libraries that would take sound from a reader’s mic and determine parameters like pitch and amplitude. Our second challenge was to make sure the interactions were designed in a user-friendly and fun way, and conveyed the concepts as intended. Our third challenge was to make sure the interactions were fluid, so that the idea of visualizing sound was conveyed well.

Luisa made sure we had a smooth workflow, from designing the UX of the interactions, to including the assets. Once the logic for the interaction was coded, she would play-test it and we would iterate on her feedback. The interactives are still a work in progress; we need to refine the styling, and add the other planned interactives. But it was a fun process to build out a standardized workflow for building and integrating the interactives into the book.

The repository for The Code Of Music book can be found here.

My specific commits can be found here.

Future Work

There is still a lot of work planned for the book but, sadly, time did not permit us to get to it all.

Some tasks that are on our wish list include:

• Proposing/contributing to fixes to some of the libraries we are using
• Further refinement of the interactive illustrations (for example, our real-time pitch detection logic).
• Integrate screen reader and alt text into the book