Artificial intelligence and music: a tool shaping how composers work

This article was translated from my capstone story published in French on Medium in May 2019

French researchers and musicians are eager to work with artificial intelligence (AI) and have started experimenting with a variety of techniques to reproduce human reasoning and learning to create art. They explained to me how their creations — a robotic pop album and an experimental play about monsters — changed the whole creative process, as if they had “poured rocket gas into a piano”.

Last February, Huawei wanted to prove that a form of artificial intelligence (AI) could finish Franz Schubert’s Symphony No 8, which he left unfinished after beginning its composition in 1822. The Chinese network equipment manufacturer worked for nine months with American composer Lucas Cantor to finish the romantic masterpiece in collaboration with an AI-equipped machine, the Huawei smartphone Mate 20 Pro. The augmented piece of work was performed by the English Session Orchestra musicians in London Cadogan Hall on February 4.

When Frankenstein meets the machine

The research team followed the rules of “deep learning”, or a ”type of AI that uses algorithms based on the way the human brain operates,’’ according to the Cambridge dictionary. After feeding the machine Schubert’s works, the software was able to formulate its own suggestions “in the style of.” Cantor then refined the results, orchestrating them into the closest possible versions of the Austrian composer’s work.

You can check out here the advertising video Huawei published in February, as the US were claiming the company was facilitating Chinese state spying operations and pressuring its Western allies not to use the Chinese technology to build 5G networks:

AI is booming in France, too. French researcher Philippe Esling works in Paris for the Acoustic/Music Research and Coordination Institute (Ircam). Last year, he took part in “Fabric of the Monsters”, an experimental play about creation and learning commissioned by Ircam and directed by Jean-François Peyret. Italian composer Daniele Ghisi and artist Robin Meier banded together with Esling’s team to compose an AI-assisted soundtrack for the play.

Esling wants to end alarmist clichés about human-replacing machines. “The AI doesn’t do anything by itself. There’s always a human post-producing the results. The goal is not to generate music by pressing a button,” he says.

“The computer is a surprise-machine”

Esling deems people are against these new technologies because they feel threatened. “If you create your work with AI, it gives the idea that AI did the work. But AI just gives us tools!”

In Mary Shelley’s 1818 novel, Victor Frankenstein’s creature, a terrifying monster who knows nothing about life or humanity, escapes from his master and discovers the concept of family by observing humans in their home. The creature quickly becomes familiar to “the human language, feelings, logic”, explains Frank Madlener, head of Ircam and commissioner of “Fabric of the Monsters”. Conceived as a tribute to Mary Shelley, the play is a metaphor of a learning process, even music-wise. That prompted Italian composer Daniele Ghisi to have the AI learn Lieder, on top of French voices and random sounds.

On the phone, 35-year-old Italy-based Ghisi explains: “The machine was learning just like the monster, from an empty state.” The errors the machine could make interested the musician — who also has a degree in mathematics — more than its mere reproductive capacities.”

You can listen here to the first track of Fabric of the monsters. It’s strange and strident:

Composing in such a fashion stimulates musicians’ creativity and allows them to own the work. The “Fabric of the Monsters” tracks credit Ghisi and Meier, while the algorithms only appear at the bottom of the website. For Ghisi, that status is ambiguous: “Every technology is a work tool. I don’t think the people who created the algorithm I used to write my electronic work are composers too.”

The issue of copyright

That doesn’t prevent him from feeling like the effort was collective. “I feel like someone who would have cultivated a field to produce a lot of flowers. Who owns the flowers? Me, because I cultivated and picked them? The people who first developed the field — the machine learning systems? The romantic composers we fed the machine with?”

French independent composer Benoît Carré, aka Skygge, has a distinct opinion: “Everything the composer creates and composes, he owns”, whether he used AI or not, he says. Carré created the pop album “Hello World” in 2017 with an artificial intelligence in Sony’s Paris lab, Flowmachines.

“Scores are not songs, they are just a structure around which you have to build your work,” he says.

You can listen here to the song “Ballad of the Shadow” , composed by Skygge on Hello World with AI:

Skygge is used to computer-assisted composing, samplers, synthesizers and the like. He sees a huge difference between musical creation software and AI. Software offers“pre-existing music”, not music “generated specifically for a project [Skygge] is working on” , while AI “suggests ideas” that match what you teach the machine.

“It’s a good alternative to the blank page when I start writing a song,” he says.

The exploitation of existing data by AI is forbidden in France, so Sony’s AI travelled to Japan, where it was fed 400 jazz scores (Cole Porter, Duke Ellington, etc.) before returning to Paris. Skygge then cut them into pieces and reassembled them in new images, led through random mouse clicks. He then chose the excerpts he liked best and refined them by composing arrangements for real instruments, recording chorus with human voices and such.

At the core of these new composing methods: the issue of copyright. Tech music startup Muzeek bet on a new status for art creators. Muzeek cofounders, musician André Manoukian and entrepreneur Philippe Guillaud, created a software which multiplies a single composition, sent by a musician, into hundreds of variations. “We turned 80 arrangements into a collection of 25,000 songs”, boasts Philippe Guillaud. “The winners of this invention are the music creators who earn a lot more royalty payments for just one track,” he explains.

Using AI for commercial purposes will be more and more common in the future, according to Rémy Demichelis, a journalist specialized in AI topics for French newspaper Les Echos: “The technology and the market are really new, it’s difficult to know what’s going to happen. There is definitely a market because background music is everywhere nowadays,” he says. “Generating music automatically cuts costs. I don’t know yet if it can become lucrative. But considering that people are willing to pay for that service, it seems plausible.”

The composer hasn’t lost all control over the machine. “AI will never help you write the ultimate hit because only humans can do that. No need to create a phantasmagoria or to refute the complete disruption of knowledge as we knew it,” concludes Frank Madlener from Ircam.