Creating AI sounds from everyday objects: feminist.ai tutorial on Tone Transfer

Published in

Feminist.AI

5 min readNov 8, 2020

--

Part of the joy of making AI accessible to everyone is the ability to create new, never before seen or heard examples of the world — taken from your own experiences and perspectives.

Screenshot from tutorial — watch the whole video below!

Feminist.AI team member and mlux founder, Michelle Carney, recently led a tutorial on how to use Google Magenta’s Tone Transfer to add AI audio over your own video, like this:

Big thanks to Michelle Cedeno, hailey mah, Michelle Gong, and Matt A for being apart of this tutorial and sharing their awesome #ToneTransfers!

Watch the tutorial for yourself here:

Watch the tutorial on on how to build your own Tone Transfer with our feminist.AI members!

What is Tone Transfer?

Gif of the Tone Transfer experience (it’s a lot more fun when you play for yourself and hear the audio 🎶🎷🤖)

Tone Transfer is a website built by the Google AI using the Magenta DDSP model. From the Tone Transfer website: “Tone Transfer uses ML to directly learn the tone of a musical instrument. The technology can extract relationships between pitch, tone, and volume from very small amounts of data, and form a high quality and expressive audio synthesis model of the output instrument.”

Their example inputs include two different singing styles 🗣🎶(acapella and Carnatic singing), instruments playing 🎹🎻(synth and cello), pots and pans clanging 🥣, and birds chirping 🦜(which were Zebra Finch song recordings from our own Michelle Carney’s previous career in computational and communications neuro!)

You can transform these example inputs or record or upload your own mp3 file into 4 different output instruments: flute 🎵, tenor sax 🎷, trumpet 🎺, or violin 🎻.

How to #tonetransfer your videos

The Tone Transfer site is great for uploading or recording your own audio, but what if you want to make a cool “tiktok style” video? In this tutorial, we’ll show you how to make a video like this! (Here are some other examples, and here). I will be using this video as my example:

Tools you’ll need

Your computer 💻 (with the website: g.co/tonetransfer)
A short video you recorded 🎥(preferably 5–10 seconds)
VLC media player ✴️(to extract the audio)
Some type of video editor 🎞(I am using iMovie)

Steps to make your own Tone Transfer Video

1. Record a video to transform!

We used examples like yeet’ing gourds, crunching leaves, blenders, ASMR slime, and typewriters!
It works best if the video you uses is 5–10 seconds. We’re going to do an “AI OFF” and “AI ON” version, so it will be doubled in time. Also, shorter videos are faster on the Tone Transfer website

2. Export the audio from that video via VLC media player (like this)

Open VLC Media Player > File > “Convert/Stream”
Drop your movie file in, select “Audio — mp3”, choose where to save it and name your file (include the .mp3 too! or you can add that after), hit save!

Be sure to add the .mp3 to your file name so the computer can know what file type it is!

3. Upload into Tone Transfer

Go to g.co/ToneTransfer and select “Add your own” and “upload” your new mp3 you just saved from VLC media player

Go to “Add your own” below the Input side, then select “Upload” (to the right of “Record”)

You can hit the play arrow in the lower left if you want to listen to your recording again and make sure it is the right one :)

4. Transform it and hear the AI output!

Once you hit “Transform” it can take up to a minute to transform the mp3 you uploaded — and it works fastest if you stay on that tab and do not navigate away

Try out your recording as different output AI instruments

5. Adjust and download your favorite!

You can click the “adjust” icon (🎚) to the right of the “play” icon (▶️) in the lower left to change the Mix, Octave, and Loudness — it might take a second to load, but play around with it and explore what you like the best!

Adjust your recoding transformation — the mix changes how much you hear the original audio, the octave changes the range of the output, and loudness is how loud the output is.

Once you have your favorite, download your mp3 of your Tone Transfer using the “download” icon (⬇️) to the right of the “adjust” icon

Download the Transformations you like the best — you can see which ones you’ve adjusted as well.

6. Open video editor (I’m using iMovie)

If you do not see your original video recording in your “My Media” be sure to drag your original file into “my media” (see this page for support)

7. Put two the same video TWICE

This is going to be important because we want one movie with our original audio, and one movie with our new Tone Transfer AI audio so we can hear the difference!

Open iMovie and 1. Add your original video to your media (you can drag it in from a folder), 2. Drag your video down to the editor space (twice! one will be ‘AI OFF’ and the other will be ‘AI ON’)

8. Take the second video and “Detach Audio”

Right click on the second video and select “Detach Audio” (see here)
Once the second video has been detached from the original audio, you can choose to delete it (or keep it if you want both AI audio and original audio playing at the same time!)

9. Upload your Tone Transform mp3, and align it to your second video

Use add your downloaded Tone Transform mp3 file to “My Media”
Drag your Tone Transfer mp3 to align with the start of your second video

10. Adjust however you want!

We added text to say “AI on” and “AI off”, and changed the second video to have a warmer “filter” — do whatever you’d like, it’s your AI art!🤖✨

11. Download and upload to wherever!

Save your new video by going to File > Share > File and select your quality
Then upload to twitter, tiktok, happy birthday messages to your friends, or wherever! 🌟 Congratulations on making AI music!