Voices of Musicians Using Neutone (Part1) — Game Change with AI: A Completely New Synthesis Method Since Granular【Shojiro Nakaoka, sound designer】
This is the first installment of a series of interviews with musicians who actually use Neutone, an AI-based real-time timbre transfer plug-in developed by Qosmo.
We interviewed a sound designer Shojiro Nakaoka (https://linktr.ee/shojironakaoka) about how he came to use Neutone and what impression he had of it.
➖Please tell us about your musical activities.
I compose and design sounds for a variety of platforms, including videos, digital signs, apps, games, and hardware — essentially any medium that requires sound.
I think my genre is mainly electronic music, but my work involves more than just straightforward sound playback. I also include interactive elements and programming that shapes and evolves the sound over time.
ーAre there any artists or musicians that have influenced you?
I have been influenced by many kinds of music. For example, one of them is David Cunningham. He is a musician that I came to know through my interest in listening to punk, Jamaican music, and UK dub.
I was intrigued by his cross-disciplinary activities: he not only released twisted pop music, but also made dub mix albums (masterpieces) and later, sound installations.
One day, I read an article about one of his sound installations. In a space with walls separated by 10 to 20 meters, he placed microphones and speakers near both walls. Then he fed back the sound generated by the visitor through the speakers by the walls so that the viewer could appreciate the sound of the space itself. I was very impressed at the time. (The Listening Room)
At the time, I did not even know these kinds of sound installations existed. Although his installation may have deviated from pop music, it was definitely connected to his dub music (which also often uses techniques such as feedback/resonance). I was surprised to see how this kind of artistic experience can be achieved by developing a single methodology. It also made me interested in digesting and deepening the methodologies of existing genres.
In that sense, when I created this audiovisual work using Lissajous images on an oscilloscope (https://youtu.be/BU7-fNt0X14), I also intended to pay tribute to the genre that got me started in music: dub.
Another way he influenced me was through his approach of creating a framework that allows music to generate itself, where the producer also appreciates the created sound. There was also the ‘Koan’ MIDI generation software supervised by Brian Eno.
ーDid your interest in such an evolution of a method itself lead you to become interested in Neutone as well?
I think so. It was for the same reason that I became interested in the Max programming environment at the time. I was excited by the new ability to program real-time audio processing by myself.
I got a similar feeling from Neutone, which uses AI to transform and create sounds, and became very interested in using it.
However, it was a bit daunting to suddenly start incorporating it, so I first attended the workshop and used it a little bit at a time while listening to the overview. I thought, “This sounds difficult, but I have to try it”. My interest in using it overpowered my feelings of intimidation
ーHow did you feel when you tried to incorporate Neutone into your own activities?
It was incredibly fun.
It was intriguing to be able to easily test the models, each of which had its own unique texture.
There are plug-ins out there that use AI on various levels, but this was the first time I had the opportunity to try training the models myself. It was a really interesting experience to be able to try using such a full-fledged AI for music production.
I used a RAVE model(https://github.com/acids-ircam/RAVE) at the time, and I trained the model on a collection of my own sounds that I had created. It was great to experience the workflow, from exporting models to loading them into a plug-in. I felt like I could incorporate AI into my own music production, and like I was able to grasp the edge of tremendous potential. I began to think that I would not be able to take my eyes off it ever again.
ーHow do you see AI plug-ins like Neutone developing and influencing the music industry in the future?
Around 20 years ago, when I first started using Max, it enabled me to process audio in real-time. I could even operate granular synthesis — a technique previously confined to electronic music laboratories — right on my laptop. This made it possible for me to experiment with advanced academic methods from the comfort of my own home.
The sound of granular synthesis itself had a profound impact on me in the beginning, but it is still relatively new among synthesis methods. Not many new synthesis methods have emerged since then. Most of them are a mixture of existing methods.
When I started using Neutone, it felt like a game-changer. For the first time in a long time, a completely different kind of synthesis method emerged, I felt. I guess there are still many challenges to overcome, but the potential changes this could bring to music production in the future are beyond my imagination.
Personally, as an avid music gear enthusiast, I’m more interested in discovering new, never-before-heard sounds than in achieving a “final form” of music. I am looking forward to that very much. I predict a new genre focused on this aspect will eventually emerge.
ーHow did you feel when you actually ran the model you tried training yourself on your DAW?
I tried training with the sounds I had created so far. I was imagining how frequently occurring sounds might stem from those loops, or how I might extract sounds from other training data. I am still in the process of experimenting and have not yet established effective methods of use, though I was having a lot of fun figuring it out by gaining experience in my own way, like inputting various sounds without thinking and listening to how a sound with a strong attack will be output, or how a chord-like sound will be output, and so on.
I think the models uploaded on the site have already been adjusted to a good level, so I would like to explore and create my own models more. When I was in charge of music production for a promotional video for ONLINE PARCO(https://youtu.be/ZYPXse6t4Nw), I mainly used existing models instead of creating models from scratch because the production time was limited.
Even so, I was able to produce some very fresh sounds that I had never encountered before, and I was very excited during the production process. I think you can feel the fun of using Neutone even more in this collection of outtakes (https://on.soundcloud.com/U3W7A).
ーWhat kind of models would you like to see in Neutone in the future?
I think all kinds of textures could potentially be covered, so it’s not so much that I wish there was some specific texture in particular. I would like to try more and more different sounds.
I don’t have a lot of experience yet, so I’m still at the stage where I can’t confidently say ”something like this would be interesting.”Anyway, I would like to try various sounds.
It would be good to have a model that is suited for sound effect production outside of music, or one that specializes in glitches that destroy sounds,
Or maybe a model that handles texture like a resonator, or one that can be used like a midi effector, etc.…
ーIf you were to recommend Neutone to others, what would you recommend?
Neutone is a totally new experience, and sometimes you get sounds that are beyond your imagination. If you like that kind of change, you should definitely try it. Those who want the output to exceed their imagination from the input.
There would be some situations where it is easier if you can imagine the output corresponding to the input, but I think it is a great toy when you want something magical to happen.
I myself haven’t been using it for very long, but at present, its mysteriousness is the most attractive aspect for me.
I believe the emergence of various models in the future will lead to entirely new developments, so I think it’s best to get familiar with Neutone now. Trying to do everything yourself from the ground up (especially with respect to AI) requires a tremendous amount of effort, doesn’t it? I feel like this tool helps us leap over that hurdle.
I’m eager to try out new models as they become available. If you want to experience something that you have never touched or heard before, I think you should give it a try.
Text and Interview : Tatsuya Mori
