Mixing UX and Research Science to Make Music

Lessons in design considerations and communication best practices, from two teams that cross-pollinated and built a ML-powered music-making tool.

I co-wrote this article with Signe Nørly, a Design Lead working at Google Research, creating human-centered, world-positive AI products.

This is the story of how a three-month collaboration between UXers and ML research scientists led to a new music composition tool, thanks to some pipe cleaners and input from musicians. Along the way, we (the UXers) learned a lot about how to design for artificial intelligence, and how to work across disciplines. (Spoiler alert: Communication is key.)

A little scene-setting: Research scientists have historically been the primary actors in developing the models that power AI-enabled systems. But often those scientists lack the user experience context required to integrate AI and machine learning into consumer-facing products. That’s why Google’s People and AI Research (PAIR) launched a three-month rotation program that embedded a cohort of UXers with a host team of research scientists. The UXers helped to drive a people-centered approach to developing ML models and products, and in return they got to build up knowledge around applied research in ML. Their learnings informed the creation of the PAIR’s People + AI Guidebook, and both sides learned how best to collaborate.

The rotation was hosted by Magenta, a team exploring the application of deep learning to enhance creativity in music and the arts. Magenta has a strong track record of publishing academic articles and releasing open-source resources for musicians who are comfortable with code. They’ve also dabbled with creating small, fun, digital music and art toys for general audiences, like the NSynth Sound Maker (a collaboration with Google Creative Lab).

The NSynth Sound Maker is based on real ML — but it’s more of a toy than a tool. To create something sophisticated required using code and a command line to work with Magenta’s more complex ML models. The end result is that the audience for Magenta’s core tools was narrowed substantially, if unintentionally.

And that’s where the UX team came in. Our mission: Create an intuitive, useful music-making tool called “Magenta Studio,” while also showcasing Magenta’s ML work.

Understanding the users

Storyboarding

To set a strong, user-based foundation for the final Magenta Studio product, we needed to understand more about users’ music composition workflows. By interviewing users about their music composition process, delights, and pain points, we were able to create several 3-frame storyboards, based on the Triptech method for evaluating early design concepts.

Example of 3-frame Triptech storyboard

Creating together

We identified a group of faculty and students at the California Institute of the Arts who were interested in Magenta’s work and fit the criteria of our target audience. We asked them to use storyboard scenarios to design their own product solutions, providing four prompts to help them get started.

One such prompt was, “Think about the person you go to when you need advice or ideas about your music. What characteristics or skills make that person someone you want to come back to over and over to get advice and ideas?” Because many CalArts participants did not have a background in ML, we suggested they approach ML-driven storyboard concepts in the same way they would ask for assistance from a person.

Example of a participant team’s final concept from the participatory design session

In addition to providing rich qualitative data like the product solution above, the storyboarding session was an opportunity for our research science partners to interact with external users, some for the first time. They were able to connect with potential users, building empathy and a deeper understanding of user needs. It also made them rethink bits of their work and want to invest more time in user research in their long-term planning.

Result!

Magenta built and launched Magenta Studio, a collection of creative ML models inspired by the user research and design work. Better yet, both the UXers and the scientists got real insight into each other’s work, and can bring those insights into the next rounds of product development.

What we learned along the way

Over the course of our three-month rotation, we learned some key lessons for both design for ML, and cross-functional collaborations in this space — all of which later informed the PAIR team’s People + AI Guidebook.

Design considerations

Through concept generation, design, and user testing, we identified five key design considerations:

• Existing mental models: A user’s mental model represents the way he or she thinks about a concept or system. We found it helpful to think about how people currently solve the problem without ML when exploring designs for ML-powered products. Their current non-ML solution might inform how to design the new ML solution and make it easier for the user to understand and adapt to it.
• Transparency: If the design is not transparent about how the system works, users may develop mental models that lead them to make inaccurate assumptions about how the system works, which in turn may affect their trust in the system. With creative UIs, it’s especially important to be transparent about how user data will be used so that users will feel comfortable trusting the product with their original material.
• Augmentation, not automation: Users tend to enjoy the process of being creative and developing their content or output. Effort is part of the creative process, and a feeling of accomplishment and pride are most likely to occur when a user is working alongside an ML system that augments their abilities. In fact, it can be counterproductive to automate process and do everything for the user.
• Error handling: Sometimes ML models fail. In Magenta Studio, a failure could be delivering a musical output that doesn’t make much sense. Failures of not delivering the expected result are even more likely with creative products because artistic taste is subjective. What sounds bad to one musician might spark an idea with another. When building an ML system, it is important to build controls into the product that allow the user to correct failures.
• Feedback controls: Enabling users to give feedback when the model does not perform as expected is a powerful method to improve the models, generate personalized content, and increase user satisfaction.

Communication best practices

We found that these four strategies helped us collaborate effectively with our research science partners:

• Show and tell is important for collaboration and outputs. Seeing concrete examples of each other’s daily work early in the program helps the other group to understand and empathize with their new coworkers. It also sets expectations for the final output.
• Spend time on explaining and understanding processes, especially timelines. How long does it take to synthesize UX research findings, create the design language for a new UI, or train an ML model? Being upfront about how long key events take will make the timeline more actionable and realistic.
• Do a design sprint together. Sprinting together helps team members get on the same page about project goals and timelines, as well as share the excitement and bond over creating something amazing together.
• Introduce designs as soon as the first strokes hit paper. A miscommunication occurred for our team around the capabilities of one of the ML models. This is one of the most common issues that happen as teams are turning ML model outputs into products. To prevent this kind of misunderstanding it’s good to share low-fidelity sketches early and thoroughly review and discuss existing and planned ML models’ training data inputs, outputs, and capabilities. It is a good rule of thumb to make sure that UXers are able to accurately describe what the specific ML model does, in their own words.