AR Melody : An Augmented Reality Application for Piano Learning

a UX design case study for Augmented Reality

Introduction

A few months ago, I was faced with choosing a topic for my final project. Our lecturers have provided several topics that we can take as a final project. But I think I want to do something more so that this is not just a final project for me to graduate. I want to experiment and try things I’ve never done before. As a digital product designer, I’m more used to making designs for platforms like the web and mobile. But I want to try something new. Then I put forward a topic that is very unique to me, I put forward the topic of Augmented Reality.

Before deciding what product to make, I did a brainstorm to select the most likely ideas to work on with limited time and limited resources of course, because considering I’ve never made anything for the AR field, I thought I was kinda reckless here. But I believe if I can do this. And then I found an interesting journal about using AR in piano learning.

The Problem

Based on several journals that I read, the problem with the piano teaching pattern system is the peculiarity of the learning method, which makes the piano teaching process very traditional. Besides that, there are also some people who do self-study or self-taught, which is usually done by watching YouTube or other piano learning videos. However, there are drawbacks to the learning process using a screen or device like this, which requires the user to visually transfer information from the device or screen which is then applied to the actual instrument.

To get the correct data and see the validity of this assumption, I tried to have a discussion with some of my friends who have done piano lessons. Here is some information I gathered:

1. How did they first become interested in learning the piano, and what did they think about when they first wanted to learn the piano?
2. How is the piano learning process carried out?
3. What are the obstacles faced during the piano learning process?

I have summarized all the conversations with 5 of my friends who had been learning piano before. Here are what they said :

After all those conversations now I can start to analyze what are they as piano learner needs and it’s so obvious.

• For the piano learners who didn’t take any piano lessons or school, they would always go to learn from youtube.
• some of them said practicing piano is so tiring and takes a long time because for those who learn piano by video and didn’t have any music knowledge would be difficult in playing music because their goals are to be able to play a song and follow the tutorial midi block on youtube so that it imposes a cognitive load for following the falling blocks and memorizing.
• when they were practice using youtube videos they got hard to recognize which music notes should they tap in the real piano because of the size difference between the piano in the video and the piano they play
• “Sometimes the video didn’t provide information about which notes should I play. So, I have to adjust the notes in the video myself to the piano I play.”
• I use my smartphone to learn the video from youtube I sometimes kinda feel confused when I have to place which notes match the piano I play.

from the information, I conclude to try to solve the problem regarding the difficulties they feel when they have to press which musical notes based on the video tutorial they watch that they have to play on a real piano.

To empathize with a user, I used to create an empathy map. Before I take a long process, I need to define definite boundaries regarding what problems I am working on and when I should stop.

why AR? isn’t android and youtube can be enough? From the information I’ve gathered the common problem when learning piano by self-taught through youtube videos is the burden of information transfer that adds to the cognitive load when they have to memorize every part, as well as the difficulty when they need to adjust the notes that fall on the piano they are playing.

As a solution, Augmented Reality (AR) technology provides an ideal piano learning environment. By projecting information onto the instrument, AR systems can create more direct interaction for users, eliminating the need to transfer information from split screens.

Ideate and Design

To be honest, since this is my first design for an AR platform, I’m quite fumbled by trying out a few AR apps. I realized that designing an AR app is very different from designing a mobile app. Not only on the appearance of the interface but the most impactful is how the user experience when using the application. AR provides the same 3D scope as the real world, this makes the immersiveness that the user feels will greatly impact the emotional experience felt.

I need to think about how accurate the tutorial blocks will fall on a real piano. At first, I thought of making a virtual piano so that it can reduce user restrictions, and also I thought of making it able to track hands while playing, so it can give points in learning the piano which makes it more interactive. I’m also thinking of making a design that can assess and measure students’ abilities. But like I said at the beginning there are limited resources and time, so I have to make a more executable design in a period of only 3 months plus technology research and development methods during the development process because there is no experience at all for this field.

All decisions I take on this idea process have been thought well by relying on the problem that has been defined at the beginning of the process with the matrix that has been decided.

Decision Matrix

Horizontal lines describe how much effort will be made to implement the idea. While the Vertical line describes how much impact you will get if you implement the idea.

After mapping the idea I decided to take these features:

• Practice playing music with the help of MIDI virtual blocks, why it would be in high impact based on a problem that I’ve mentioned before, the burden on information transfer while practicing piano through video can lead to tired and frustration.
• Tempo setting to adjust the fast and slow tempo from the played music. Tempo setting can make it easier for users to memorize the notes from the part of music that they were played
• Virtual block precision calibration with piano, this can able user to adjust falling midi block with their piano.
• Music controller (playback, pause, restart) as we know that users will be practice through video, this feature will be a must-have supporting feature. playback allows users to repeat the part they are memorizing while restart made user able to repeat the song from the start

In the early stages, I need to design how the system will display objects and how the interactions between objects and users.

After I get a clear picture of how the interaction of the application will be made, here are the user flow

I created the wireframe using Figma. This is the first design I made, there is design interaction in this application process.

Unfortunately, during the process. I did several research for AR prototyping tools, but I didn’t find what tool at that time, so I needed to implement them directly with Unity to be able to do usability tests. I know this is a very large and tiring cost, but I didn’t find another way at that time because our platform specifications were targeting android mobile and using VR Box. Why am I choosing android for the platform? It’s because the range of users that will be obtained is quite large. However, recently I found an application for AR prototyping, that is Adobe Aero but as I know it can be implemented on iOS only. But you better check on their page Adobe Aero.

Apparently, the results of the first design have many shortcomings, here is the feedback that I got:

• the calibration section can only move sideways, which makes the accuracy results for the precision of the virtual block fall with the piano not good because sometimes the object appears too behind from the piano
• The set tempo cannot return to the normal position, it makes the user need to exit the application first to repeat the song to be played

So, here is the second iteration of the design I made for this app

AR Learning Practice Wireframe

Style Redesign

In this second iteration, I also do redesign the style and color, I do believe in how powerful a beautiful interface will be to gain user interest and excitement.

Overall, the user flow of this app will be as follows

I went back to doing usability testing on this latest version of the application, and yes.

Design is a never-ending saga. Nothing is perfect except god :’)

I got feedback again of course. what distinguishes it from the previous version is, in this second version I do usability testing not only on potential users but also potential companies, thank God this company appreciates what we offer them, and here are some of the feedback I got:

• need to add a music sheet, because if the user only plays using the tutorial midi beam, then the user cannot play music. But the user can only play songs, not music
• can add some piano playing techniques such as C major scale, D major scale, F major, etc
• according to them if the user is a child, the initial settings for the application are still too complicated
• The application is still too laggy and there are too many scans to using the app, it’s still not seamless (more on engineering problems)

This information is kind of new for me because at the beginning of this process I had been asking some people who mostly learn by themself. So when I approach some institutions and piano learning schools I gain some new fresh information about how the piano teaching method is. But, as a designer to make the process that is done does not become an infinite loop, because this may be what we call a wicked problem. Therefore, at the beginning of the process, I make sure what is the main problem that I will try to solve. In my case, I want to improve the piano learning process through YouTube by reducing the burden of transferring information from the learning process. This means this information will be very useful for the further development process, not for the process I’m currently working on.

Here is some image from the implemented system :

if you want to see how this application work, you can check on the video below

I also do quantitative research, I use Likert method for the research because that is the method that my lecturer advise me to use. Likert items are used to measure respondents’ attitudes to a particular question or statement.

Conclusion

Overall, users have a pleasant experience after trying out the new design prototype. Innovative learning methods and beautiful UI can increase user interest in using the application. Calibration settings to adjust the accuracy of the falling of the blocks on the piano enough to give a good impact on the user experience while practicing piano songs through video.

However, not a few complained about the need for a VR Box, it became a limitation for them, because not many had it and to do the initial setup of the application was quite complicated because it was very laggy and the user gave a not very good response to it.

Key Learning Point

what I learned :

• The design process is an abstract process. I mean, it’s hard to know for sure the answer from the process I’m working on is correct or not.
• Design is not a linear process it is necessary to do a lot of experimentation and do a lot of ideation to get the right result. results that meet the constraints and requirements of the project.
• AR provides the same 3D scope as the real world, this makes the immersiveness that the user feels will greatly impact the emotional experience felt.

obstacle :

• Frustration, because of the lack of learning resources, it was enough to make me tired and confused in the middle of the process to determine each design decision.
• Target user that is too specific (must to have a piano and VR Box) making it quite rare. To get around that, I do a lot of exploring every community (literally any community) through social media

Special thanks to my teammate Aliyyulman Jihan and my lecturer to help me decide every decision and finished this project.

Also thanks to my mentor Ravi Mahfunda, for reviewing this article and give me a lot of feedback and lessons.