Hey Alexa, meet Music

Aditya Ketkar
Jul 19, 2019 · 4 min read

A look behind the scenes of our winner entry for Alexa India Hackathon

Image for post
Image for post
A BTS of making the winner entry for Alexa India Hackers League, 2019.

Introduction

This story will talk about our experience at Alexa India Hackathon, held at Amazon during my tenure there as an intern in June 2019. I teamed up with two of my experienced colleagues Srini and Abhishek for the hackathon. There was no problem statement defined, but the idea proposed had to involve developing a skill for Alexa. Enough intro, onto the fun part now!

Inception of the idea

A lot of us learn or want to learn music out of hobby. A major setback of traditional music learning experience is that an instructor has to guide you not only through your learning phase, but also in the practicing phase. This part is time consuming and has the potential to be automated by Alexa.
Through our proposed Alexa skill, we wanted Alexa to act as the feedback provider during your practicing phase. In short, Learn music from a tutor, and practice it with Alexa! (This idea was a brainchild of Srini)

Overview of the proposed Solution

Image for post
Image for post
The high level flow of practicing with Alexa

CAVEAT : Alexa is not allowed to listen to audio of more than 8 seconds in length. Also, Alexa cannot store any of the audio in any form due to privacy policy of Amazon.
Therefore, Abhishek made a companion app for this project. All a user needs to do is to play music with the companion app activated and press the upload button in app. This was done for the purpose of demonstration in hackathon. The production (hopefully xD) version will probably support direct recording user’s music on Alexa enabled device with due permission.

Here is what your personalized conversation with Alexa will look like -

Image for post
Image for post

The video demonstration is also available here — AlexaMusicAcademy

Scoring Mechanism

There are a whole lot of complexities in coming up with a similarity criterion for audio. Direct comparison will not work here ( what if your audio is 100% similar, but there is a time lag of a millisecond? Yes, it will be scored as 0% similar). Also, factors like noise and difference in calibration of instruments has to be taken into account. This was a problem which left music producer companies scratching their head for a long time. They simply could not afford a manual workforce to moderate all the copyrighted music floating around internet!

Storytime. In 1999, 3 friends with zero engineering background teamed up to solve this problem. They found another guy — a genius with 4 degrees from Stanford, and started a company called Shazam. Shazam invented the technology Acoustic Fingerprinting, which became an overnight hit among music lovers and companies as well. This was later acquired by Apple for 400 million USD!

We found an open source library called chromaprint which uses the same technology to compare audio files. The boilerplate code was taken from this brilliant medium article by a fellow Amazonian Shivam Aggarwal. Under the hood, it compares the audio files for different offsets and returns the result from offset with best match.

We have created a simple online audio similarity scoring app using the same, in which you can compare two audio files for similarity. Here is the link to webpage. You can try out the webapp HERE.

Image for post
Image for post
WebApp for similarity score for audio files

Happy Ending :)

About a week and a half after the demo presentation, we received a mail stating we were the WINNERS of the hackathon!

The hackathon was a great journey, involving a blend of joy of creating, enriching experience of working with a great team and a sleepless but fruitful night. I want to thank my team members Srini and Abhishek for giving me the opportunity to team up. This has definitely made hackathons my newfound interest, so hope to see myself participating in one soon!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store