Writing a Twitch Bot Which Automatically Detects Music Running in Live Streams Using ACRCloud

This is the story behind my Project called Twitch Music ID, which originally started out as a school project but rapidly gained the attention of many streamers and is now part of more than a hundred channels.

Tech Used in This Project

  • ACRCloud
  • Cloud Firestore
  • Written in TypeScript running on Node.js
  • Many awesome open source npm packages! 📦 ❤️

The Challenge

As I previously mentioned, the project originally started out as a school project. In my school we had the possibility to work on a project of our choice for some months and then present the technologies we used later at the end of the school year to get our final grade on it.

I am a big fan of video games and also like watching people play them on a platform called “Twitch.tv”. Many live streamers there have music running in the background. Most of them though, do not display the current song on their stream which leads to users asking in the chat what song is currently playing. And the more viewers a stream has, the more people tend to ask. So I had quickly found my challenge:

“Create a Twitch bot which can detect music running in live streams.”

Of course, there were already other bots performing similar tasks, such as the StreamElements or StreamLabs bots. The issue with these are, that they are hooked to the streamers Spotify account and use the Spotify API to get the currently playing song. This doesn’t work when the streamer is listening to music on any other platforms though, such as YouTube Music or Deezer, which is a big flaw. I wanted to create a bot which can simply join a channel and “do its thing”, without much or any setup needed at all.

Where to Start

The first step when starting a new project is always making a quick “sketch”. This way, you can already find some potentials flaws you then don’t run into later on, which can save you a lot of time. I am speaking from experience. I have quickly decided to write the chat bot in TypeScript since I am most familiar with it and compiled to JavaScript, it can be run on a server around the clock. But I will get more into that later. Here’s what the final “sketch” looked like:

Twitch Music ID Project Flowchart

Song Identification

The key part of this project is to get the actual songs from the converted mp3 file. I have compared a total of four third party services and found out that ACRCloud is the most reliable of them. They offer lots of features such as their Audio Recognition API which I am using for this project. Their API analyzes a given mp3 file and returns found songs in a fraction of a second and has had the best detection rate out of my 50 tested sound files recorded from various streams.

What it Does

While looking at the flowchart above you may wonder “Why is there a separate API server between the chat bot and the ACR Cloud API?”

Having the separate API server between the chat bot and the ACRCloud API enables me to process the data before handing it back to the chat bot. Here’s what happens on the API server when a request from the chat bot comes in:

  • Upload the given sound sample to a storage bucket
  • Add a new database entry including some useful attributes such as the time the request was received, the user who requested it, the complete message, …
  • Make a new HTTP request to the ACRCloud API which then detect songs the sample may contain

after the ACRCloud responds and returns any found songs, the API server then

  • Parses the response body
  • Stores found songs in the database (not the actual songs, just objects containing the song name, artist, label, …)
  • Queries the YouTube API to get a YouTube link for the found song
  • And finally responds to the chat bot, which then again responds to the user with a short, clean message including found songs

And all of this takes about 0.5 to 0.75 seconds average! (After the initial five seconds the chat bot takes to listen to the stream)

Channel Customizations and Live Updates

Every streamer is different and so is their chat. So the goal for the bot was to be as customizable as possible. I am using Googles Firestore as the main database which allows easily storing and retrieving objects in JSON format. On top of that, you can also edit database entries on-the-fly in the Google Cloud Platform Console and applications can subscribe to these change events.

This makes it really easy for me to change channel configurations in the database and the bot being able to instantly pull these changes and make use of them:

Example of How to Make Use of Firestore Real Time Updates

A channel configuration contains:

  • The name of the channel
  • A cool down period (defaults to 90 seconds) in which no songs can be requested
  • An object containing various message formats which differ for every channel
  • A set of triggers the bot scans messages for
Example of A Channel Configuration Object

As for the messageTemplates , the bot fills out all values between the %-signs, such as %REQUESTER%, with their respective values.

What’s to Come

Despite the official school project being over already and everything done so far, there’s a lot more planned. One of the main features being able to add the bot to any channel via a web dashboard where you can log in with your Twitch account and manage the bot configuration yourself.

Another big step coming up in the very near future is the Chat Bot part going open source! 🎉

You can already find the repository for the project over at my GitHub: 4dams/twitch-music-id.

Thank You

That’s pretty much everything about this project so far! If you have any questions, feel free to let me know!

You can also email me over at juri@4da.ms

I also want to use this opportunity to thank everyone who is participating in this project. Especially all the streamers who allowed me to run this project in their channel: RATIRL, ROSHTEIN, Kurttens, linnuwu, ParnstarZilean, RANGERZX, xArmin, Kattawina_, and many more…

Also thanks a lot to the people over at ACRCloud for offering such a great platform and API for audio recognition. They have helped me a lot and the project would not be possible without them! ❤️

--

--

--

I am a 19 years old Front End Developer from Germany

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Crowdsourced Testing or in-house QA?

Gtk#: The Downsides

Vision Network x Im Community AMA Recap

comely Black onyx Si http://ift.tt/2jBkUYb

Fledge over the flow …

NLP Lecture Series, From Basic to Advance Level- (Part-2)

A Beginner’s Guide to Web Animations

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Juri Adams

Juri Adams

I am a 19 years old Front End Developer from Germany

More from Medium

Make your development easier with version controlling and NoSQL

How to make Programming Fun?

Image with person frustrated in front of laptop

Startups and Tutorials

Serverless API in Ruby with AWS Lambda