Image for post
Image for post

Writing a Twitch Bot Which Automatically Detects Music Running in Live Streams Using ACRCloud

Juri Adams
Apr 10 · 5 min read

This is the story behind my Project called Twitch Music ID, which originally started out as a school project but rapidly gained the attention of many streamers and is now part of more than a hundred channels.

Tech Used in This Project

  • ACRCloud
  • Cloud Firestore
  • Written in TypeScript running on Node.js
  • Many awesome open source npm packages! 📦 ❤️

The Challenge

As I previously mentioned, the project originally started out as a school project. In my school we had the possibility to work on a project of our choice for some months and then present the technologies we used later at the end of the school year to get our final grade on it.

I am a big fan of video games and also like watching people play them on a platform called “Twitch.tv”. Many live streamers there have music running in the background. Most of them though, do not display the current song on their stream which leads to users asking in the chat what song is currently playing. And the more viewers a stream has, the more people tend to ask. So I had quickly found my challenge:

“Create a Twitch bot which can detect music running in live streams.”

Of course, there were already other bots performing similar tasks, such as the StreamElements or StreamLabs bots. The issue with these are, that they are hooked to the streamers Spotify account and use the Spotify API to get the currently playing song. This doesn’t work when the streamer is listening to music on any other platforms though, such as YouTube Music or Deezer, which is a big flaw. I wanted to create a bot which can simply join a channel and “do its thing”, without much or any setup needed at all.

Where to Start

The first step when starting a new project is always making a quick “sketch”. This way, you can already find some potentials flaws you then don’t run into later on, which can save you a lot of time. I am speaking from experience. I have quickly decided to write the chat bot in TypeScript since I am most familiar with it and compiled to JavaScript, it can be run on a server around the clock. But I will get more into that later. Here’s what the final “sketch” looked like:

Image for post
Image for post
Twitch Music ID Project Flowchart

Song Identification

The key part of this project is to get the actual songs from the converted mp3 file. I have compared a total of four third party services and found out that ACRCloud is the most reliable of them. They offer lots of features such as their Audio Recognition API which I am using for this project. Their API analyzes a given mp3 file and returns found songs in a fraction of a second and has had the best detection rate out of my 50 tested sound files recorded from various streams.

What it Does

While looking at the flowchart above you may wonder “Why is there a separate API server between the chat bot and the ACR Cloud API?”

Having the separate API server between the chat bot and the ACRCloud API enables me to process the data before handing it back to the chat bot. Here’s what happens on the API server when a request from the chat bot comes in:

  • Upload the given sound sample to a storage bucket
  • Add a new database entry including some useful attributes such as the time the request was received, the user who requested it, the complete message, …
  • Make a new HTTP request to the ACRCloud API which then detect songs the sample may contain

after the ACRCloud responds and returns any found songs, the API server then

  • Parses the response body
  • Stores found songs in the database (not the actual songs, just objects containing the song name, artist, label, …)
  • Queries the YouTube API to get a YouTube link for the found song
  • And finally responds to the chat bot, which then again responds to the user with a short, clean message including found songs

And all of this takes about 0.5 to 0.75 seconds average! (After the initial five seconds the chat bot takes to listen to the stream)

Channel Customizations and Live Updates

Every streamer is different and so is their chat. So the goal for the bot was to be as customizable as possible. I am using Googles Firestore as the main database which allows easily storing and retrieving objects in JSON format. On top of that, you can also edit database entries on-the-fly in the Google Cloud Platform Console and applications can subscribe to these change events.

This makes it really easy for me to change channel configurations in the database and the bot being able to instantly pull these changes and make use of them:

Image for post
Image for post
Example of How to Make Use of Firestore Real Time Updates

A channel configuration contains:

  • The name of the channel
  • A cool down period (defaults to 90 seconds) in which no songs can be requested
  • An object containing various message formats which differ for every channel
  • A set of triggers the bot scans messages for
Image for post
Image for post
Example of A Channel Configuration Object

As for the messageTemplates , the bot fills out all values between the %-signs, such as %REQUESTER%, with their respective values.

What’s to Come

Despite the official school project being over already and everything done so far, there’s a lot more planned. One of the main features being able to add the bot to any channel via a web dashboard where you can log in with your Twitch account and manage the bot configuration yourself.

Another big step coming up in the very near future is the Chat Bot part going open source! 🎉

You can already find the repository for the project over at my GitHub: 4dams/twitch-music-id.

Thank You

That’s pretty much everything about this project so far! If you have any questions, feel free to let me know!

You can also email me over at juri@4da.ms

I also want to use this opportunity to thank everyone who is participating in this project. Especially all the streamers who allowed me to run this project in their channel: RATIRL, ROSHTEIN, Kurttens, linnuwu, ParnstarZilean, RANGERZX, xArmin, Kattawina_, and many more…

Also thanks a lot to the people over at ACRCloud for offering such a great platform and API for audio recognition. They have helped me a lot and the project would not be possible without them! ❤️

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store