The Day Cymbal Got DDoS’d

by Gabe Jacobs

You never think about why someone would want to harm your app until they do. Cymbal? A social music app? Why would anyone have something against us? The thought of an attack seemed plausible, but so unlikely that it was never taken seriously. Furthermore, if everyday you go into work thinking about how to get more people to join your app, it’s hard to take a step back and think about ways to keep certain people out of your app. We want to be working on building new features, making an impact in the world of music, changing how people listen — that’s the fun part.

Unfortunately, that’s not real life. In real life, fun things can be ruined by bad people. It can happen to anyone, at any time. Cymbal learned that the hard way.

Let me set the scene.

It was June 29th, 2016. I was watching a movie with my roommates after having an amazing day at work. We had just run a massive contest with our partner The Needle Drop, our user numbers were skyrocketing, and we were the highest we had ever been in the App Store rankings at #61 in Music. Even though you try not to focus too hard on user numbers, seeing the amount of activity certainly felt amazing. It was one of those days that we all dreamed of having when we first started this thing. Plus, it was the start of summer in New York City — hello sunshine.

Everything changed at 9:51 PM. Suddenly, we saw an astounding spike in user signups.

We were getting hundreds of new signups per minute! This is the sort of spike you see when someone like Beyonce tweets about your app. At first, we all assumed it was something amazing like that (Beyonce if you are listening, please, it’s not too late, I beg). We actually spent some valuable time searching for tweets and articles to see if we had been written up somewhere.

Sam had realized that every signup we were getting had an IP address that came from countries like Saudi Arabia, Iran, Oman, etc. This didn’t immediately scream “attack,” but it certainly was strange. I decided to look at the logs to see for myself what Sam was talking about.

Before I go on with the story, let me very quickly explain what a DDoS attack is.

Servers are machines that take requests to provide data to users. No matter how much money you put into a server, or how efficient your algorithms are, there is always a limit to how many requests your server can handle at a time. A DDoS (distributed denial-of-service attack) is typically carried out by flooding a server with an unnecessary amount of requests in an attempt to overload the system.
Think about it like this: A normal person could probably sign up for Cymbal in about one minute. They also would probably not sign up again from then on. Now imagine if you could build piece of software that signed up for Cymbal in 1 second. Imagine if you could run that piece of software in a never ending loop on 40 machines at once. That’s a DDoS attack.

This brings me back to seeing those usernames of people signing up for Cymbal: 2el, 1_6, afv, 41r, 2I2, and so on. This was a crystal clear sign of a brute-force, DDoS attack. Someone was running a piece of software on our app that was entering a random permutation of three characters into the username field until it finds one that is not being used. If “2el” is taken, use “1_6.” If “1_6” is taken, use “agv,” and so on. Not only was our app about to crash, but all these users we thought were awesome music nerds were actually fake profiles created by a bot. Suddenly, it felt like everything I was happy about was a lie.

On the left is screenshot of our user database at the time of the attack. Charlie (our CEO) noticed that the usernames were not just a random mix of three characters, but also followed a predictable, sequential pattern: @aaa, @aab, @aac, etc. There were thousands of fake profiles like these. On the right is a screenshot of what one of profiles looked like. They often included Arabic as their full name and an avatar like the one above. We don’t believe these people were actually from The Middle East, but instead, spoofed their IP address so that they could hide their real location.

By 10:17, things were starting to get dire.

Amadou informed us the database was about to crash and events were being fired at an insane rate. One user who had just signed up with the username @518 jumped from zero to 100 followers in just ten seconds. We had never dealt with anything like this before, so we had to snap into action and get all the help we could. It was going to be a long night.

At midnight, the identity of our attacker was revealed.

One of our co-founders, Mario Hall, sent us an Instagram direct message he received.

Soon after we started receiving comments on an Instagram post we made announcing our contest. It seemed to be the same crew, whoever they were, and they were openly mocking us to our faces while we were still reeling from their attack.

Not only did this user @vqlv write on our Instagram account, “Damn @160z most likely all the accounts made today crashed the app,” but his profile read, “Cymbals- Verified and closing”

What. Was. Happening.

After some quick research Charlie, our CEO, discovered the “UG” could refer to a group called UGNazi. There’s no way to tell if the people who attacked us are part of this group, a spinoff, or a copycat, but reading the Wikipedia article was fascinating. UGNazi specializes in DDoS and is responsible for bringing down Twitter, Six Flags (why!?), Wawa (double why!?), and many more big name companies. Considering their name, their notoriety for DDoS attacks, and the fact that they use scare tactics such as Nazism and terrorism, it makes sense to us that these were the attackers. Oh, and by the way, they are all kids.

We spent the next three hours doing everything we could to stop the attack. At this point Cymbal was still totally unusable and our users were already complaining about not being able to log in. Amadou Crookes (our wonderful CTO) and I reached out to everyone we knew in tech to ask for advice and, thankfully, we were able to reach one of closest advisors, Elias Torres, near midnight.

Elias suggested we get a service set up called Cloudflare, a company that specializes in protecting against DDoS attacks. Believe it or not, Cloudflare has a 24/7 hotline to call when you are under attack. So at 1:40 AM, Amadou and I hopped on the phone with an incredibly helpful and smart Cloudflare support agent. He explained to us that this was a layer-7 DDoS attack and helped us set up blocking mechanisms to ensure that our app would immediately reject the requests from the attackers, saving our database from overloading.

This advice was like a cold glass of water in the desert. He immediately helped us set up a temporary block on signups from the region the attack was coming from, which made a big difference. At around 2:30 AM, the attack had subsided and our defense seemed to be holding up. After certainly one of the craziest days our lives, Amadou and I called it a night.t

The following weeks were spent ramping up our defense mechanisms and trying to remember all the lessons we should have retained from Ming Chow’s “Introduction to Computer Security” class in college (the first lecture is about DDoS 🤦). We never ended up responding to the messages they sent us on Instagram, and even though some may have expected we got attacked, we have never publicly said anything about it until today.

At the end of the day, we all look back on this in a positive manner. First of all, completing a DDoS attack takes time and money, so in a way it’s flattering what happened to us. Whoever attacked us did so because they thought their attack would have an impact. That is to say, maybe an attack like this is a sign of our growth and success for an app. It could be an indicator that Cymbal is entering the mainstream. That was at least how we thought of it.

Furthermore, it forced us to take the time and energy to make sure this could never happen again. It forced us to learn about rate limiting and identity verification, things we should have thought about when we first started. If an attack like this ever happens again, we’ll be much quicker to respond and know how to stop it. In today’s world, that is valuable knowledge.

No app can be completely impervious to attacks, least of all a tiny team like ours. It seems like major security vulnerabilities are exposed every month, with millions of emails, passwords, security credentials, and private information exposed all the time. This is the state of the internet, and we experienced it firsthand.