Go to the profile of Dima Ryazanov
Dima Ryazanov

Collecting Pokémon at Scale

On Wednesday 8/17, we launched Insta-PokéGo, the first mainstream PokémonGo bot that runs straight from your browser with just one click. To our knowledge, Insta-PokéGo is also the most effective bot at catching rare Pokémon not in your Pokédex — it takes advantage of an army of scout bots, to know exactly where to go (more on this later).

We had some pretty impressive stats after running Insta-PokeGo for a week: we ran the bot over 400K times, caught 3.8M Pokémon, and collected 2.1B XP. Two million visitors came to our site from almost every country in the world — including tons where PokémonGo had yet to launch! We even got visits from strange places like Sudan and Syria:

And let’s not forget Madagascar.

Now, what does it take to accomplish that? We started by running 40 bots — but our queue size got so out of hand that over the next four days we had to scale it up to over 900 concurrent bots. This required running five double-extra-large EC2 instances: that’s a total of 40 CPUs and 110GB of memory! And this doesn’t even include the database, or the static content of the website.

For those of you who have used EC2 before, here’s a screenshot:

Ok, part of the problem was the fact that we wrote the whole thing in Python… but let’s ignore that for now.

The process of getting to that point — or, in fact, getting anywhere at all — wasn’t exactly smooth. Let’s start from the beginning.

Google Auth

Login is the users’ first interaction with the app — as well as with our site. Since any reasonable user already has a Gmail account and no one really cares about Pokémon Trainer Club, we felt that Google Auth was a good place to start. Google provides a nice OAuth API that allows any app to use Google login, request specific permissions from the user (such the user’s contact list), and so on. You’ve probably seen it before:

This has one key feature: you never have to give your Google account’s password to anyone but Google itself. (We all know that you shouldn’t give out your passwords to random sites, right?) Google then creates an access token, gives it to the website, and the website uses it to verify the user’s identity.

So, all we need to do is create an API app, ask the user for access, get an access token, and send it to Niantic. Seems easy.

There’s only one little problem. Niantic developers did something questionable, making their app request way more permissions than what’s normally allowed (you can find plenty of info about it elsewhere). Besides being an obvious security risk, it was a deal breaker for us because Google doesn’t actually allow normal apps to request such permissions — preventing us from getting an access token that would work with PokémonGo. (Niantic developers have supposedly fixed this issue — however, OAuth still does not appear to work with PokémonGo.)

This left us no choice but commit the mortal sin of asking for the user’s password — and suffer the consequences. We only use the password once, to get an access token, and do not store it — but even that causes a whole list of problems:

  • It makes us look like a scam or a phishing site
  • Two-factor authentication is not supported (users need to create an app-specific password)
  • Google sometimes rejects the login attempt because it looks suspicious
  • Users sometimes get emails about a “login from a computer in Oregon” (that’s the location of the Amazon data center we’re using)
  • Users would sometimes have their passwords reset

Overall, it’s a pretty awful user experience. So, we ended up adding support for Pokémon Trainer Club accounts, which solved all of our problems. Oh wait, just kidding. We not only had to ask users for their Pokémon Trainer Club passwords — but we ended up having to store these passwords in our database (another mortal sin!) because the access tokens issued by Niantic are bizarrely short-lived. Not ideal, but we made sure to tell our users to create new Pokémon Trainer accounts that didn’t share a password with any other apps or websites.

Proxies

Eventually, we got logins to work, we got the bots to work, and it was time to deploy everything to production — i.e., run it on AWS.

Well, turns out, Niantic developers are pretty clever. When the user’s requests to PokémonGo come from IPs that belong to Amazon’s data centers, it looks very unlike a normal user running the app on his/her phone — and very much like someone running a bot on AWS. So of course, Niantic blocked all of those IPs. That was too easy.

We were left with one choice: use a proxy server. But of course, Niantic would quickly catch on and block its IP as well. Instead, we used a botnet of proxies — err, a paid service that provided “private reverse proxies with residential IPs”. With enough IPs, it would be pretty much impossible for Niantic to block them all.

Given the nature of the service we were using, it was not surprising that many of those IPs were unreliable. Some just didn’t work. Some appeared to be blocked. So, whenever a bot encountered a problem… we just picked a different IP and tried again. It was a pain — but this was the best we could get. Other proxy services were either already blocked or worked even less reliably.

Scouting

There are actually lots of other Pokémon Go bots out there, so you might wonder why we even bothered building our own, or what makes ours different. Most of the existing bots did one of two things:

  • Wander around and create a map of known Pokémon — mainly for people who wanted to know where to go to catch them
  • Wander around and catch whatever Pokémon happen to be there

Our solution was to combine the two: we had a small number of accounts that did nothing but find Pokémon and store their information in a database; then, when users started a bot, it would go directly to the known Pokémon and catch them. This has several advantages, such as prioritizing rare and valuable Pokémon not in the user’s Pokédex, and reaching them quickly instead of spending time looking around. The design also scales really well: a small number of scouts can provide information for any number of users.

We actually used an existing project called PokemonGo-Map for running scouts. We made minor tweaks to it to make it store data in a way that could be used by the user bots, but otherwise, it worked great as is. It also produces pretty maps:

Look, it’s a Pidgey!

Given that data, the user bot’s job was pretty simple: pick a valuable Pokémon, move there at 60mph (Niantic doesn’t seem to mind that speed; you don’t even need to follow roads!), catch it, and move on to the next one. Oh, and stop at Pokéstops along the way. That’s it!

Now, remember when I said “a small number of scouts”? Turns out, a user who is using PokémonGo 24/7 looks suspicious enough to cause problems. Even if it’s various kinds of “soft bans”, it means having to constantly rotate our scouts, and often even create new accounts. We added some logic to automatically detect and rotate out banned scouts; that improved the situation, but didn’t solve the problem completely.

Unlike the rest of our system that mostly worked on its own, scouts constantly required manual intervention. We even ended up setting alerts that triggered when the number of known Pokémon in a city dropped to zero. (This, by the way, is why we only support four cities — it’s enough work as it is.)

Wrapping things up

After scaling up for a week, we realized the entire team was leaving for Burning Man )’( . We also heard rumor that Niantic would be releasing a new wave of anti-botting measures that week. Worried that Niantic might try to ban our users while we were gone, we decided to call it a wrap.

In one week, our bots had played 12 years of PokémonGo and earned enough XP to level up 10,000 accounts from 0→20. Not bad for a 3 week side project ;)

P.S. We just realized that Insta-PokéGo was basically like Burning Man. Get a team together, spend thousands of hours building something beautiful, and then burn it to the ground after a week.

— Dima, Steve, Nate, Chris, Angela, and Jiggity