Automating Zelda
The Legend of Zelda: Breath of the Wild is an amazing game that I’m really bad at. I’ve read that the average amount of time to complete the main story is about 45 hours but I’ve spent about 90 hours on it and have only defeated 3 out of the 4 Divine Beasts.
If you’ve played Breath of the Wild for a while, you’ll notice that in order to buy certain things like sets of armor or fancy arrows, you’ll need rupees, the main currency of the game. Unfortunately, rupees aren’t that easy to come by. Breath of the Wild enthusiasts will tell you that one of the fastest ways to get rupees is by playing the Snowball Bowling mini-game.
In the northwest, a short glide down from Hebra Tower, there’s a guy named Pondo that spends his days making large snowballs.
If you chat with him, he’ll offer a dubious proposition. Roll this snowball down the hill and try to knock over the pins.
Each game costs 20 rupees. If you get a strike, he’ll pay you 300 rupees. That’s a 280 rupee profit!
It’s harder than it looks. Even the best snowball bowlers will throw a gutter ball every now and then. I am not one of the best snowball bowlers.
There are several guides on the Internet that can teach you how to become a better Zelda bowler. One strategy involves standing a certain way on the rock so that part of Link’s right leg is bent and positioning the Sheika Sensor over the Moose statue’s right eye and then quickly straightening your position before you throw the snowball. This process is as frustrating as it sounds.
One night, while grinding away at this task and failing to knock over pins, I thought, “There has to be a better way.” If there are people that are good at bowling in Zelda, then that means a computer can be really good at bowling in Zelda. We simply have to record and replay the movements for a successful strike. So I started researching how Nintendo Switch controllers work.
As you can imagine, there are many, many people trying to reverse engineer how different aspects of the Switch work. The most complete guide that I’ve found is this one:
It has detailed schematics of how the Joy-Cons connect to the Switch, both through the physical connection and the bluetooth connection. I found this Github issue in particular to be a fascinating read:
It tells a story of a bunch of strangers working together to reverse engineer how the controllers work. It’s hard to determine exactly what their motivations are. Maybe they want to create counterfeit Joy-con controllers, or maybe they just want to change the colors of them in the software? I started bouncing around from one forum to another until I came across this post:
and it’s accompanying repo:
I don’t think it really got the fanfare it deserved, but I’ll try to summarize the findings.
There was a controller for the Wii U called the Pokken Tournament Pro Pad.
It connects over USB. With a recent system update, Nintendo added support for this controller on the Switch. When it’s connected to the Switch, it is recognized as a Switch Pro controller. So regardless of whether or not the Switch Pro controller has been reverse engineered, this one has, which means that a USB device can be pretend to be this device!
It turns out that pretending to be a USB device is a solved problem. This is a Teensy:
It has an Atmel 90USB1286 microcontroller which combined with software called LUFA can emulate a whole range of USB devices including keyboards, mice, and most importantly for this article, joysticks. I’ve mostly seen them used in building custom mechanical keyboards, but had never used one before myself.
My initial attempts at modifying the proof-of-concept to automate Zelda were not successful, but then I got lucky and discovered this fork of the original project:
There’s a game for the Nintendo Switch called Splatoon 2 that lets you draw posts. This fork takes image files as input and emulates the Pokken Tournament Pro Pad controls and buttons to draw a 1-bit version of the image automatically. Here an example of the output:
After reading the source code, I realized that it had all the elements I needed to automate bowling in Zelda. It demonstrated how to sync the controller and programmatically simulate button presses, so all I needed to do was modify it to take input from a script rather than drawing an image.
Since I was coming up with the script myself, I derived a language similar to Logo. You may remember Logo, the beloved programming language for children in the 80s. It has a cursor called a Turtle, which you control through simple commands. It has a syntax like this:
A program like that will generate visual output like this:
If you think about Zelda, Link can be programmed the same way. Here are the list of commands that I created:
THROW and TRIGGERS are composite button presses. When you throw the snowball in Zelda, you have to push the analog stick forward while hitting R. TRIGGERS hits both the L and R triggers at the same time, which is necessary to activate the controller when it is first plugged in. Finally, NOTHING acts as a delay. There are various points in the flow where you need to wait for the next scene to load or to let Pondo finish talking.
The initial script I wrote, which had Link picking up the snowball, walking left, walking up, and throwing the snowball, missed the pins completely. It looked a little something like this:
From there, I painstakingly tweaked the amount he walked left, up, and the throw until he finally got a strike. Next I scripted the entire conversation with Pondo, including pausing at the right times. When I was done, the script could initiate the mini-game with Pondo, throw the snowball, get a strike, and complete the mini-game. Finally I created a loop so that immediately after it completed the mini-game, it would start a new one with Pondo.
Here’s the script in action, which starts after I plug in the Teensy. Over the course of almost an hour, Link throws 22 strikes in a row, increasing my rupee collection from 5827 to 12007.
If you’d like to try this yourself, the instructions and source code are available here.
