Applying Deep Learning to Puzzle Level Design
Over the winter break, my team at Calclavia started working on a mobile game called Relay. Relay is a puzzle/arcade game where the goal is to relay a circle to a goal block (now available on App store and Google play!). As you swipe to change the circle’s direction, the circle moves faster, making it harder to swipe on time. If you hit any walls, you lose.
One of Relay’s key feature is that every level you play is different and tailored to your skill level, giving you a unique play experience every time you come back to the game. Traditionally, game designers would create a set number of levels and release it along with the game. However, our vision for Relay was to create a more personalized and refreshing experience. Our design posed an engineering challenge: how can we automatically design levels in a game that are fun and engaging?
Most games that create “infinite” content use sophisticated algorithms. Minecraft is a game that creates procedural terrains so you have an infinitely large world to explore. Instead of using an algorithm to generate a level in the game, we used a novel approach by training an AI to be the level designer. Unlike an algorithm where the programmer explicitly gives rules to create a level, our AI learns how to design levels through experience. We used deep reinforcement learning as a means to train the AI to design levels that match desired qualities.
I like to think of reinforcement learning like training a rat to do a task. Let’s say you want to train the rat to press a lever whenever it hears a bell ring. You would give the rat a reward (cheese) whenever it does something right or a punishment (an electric shock) when it does something wrong. (Don’t worry, no rats were hurt to make this game!) After making many mistakes, the rat eventually learns to pull the lever. In psychology this is known as conditioning.
Similarly, we created a reward/punishment environment to train our AI level designer for Relay. Whenever the AI designs a good level, we reward it. Whenever it designs a poor level, we punish it.
This raises a tough question: what defines a good level?
A video from Extra Credits states that pacing in a game should resemble pacing in a story — have some kind of exposition, rising action, climax and resolution. The idea is that humans become acclimated to stimuli and the most interesting experiences come from an oscillation in the amount of engagement an entertainment provides.
I took this advice and created a reward system such that whenever the AI follows this pacing curve in its design, we gave it a reward. Pacing in Relay comes in the form of swipes the player has to make in succession to reach a goal. The game becomes more intense as the circle moves faster and the player has to swipe faster. Based on this knowledge, I designed the system so that it rewards levels where the number of swipes needed increases until a certain climax, then decreases so the player has a chance to relax before starting over in the engagement curve.
Surprisingly, this reward system worked remarkably well. The AI learned a design philosophy and created interesting levels. I took this system a bit further and trained the AI to create different levels based on a desired difficulty, allowing each player to get levels based on their skill in the game.
Relay is definitely not the first game out there to use AI, but I think as machine learning techniques advance, it definitely opens a whole new level of AI that can be applied to games. Just imagine what it will be like if characters in RPG games employ natural language processing to understand what you say to curate a better response! As a developer, I’m definitely going to look more into applying machine learning techniques for my future projects.