Applying Deep Learning to Puzzle Level Design

Over the winter break, my team at Calclavia started working on a mobile game called Relay. Relay is a puzzle/arcade game where the goal is to relay a circle to a goal block (now available on App store and Google play!). As you swipe to change the circle’s direction, the circle moves faster, making it harder to swipe on time. If you hit any walls, you lose.

A level in Relay

One of Relay’s key feature is that every level you play is different and tailored to your skill level, giving you a unique play experience every time you come back to the game. Traditionally, game designers would create a set number of levels and release it along with the game. However, our vision for Relay was to create a more personalized and refreshing experience. Our design posed an engineering challenge: how can we automatically design levels in a game that are fun and engaging?

A video explanation of Relay’s AI

Reinforcement Learning

Most games that create “infinite” content use sophisticated algorithms. Minecraft is a game that creates procedural terrains so you have an infinitely large world to explore. Instead of using an algorithm to generate a level in the game, we used a novel approach by training an AI to be the level designer. Unlike an algorithm where the programmer explicitly gives rules to create a level, our AI learns how to design levels through experience. We used deep reinforcement learning as a means to train the AI to design levels that match desired qualities.

I like to think of reinforcement learning like training a rat to do a task. Let’s say you want to train the rat to press a lever whenever it hears a bell ring. You would give the rat a reward (cheese) whenever it does something right or a punishment (an electric shock) when it does something wrong. (Don’t worry, no rats were hurt to make this game!) After making many mistakes, the rat eventually learns to pull the lever. In psychology this is known as conditioning.

Similarly, we created a reward/punishment environment to train our AI level designer for Relay. Whenever the AI designs a good level, we reward it. Whenever it designs a poor level, we punish it.

This raises a tough question: what defines a good level?

Level Design

A video from Extra Credits states that pacing in a game should resemble pacing in a story — have some kind of exposition, rising action, climax and resolution. The idea is that humans become acclimated to stimuli and the most interesting experiences come from an oscillation in the amount of engagement an entertainment provides.

Pacing curve for Star Wars

I took this advice and created a reward system such that whenever the AI follows this pacing curve in its design, we gave it a reward. Pacing in Relay comes in the form of swipes the player has to make in succession to reach a goal. The game becomes more intense as the circle moves faster and the player has to swipe faster. Based on this knowledge, I designed the system so that it rewards levels where the number of swipes needed increases until a certain climax, then decreases so the player has a chance to relax before starting over in the engagement curve.

Surprisingly, this reward system worked remarkably well. The AI learned a design philosophy and created interesting levels. I took this system a bit further and trained the AI to create different levels based on a desired difficulty, allowing each player to get levels based on their skill in the game.

Notice how the AI designed the level so that the first two swipes had to be made in rapid succession (exposition). The following two swipes were then designed to be easier and gave the player more time to prepare (rising action). The last several swipes had to be made quickly and were quite intense, serving as a climax for the level.

In addition to designing an environment, we also had to design a deep learning model architecture that is small enough to run in JavaScript using Keras.js. To learn more about the model architecture we used, I suggest checking out our Github repository.

Closing Thoughts

Relay is definitely not the first game out there to use AI, but I think as machine learning techniques advance, it definitely opens a whole new level of AI that can be applied to games. Just imagine what it will be like if characters in RPG games employ natural language processing to understand what you say to curate a better response! As a developer, I’m definitely going to look more into applying machine learning techniques for my future projects.

If you want to check out the levels in Relay, you can download Relay from the App store or on Google play!