Can AI Replace An Entire Game Engine? (Nvidia GameGAN)
Overview of the paper “Learning to Simulate Dynamic Environments with GameGAN” by Seung Wook Kim et al.
Imagine building an entire game engine like the unreal engine simply by training a neural network on large datasets. This means we can completely skip the coding and development of graphics engine, physics engine, game logic, etc. One day, it might be possible to simulate the physics and the appearance of our world by simply training a neural network on real-world camera footage. Sounds very enticing now, doesn’t it?
So today I want to share an overview of the research paper that takes first steps towards achieving this goal. Published by Nvidia at this year’s CVPR conference, it is titled “Learning to Simulate Dynamic Environments with GameGAN”.
With this work, Nvidia was able to recreate the entire game of PacMan with just neural networks. The network was given many gameplay examples of PacMan and was trained to learn the rules of the game, the dynamics behind interactions of different game characters and elements as well as the visual appearance of the game environment. As you can see, the results obtained from this are quite incredible!
Let’s take a look at the GameGAN engine. Like any traditional game engine, this has a physics engine, a graphics rendering engine and a memory module. But the big difference is that instead of coding these engines by hand, they used different neural networks like LSTM, Neural Turing Machine and GANs. They even managed to learn the difference between the static and dynamic parts of the game, like the walls which are static and the moving characters which are dynamic.
The Physics engine takes 3 inputs, the action from the controller to move the player, the image showing the current state of the game, and a latent vector representation of that image stored in the memory module. It then updates this latent representation of the game and updates the memory module along with the action taken for future use. This helps it to maintain the game’s consistency like object permanence over longer period of times. Finally, both the updated memory and the game state are fed into the graphics engine to render the next image of the game, and then the loop continues.
This way, the entire game can be rendered and played by neural networks without hand-coding any of the game elements. Also note that this makes porting the game engine to different platforms very easy as long as it supports the neural network framework. The authors also present their results on other environments like the classic shooter game Doom and a custom version of PacMan that generates new levels on its own.
While this project is still in its infancy, maybe one day we will be able to use something like dash-cam footage from a race car and use it as training data to learn and recreate an entire car racing game. Now that would be revolutionary in game development!
Thank you for reading. If you liked this article, you may follow more of my work on Medium, GitHub, or subscribe to my YouTube channel.
