DevOps Practices in the Gaming Industry — Differences with Traditional Industries
Introduction
Game development is a complex process that requires coordination between multiple teams working on different aspects of the game, from design to development to testing. DevOps practices and tools have helped to streamline the game development process by automating the build, testing, and deployment of game assets.
In this first article, we will explore the trends in game development and how to integrate game engines and centralized Code Versioning Systems (CVS) in CI/CD pipelines, the most important toolchain of the DevOps culture.
Prerequisites
To follow along with the examples in this article, you should have a basic understanding of the DevOps culture and its main practices: CI/CD and working with Code Versioning Systems. We will focus our examples on Jenkins or GitLab CI, since they are the two of the most widely used CI/CD orchestrators in the industry.
Skills needed for DevOps Engineers starting in this industry
If you are a DevOps engineer interested in working in the game development industry, there are some skills that you should ramp up to be successful. Here are some of the skills that game development companies look for in DevOps engineers:
- Game engine knowledge: Knowing about popular game engines like Unity and Unreal Engine is essential for working in the game development industry. Familiarity with the workflows and build systems of these engines can help you better integrate them in CI/CD pipelines.
- Scripting and automation: DevOps engineers need to have experience with scripting and automation tools like Bash, Python, and PowerShell. Game development involves plenty of repetitive tasks that can be automated using these tools, so having this expertise can make your job easier.
- Mixing cloud computing, virtualization and on-premises hardware: Many game development companies use cloud computing services like Amazon Web Services (AWS) and Microsoft Azure to host their games and manage their infrastructure. Having experience with cloud computing and containerization tools like Docker can be a valuable asset for a DevOps engineer in the game development industry.
Each cloud provider has created their own set of gaming-oriented services, such as Gamelift for AWS, PlayFab for Azure, Agones by Google (cloud agnostic), and so on.
On the other hand, this particular industry also uses a lot of on-premises infrastructure. This is due to strong intellectual property rules that need to be protected, as games in early development are usually kept in secret, even from internal teams working on other products/areas. - Version control: Game development involves managing large codebases and assets, so experience with version control systems like Git and Perforce is essential. Understanding how to manage branches, merge code, and resolve conflicts is critical to the game development process.
Key differences between DevOps Practices in the Gaming Industry vs Other Industries
Here you’ll find some key differences between DevOps practices in the gaming industry and other traditional ones:
Version Control Systems
Version Control Systems (VCS) are essential tools for DevOps engineers and developers alike in the gaming industry. There are two main types of VCS: centralized and decentralized. Each type has its advantages and disadvantages.
A decentralized VCS, like Git, allows for multiple local repositories, each containing a complete copy of the code. Developers can work on the code locally, committing changes to their local repository. They can then push these changes to a remote repository when they are ready to share them with the rest of the team. This allows for greater flexibility and allows developers to work offline, but it can also make it harder to manage conflicts and ensure consistency across the team.
A centralized VCS, like Perforce, has a single central repository that stores all the code. Developers need to check out code from the central repository to work on it and commit changes back to the central repository. This creates a single point of failure but also ensures that there is only one version of the code that everyone is working on. This makes it easier to track changes, manage conflicts, and maintain consistency across the team.
Here are some key differences between centralized and decentralized VCS:
Working with big files in a VCS
The gaming industry is known to handle huge files, from graphic and animation renders to assets like audio and video files. In this area, the choice of a VCS is critical since centralized and decentralized handle them very differently:
Because of this, the gaming industry generally uses Perforce (or similar) for developing games while using decentralized ones (almost always Git) for general developments like APIs, backends and live services.
Since Perforce is the go-to solution of choice for most of the companies in the gaming industry, we’ll focus our examples on this tool.
How do I integrate them into a pipeline?
Perforce, as we said before, is a centralized version control system that provides a wide range of features, such as high-performance file locking, atomic commits, and efficient branching and merging. It can be integrated into a CI/CD pipeline using the p4 command-line tool. Here is an example of how to sync a Perforce depot in a Jenkinsfile:
pipeline {
agent any
stages {
stage('Sync') {
steps {
sh 'p4 -c <client_name> -u <username> -P <password> sync //depot/path/to/files/...'
}
}
}
}Game Engines and CI/CD Pipelines
Integrating game engines in CI/CD pipelines can be a complex process, depending on the game engine being used. Some game engines are easier to integrate than others, while some require more effort and expertise.
What is a Game Engine?
A game engine is a software framework designed to help developers create video games more efficiently. It provides a set of tools and features that help with tasks such as graphics rendering, physics simulation, audio processing, and more. Game engines can be used to create games for a variety of platforms, including desktop, mobile, and consoles. There are many game engines available in the market, but some of the most popular ones include:
- Unreal Engine: Developed by Epic Games, Unreal Engine is a popular choice for creating high-quality 3D games with advanced graphics and physics. It offers a wide range of tools and features, including a visual scripting system, a robust editor, and support for virtual reality development. Some of the most famous games created using Unreal Engine include Fortnite, Gears of War and Batman: the Arkham saga.
- Unity: Developed by Unity Technologies, Unity is another popular choice for game development. It is known for its ease of use and cross-platform support, making it a great choice for indie developers and small teams. Unity offers a range of features, including a visual editor, scripting tools, and support for 2D and 3D games. Some of the most famous games created using Unity include Pokémon Go, Cuphead and Genshin Impact.
While these are just two examples of the top choices in the industry, each game engine has its own strengths and weaknesses. As a DevOps engineer working in the gaming industry, it’s important to have a basic understanding of the most popular game engines and their capabilities.
Each engine has its own complexity for integrating them in CI/CD pipelines. They have command line tools, and you can also find uncountable wrappers and libraries done by their respective communities.
Integration of game engines into CI/CD pipelines
Next, we’ll focus on Unity and Unreal Engine, and we’ll see their ease of integration with CI/CD pipelines.
Note: All examples are done with the licensed version of each engine. Free (or community) versions of the engines may not work like the examples shown below. Please refer to the official documentation for that.
Integrating Unity
Here’s an example of how to integrate Unity in a Jenkins pipeline:
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
sh "${UNITY_PATH}/Unity -quit -batchmode -projectPath=${PROJECT_PATH} -executeMethod BuildPipeline.BuildAndroid"
}
}
}
}
}This pipeline builds an Android game using the Unity command-line tool and archives the game artifacts for further use.
With GitLab CI:
image: unityci/editor:2020.3.14f1-linux-0
variables:
UNITY_LICENSE: "true"
stages:
- build
build:
stage: build
script:
- unity-buildThis pipeline builds a specific game (which has been declared in a previous step), and uses a specific container image that Unity officially provides for these tasks.
Integrating Unreal Engine
Here’s an example of how to integrate Unreal Engine (these examples are based on Unreal Engine 4) in a GitLab CI pipeline:
build:
stage: build
script:
- ${UNREAL_ENGINE_ROOT}/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=${PROJECT_ROOT} -noP4 -clientconfig=Development -serverconfig=Development -nocompileeditor -stage -cook -pak -archive -archivedirectory=${ARTIFACTS_DIR}
artifacts:
paths:
- ${ARTIFACTS_DIR}Unreal also provides special container images for ease of building and compiling, and can be used as follows:
image: unrealengine:latest
stages:
- build
build:
stage: build
script:
- ue4 version
- ue4 update
- ue4 setroot /path/to/UnrealEngine
- ue4 makefile
- ue4 buildThis pipeline builds, cooks (that is, compiles the assets into the main artifacts), and packages a game using the RunUAT.batscript provided by Unreal Engine and archives the artifacts for further use.
Deploying to DevKits
When it comes to deploying the games that are developed by the companies, the DEV, QA and PROD environments usually end up being Devkit consoles, which are significantly different from regular PCs. Console devkits are specialized hardware units provided by console manufacturers such as Sony, Microsoft, and Nintendo. These devkits allow game developers to create and test games on the actual hardware platform that the game will be played on rather than on a generic PC or other hardware. They are essential tools for game developers to ensure their games are optimized for the console hardware and perform as intended.
Integrating and automating deployments to console Devkits
After building a game, the next step is to deploy it to the target platform. The process of deploying to popular devkits such as PlayStation, Xbox, and Nintendo involves a few steps, which can be automated as part of the CI/CD pipeline.
To accomplish this, DevOps engineers need to first connect the devkit to their network. As these machines are physical special consoles provided by the platform owners, this part of the development process is generally done on-premise. This can be done using a USB or Ethernet cable, depending on the platform. Once connected, they need to install the appropriate SDK (provided on the official site for each vendor) and toolchains for it.
After installing the prerequisites, the next step is to create a build that targets the devkit. This involves setting the appropriate compiler flags and build configurations to generate the package in the correct output format for the target platform. Once the build is complete, the next step is to transfer the game files to the devkit. This can be done using a file transfer protocol such as FTP or SCP. The game files need to be transferred to the correct location on the devkit, and any dependencies, such as libraries or assets also need to be transferred.
To automate this process as part of the CI/CD pipeline, we can use any CI/CD orchestrator we’d like, but as before, the examples will be focused on Jenkins and GitLab CI. We then can create scripts that handle the deployment process, integrating the SDK command line, from connecting to the devkit to transferring the game files and launching the game.
Deployment examples of SDK integrations for the most common consoles
To integrate the different SDKs with our CI/CD toolchain of choice, we need to follow these steps:
- First, we have to install the corresponding SDKs in our orchestrator’s worker machines.
- Then we need to package our game using the corresponding SDK, creating a target output that can be deployed to our devkit console.
Each console has its own formats and can easily be found in its documentation. - Use the vendor’s Portal API (Nintendo Developer Portal, Xbox Live API, PlayStation Network (PSN) or PlayStation Developer Network (PDN) APIs) to upload the package to their stores or to private servers.
- Finally, we use our orchestrators to automate the deployment of the package to a physical Nintendo console, using the corresponding command line tools for each SDK, as instructed by their official docs.
PlayStation SDK
Jenkins example:
stage('Deploy to PlayStation') {
steps {
script {
def buildNumber = env.BUILD_NUMBER
def buildPath = "${env.WORKSPACE}/builds/${buildNumber}/game.pkg"
sh "/usr/local/ps3dev/bin/npdrm/package_finalize --contentid=UP0001-XXXXXX_00-GAME000000000000 --pkg-dir=${buildPath}"
sh "/usr/local/ps3dev/bin/npdrm/package_sign_npdrm --contentid=UP0001-XXXXXX_00-GAME000000000000 --self=${buildPath}/USRDIR/EBOOT.BIN --pkg=${buildPath}"
sh "/usr/local/ps3dev/bin/ps3load 192.168.0.10 ${buildPath}/game.pkg"
}
}
}GitLab CI example:
deploy_playstation:
stage: deploy
script:
- buildNumber=$CI_PIPELINE_ID
- buildPath="${CI_PROJECT_DIR}/builds/${buildNumber}/game.pkg"
- /usr/local/ps3dev/bin/npdrm/package_finalize --contentid=UP0001-XXXXXX_00-GAME000000000000 --pkg-dir=${buildPath}
- /usr/local/ps3dev/bin/npdrm/package_sign_npdrm --contentid=UP0001-XXXXXX_00-GAME000000000000 --self=${buildPath}/USRDIR/EBOOT.BIN --pkg=${buildPath}
- /usr/local/ps3dev/bin/ps3load 192.168.0.10 ${buildPath}/game.pkgNote: To get official documentation from Sony, you’ll need to get a dev license from the Sony PlayStation program, which involves payments and signed NDAs. If you work on a project that involves the PlayStation SDK, the development team that you’ll work with will need to provide you with documentation that instructs which libraries and commands they use to build their artifacts. There are also many wikis devoted to PlayStation development, but are not backed or supported by Sony. Commands in these PlayStation examples are dated representations for older platforms that are no longer supported.
Xbox SDK
Jenkins example:
stage('Deploy to Xbox') {
steps {
script {
def buildNumber = env.BUILD_NUMBER
def buildPath = "${env.WORKSPACE}/builds/${buildNumber}/game.xex"
sh "/usr/local/xbox/bin/xextool -x ${buildPath} ${buildPath}.bin"
sh "/usr/local/xbox/bin/xex2bin ${buildPath}.bin ${buildPath}"
sh "/usr/local/xbox/bin/xblcli.exe push ${buildPath} --target 192.168.0.10 --remotePath \\Device\\Harddisk0\\Partition1\\game.xex"
}
}
}Gitlab CI example:
deploy_xbox:
stage: deploy
script:
- buildNumber=$CI_PIPELINE_ID
- buildPath="${CI_PROJECT_DIR}/builds/${buildNumber}/game.xex"
- /usr/local/xbox/bin/xextool -x ${buildPath} ${buildPath}.bin
- /usr/local/xbox/bin/xex2bin ${buildPath}.bin ${buildPath}
- /usr/local/xbox/bin/xblcli.exe push ${buildPath} --target 192.168.0.10 --remotePath \\Device\\Harddisk0\\Partition1\\game.xexNintendo SDK
Nintendo’s example may seem simpler, as they provide a set of pre-made and customizable scripts for these tasks.
Jenkins example:
pipeline {
agent any
stages {
stage('Build') {
steps {
// Run Nintendo Switch SDK build command
sh './NintendoSDK/build.sh'
}
}
stage('Deploy') {
steps {
// Use Nintendo Switch SDK deploy command to push build to devkit
sh './NintendoSDK/deploy.sh'
}
}
}
}Gitlab CI:
build:
stage: build
script:
# Run Nintendo Switch SDK build command
- ./NintendoSDK/build.sh
deploy:
stage: deploy
script:
# Use Nintendo Switch SDK deploy command to push build to devkit
- ./NintendoSDK/deploy.sh
environment:
name: production
url: https://example.com
only:
- masterConclusions
DevOps practices in the gaming industry require a specialized set of skills and knowledge. It is important to understand the most widely used game engines and their integration with popular CI/CD tools. Also, while Git is a popular choice for version control in most industries, Perforce is often preferred in the gaming industry due to its ability to handle large binary files efficiently.
DevOps engineers looking to enter the gaming industry should also be familiar with the specific challenges and requirements that are unique to game development. From handling large binary files to managing complex game engine builds, there are many factors to consider.
Overall, DevOps practices in the gaming industry are constantly evolving, and staying up-to-date with the latest trends and best practices is crucial for success. By leveraging the right tools and technologies and collaborating closely with development teams, DevOps engineers can help ensure that game releases are smooth, stable, and successful.
