Write, Compile, Distribute, Test; The Sisyphean State Of Multiplayer Game Development
I’ve spent the past four years working on multiplayer games and applications, and in that time I’ve learned that the multiplayer facet of a software project ends up consuming a disproportionately large amount of time to develop, debug, and maintain. I’d like to tell you about four main reasons for this additional complexity, and what is being done in modern tooling to address these issues.
1. Long development loops
Let’s say you want to change some multiplayer aspect of your game. To fix it, you have to write new code for the feature, compile it into an executable, send it to your tester(s) and finally verify that that the feature works as you intended it to.
The time that this process takes is considerable, and as a result most multiplayer games are simple in scope compared to their single player counterparts.
There was a time when hardware limitations necessitated writing code as close to the hardware as possible, but in a modern device runtime-interpreted languages are more than sufficient to meet framerate when used for client logic.
Some facets of a game, like iterating through the scene graph or running animations, still need to be run with a performance friendly language like C++ or C# with burst, but these hot paths generally make up a minority of the codebase. The majority of the codebase is defining how the state of the application changes in time. Most paths are run infrequently, and many of those paths can be run over multiple frames if necessary.
A few games allow creators to write code in a multiplayer environment using an interpreted language. Google’s GameBuilder uses Javascript, Frooxius’s NeosVR uses a node based language, and my own project Gamelodge uses a Python-like language called Miniscript. With these, game development is no longer a segmented, disjointed experience. You write code while the game is running, fluidly transitioning from building to testing and back. You can potentially do this concurrently with other people working on the same project. This vastly improves the rapidity of iterations, and empowers people to be more ambitious with the games that they create.
2. Networks are Either Lossy, or Latent
Over a network, data is almost always sent as either TCP or UDP. The former ensures that data always reaches its intended destination, but that data takes an indeterminate amount of time to get there. This manifests in networking that sometimes feels responsive, but can become laggy over time and under poor network conditions.
UDP, on the other hand, has packets of data arrive to the destination as quickly as possible, but it silently loses packets of data at random. Over the internet on an average network, around 1% of packets can be expected to be lost. Without proper loss handling, this can result in a divergence of the simulation, where different users see different things.
An example of a type of data that’s best handled via UDP would be user audio. Humans can comprehend audio with a 1% loss rate with relative ease, whereas a latency of even a few seconds can make conversation intractable.
Good games, in general, use a hybrid of the two in order to play to the strengths of both. Persistent changes are handled over TCP whereas changes that are only valid for a few frames are handled via UDP.
A relative newcomer to the field which bears mention is SRT. Released to the public as an open source project, SRT affords the developer with a balance between latency and reliability. Connections have a constant configurable latency, and packets that are loss are retransmitted as long as the latency deadline has not yet expired. This system is ideal for live video, where users expect both low latency and video without corruptions.
3. Replicating bugs is difficult
Imagine users are playing your game, and something breaks and the game becomes unplayable. How do you find out what broke to fix it?
The sorry state of current affairs is that users describe as best as they can what happened and — if they’re particularly helpful — they may even send you the logs of the game when it broke. Then the developer has to read these logs, build a mental model of the game state, and make an educated guess as to what went wrong.
Then the developer has to verify that their proposed change does indeed address the issue. But if the issue was due to network loss, a networked race condition, or a hacked client sending invalid or unexpected data across the network, actually reproducing the issue becomes intensely challenging.
One way to solve this is to use a reversible language like Beads. With Beads, you can step both forwards and backwards through the execution of the program. This way users could simply send you the state of their game client, and you could step backwards in time to see when the issue presented. The downside is that writing code in this way is very different than most languages, and programming common game logic may be more challenging than a linear language.
Another technique to address this, pioneered by Age of Empires and implemented in Gamelodge, is to allow the end user to record the entire networked game state and it’s evolution over time. With this, users make a recording of their game that includes the issue, and sends it to the developer. The developer can then replay the recording and see exactly what the user was seeing, as well as the networked state. They can even replay the recording with edited scripts, so that they can quickly verify that a change fixes the issue.
Admittedly, this is far from a panacea. Some kinds of bugs can’t be reliably reproduced in this way. For example, most game physics implementations are nondeterministic, so a recording might not reliably replicate the behavior experienced by a user.
4. Ancient APIs and Inefficient Abstractions
If you want to send data over the network on Windows with C/C++, you’ll have to use winsock2, an obtuse API provided by Microsoft that has changed negligibly since it’s release in the late 90s. Working with sockets in this way is verbose, dangerous, and it’s easy for even an experienced developer to run into undocumented silent bugs that can cause unnecessary packet loss.
Fortunately, a number of packages exist to abstract away the process of synchronizing state across clients. The most prevalent among them, Photon, provides a system that makes creating networked objects and users trivial.
However, this development ease comes at the expense of bit-inefficient network messages. If you want to synchronize the position and rotation of an object in three dimensions, you’d need three floats for position and three floats for the rotation. If sent directly in the simplest way possible, this would be 24 bytes on the network. However, due to the overhead of Photon’s type-safe serialization this would actually be a 30 byte message, giving the user an unnecessary 25% increase in network traffic over the naive implementation.
Additionally, due to the way that Photon packages, prioritizes, and routes messages, the actual information throughput is substantially less than that facilitated by most networks.
The result of this is that most multiplayer games that use these kinds of packages to ease development end up severely limiting the scope of what is networked. This manifests in games with few synchronized objects and artificial limits on the number of concurrent users.
Games with a well optimized networking architecture fall into a common pattern of beginning messages with a fixed size tag specifying the type of message, followed by the actual payload. For example, a message might begin with a single byte signifying that the message contains the position and rotation of an object, two bytes designating which object the message is for, six bytes for the compressed position, and five bytes for the compressed rotation. The entire message, including overhead would only be 14 bytes.
This, along with network prioritization and culling, mean that game creators can make ambitious projects with dozens of users and thousands of synchronized interactable objects.
There are many pitfalls that beset multiplayer game development, but these pitfalls can be navigated by taking advantage of emerging software solutions and implementing those tools efficiently. My hope is that if developers are able to build multiplayer games easily, iterate quickly, and create without limitations, then we will see ambitious titles that redefine multiplayer gaming.
If you want to play multiplayer games made by users, or if you want to build your own games, check out Gamelodge on Kickstarter.
