Netcode Series Part 5: Infrastructure (from Matchmaking to Play)

One of many custom dashboards to tell us how each region is doing.

Getting clients to play on a server quickly and effectively is a big feat. That’s certainly what, as a game developer, you should be focusing on.

In an increasingly social gaming sphere, however, you’re going to need infrastructure and services to support scaling game servers, account management, matchmaking, party systems, etc.

There’s tons of services out there you can use out of the box to support your efforts. Microsoft, Amazon, Unity, Epic all are building software to service game developers with their needs. Mainly, of course, to attract developers with the promise of an easy and fast integration, and an ultimate ecosystem tie-down. If you’re new to services, I recommend starting there unless you have highly specialized needs.

My background comes from creating game services from scratch and running them on bare metal and the cloud, so at Super Bit Machine, we tailored our solution to the needs of our first game, ARMAJET.

Services and orchestration

List of active pods in a specific environment, autoscaling to our needs.

Our services are APIs that are developed, deployed, managed, and virtually separate from one another, with no cross-dependencies. While that’s mostly true, the game still ultimately needs all client-serving applications to function.

There are plenty of articles on the web arguing in favor of microservices or monoliths. We have a mix of the two to take advantage of fast iteration on new projects, while not having to split up our main legacy backend.

• Platform: serves all game data and authentication for players. This is the first point of contact for the game and the largest service. It’s the only one allowed to make permanent changes to account status, progression, etc.
• Matcher: accepts requests from players who want to start playing, and runs queries against the requested criteria. When a match is established, it requests a new server instance.
• Portal: entry-point for all servers requested. Manages scaling regions and routes requests from clients who want to play, as well as Matcher.
• Social: it provides functionality to enable real-time push messaging, party grouping for players, chat, etc.

All services are built by our Jenkins CI pipeline, spit out a Docker image which is then uploaded to the corresponding environment and deployed via Kubernetes. This goes from local bare metal for internal development environments, all the way to production cloud (AWS, Rackspace, etc.).

As an application developer, this is all transparent. Once a commit to a specific repository branch and tag, the app gets built and deployed to the target environment in anywhere between 1 to 5 minutes depending on the size of the service. The rollout restart deployment Kubernetes command is configured to launch the new versions and shut down old ones without any service interruptions.

Kubernetes has resource allotments configured on a service basis, so as it horizontally scales depending on traffic, we’re keeping our costs to a minimum.

Platform

Our platform service owns all the data and is the only service that has read/write properties on it. If other services need to access player data, this is their way in.

This is a classic REST API, with connection to relational (SQL-like) and Redis type datastores. Safe, reliable, not reinventing the wheel here.

Platform serves:

• Player authentication, via Device ID, Platform ID (Steam/GameCenter/GooglePlay/etc.) and email (Superbit ID)
• Player data retrieval (inventory, stats)
• Game data retrieval (list of items, their stats, etc.)
• Game’s meta progression
• Game Server’s endpoints for player auth and score submission

Matcher

Our Matchmaking service connects players together, runs calculations and groups players together resolving the final rules. Once the matchmaking session is deemed completed, Matcher requests a Server instance from Portal, which returns a qualified server ready to accept players.

Matcher is an API made for polling. It allows websocket push connections, but for most cases, polling it to receive updates is sufficiently fast even for a mobile in-and-out game.

It connects and queries a Mongo datastore to manage active matchmaking tickets and find the best matches.

It takes into account:

• Party size
• Criteria selected (game modes, number of players, etc.)
• Device type (PC, Mobile)
• Input system
• Region Latencies (pings)
• Skill Rating
• Win/loss history

All these variables are used to do multi-dimensional searches to find the best match, and autotune based on how many players are online.

There will be another article going in depth about our matchmaking and how it works at scale.

Portal

It’s the entry point into the Game Server. Portal accepts requests for new servers from Clients and Matcher directly. It launches new servers, manages all active server instances, and routes players.

After launching the server, Portal returns a host so the clients can connect over UDP via IPv4 or IPv6.

Like Matcher, this is also a lightweight API, and it’s the only one that actively manages multi-region game servers.

Portal receives heartbeats from all active servers, alongwith the active players, rules and state of the games. As long as a game server has a region code associated with and is deployed within our security group, it’s recognized as a new potential destination and registered in an active server region.

I’m pretty proud of this one because we can decide to whip up a new supported region and deploy it on AWS, and all client and services will now be open to the new destination automatically. We’ve done that a few times with our Indian/Australian/Korean regions as we were testing quality of play and comparing latencies from our main APAC servers in Singapore.

The full connection flow looks like this:

From the moment they click play, to when they actually play.

Social

This isn’t an essential service, but it enables partying up as well as other components that enable group experiences and social graph. These are:

• Party system
• Chat
• Social Real-time messaging system (friending, etc.)

Social is an API with Redis Pub/Sub as backbone. It only manages transient data and is responsible for all real-time connectivity outside of the game server itself. Websockets are used here, with classic polling fallback.

Today we looked at most of the services that are required to bring a player from a game homescreen to a game session. You can build all of this as microservices or a monolithic backend, you can use off the shelf software or plug into the many SaaS on the market today.

On the next article, we’ll dive into the nitty gritty of our cross-platform matchmaking system and how it adjusts to any amount of CCUs to provide the best experience.