Grid Online — MMO Architecture

Grid Online will be a massive multiplayer game and needs some solid networking code for the backend and world synchronisation. We’re using Unity3d as our game engine which contains a form of networking & multiplayer code. Trying to build MMO architecture out of Unity’s existing networking code is difficult (if not impossible), especially on projects requiring significant size and scale. I’ve developed our own protocol layer on top existing tools available to us, opening up some interesting possibilities.

Communication layer

I came quick to realisation that Unity isn’t build for what we need. We require a multi-scene approach where we can have as many scenes running as we want at the same time. I’ve tried various approach where I’d tried to run multiple scenes scattered, outside of the range of the camera, across a single scene. Problem is that it doesn’t really scale well and I don’t think that Unity is set up to handle that many players at once.

This called for a different approach and after many hours of thinkering I’ve come up with the following approach on how we will implement our Massive Multiplayer Tech. The networking architecture will exists out of three major components:

• Master Server
• Slave Server
• Game Client

Slave Server

The Slave Server represents an area within our world and takes care of local world (area) synchronisation and handles the players currently in the area. This part of the networking code is using Unity’s networking code and could, as far as I can tell right now, handle all of the player and object synchronisation. If at some point it turns out that Unity’s networking code is lacking I’ll write my own networking synchronisation code and use Unity just as render engine. The Slave Server is in constant communication with the Master Server and makes sure that it can pass on authorisation to another Slave if desired (e.g. when a player moves to another area).

Master Server

The master server is responsible for managing player authorisation and slave management. As each slave represents a local area it must know when to create new areas and when to move players over to another slave. The master server has the ability to create new slaves on the fly when requested. For example if player A moves to Area X, but X is not yet running it can spawn a new instance, wait for the slave to become available and notify the player of the new slave.

Using this approach of Master and Slave it’s relatively easy to implement load balancing. If a slave becomes too populated the Master server could easily decide to offload some players (or players as they move to the crowded area) to a new slave instance with the same scene. It could even decide which player should move to the crowded and which shouldn’t. For example players within the same team should always be on the same slave instance and not be divided among several instances; or players whom wish to trade with one another, these players should always be moved to the same slave instance. Perhaps we’ll even let players force themselves to a certain instance if needed. I think there are more games that use a similar approach to this kind of load balancing.

Game Client

This is the client which the players will use to interact with the game. It will consist of the render-engine using Unity3d and the networking core components for its communication. The Game Client will also allow the player to authenticate, create new characters and join the game using one of their existing characters. Under the hood it will be using the the Grid Networking Protocol to communicate with the Master and the Slave servers.

Grid Networking Protocol (GNP)

I’ve developed our own network protocol on top of Unity’s NetworkTransport for the communication between the three componets. It handles a handshake procedure where peers first acknowledge each other and make sure they speak the same networking language (or dialect); e.g. network version numbers. Once both peers have acknowledged eachother they are ready to start exchanging messages. Usually this starts with an authentication procedure. Once a game client connects to the master server it will verify the entered credentials, which are passed by the user. The player’s client will retrieve the character list from the master server and can select one of the characters to spawn at the last known location. All of this messaging happens in the background is handled by the Grid Networking Protocol.

GNP is message based and naturally asynchronous. It’s created in such a way that it can easily be used by other processes. The system is loosely coupled, meaning you can create new communication channels from one part of the system and listen for events from another part of the system. For example think of a market, where players can put up items for sale and other players can place bids or buy straight out. A process like that requires some central processing & authoritative and communication between the authoritative and client. The Master server can function as the central authoritative and the client could initiate a trade, the communication will naturally happen over the GNP.

Sharing is caring

The networking protocol we’re developing for Grid Online is developed as a stand-alone package and can easily be integrated in to other projects. It’s possible that in the future we’ll release it to the public but we’re not quite sure yet. Personally I’m a big supporter of Open Source Software and I’ve developed my fair share of OSS in the past, but once out there we’ll have to be willing to carry the responsibility and we’ll have to handle the issues and requests. This is something that could easily turn in to a day job and at present I (we) have got our hands full developing a game. In any case the decision will be a well weighted one.