Photo by israel palacio on Unsplash

Sharing Game Data: A Guide

How to flip the script for better data flow

Here at Mighty Bear, we build online multiplayer games. Unity has remained our game engine of choice and we use Java coupled with SpringBoot as our backend. For every multiplayer project, the question that needs answering is: How will we be sharing game data between the engine and our backend? Unity is managed with C# and Java is, well, Java. The challenge is finding a way to share data across 2 very different environments while ensuring consistency of data, speed of development, and reliability of features.

While our implementation is specific to Unity (C#) and Java, what we are doing here could be applied to any game engine and backend as long as it’s in one of these languages:

• C++
• Java
• Python
• Objective C
• C#
• JavaScript
• Ruby
• Go
• PHP
• Dart

Our Process

The game data first is set up in Unity by our game designers. The usual workflow then dictates that we export this data structured as protocol buffer messages in a git submodule. The backend git repository then uses that git submodule to load the game data, while protocol buffers provide the thread that ties it all together. If that intrigues you, read on. If that was alien speak, also read on — you may wind up fluent yourself…

Some of you might be thinking, “Why are we talking about protocol buffers, a tool typically used for sending data over the wire?” Protocol buffers are really good at one thing and it’s right there in the bio on the official PB site:

Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serialising structured data. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

I am going to break this process down using a concrete example.

Data Structure

In Butter Royale we have Seasons, where players can claim limited-run rewards based on the Season-specific currency they earn in matches. Every Season has a defined end time, at which point a new Season beings. This is how the protocol buffer data structure for this feature is defined:

message Seasons {
   map<uint32, SeasonDefinition> values = 1;
}
message SeasonDefinition {
   google.protobuf.Timestamp end_timestamp = 1;
   CurrencyCode season_currency = 2;
   repeated SeasonPassItem season_pass_items = 3;
   string season_unlock_key = 4;
}
message SeasonPassItem {
   uint32 required_currency_amount = 1;
   Item item = 2;
}

Each season corresponds to a SeasonDefinition , which contains a list of SeasonPassItem. Each SeasonDefinition is then referenced in Seasons and a key representing the Season number is included with thatSeasonDefinition in values. This is all the information that is required to set up seasons in Butter Royale. Using this definition, we can generate the classes for C# and Java. We will come back to the SeasonDefinition later.

Data Setup

We use ScriptableObjects in Unity to set up the data. In this example, we have a SeasonData ScriptableObject that looks like this.

Season Data Setup

Here, the designers can configure the time a Season ends, the applicable currency, and the rewards available. In terms of rewards, they’re afforded the flexibility to define whether the player should earn currency or unlock one of the Season’s skins, flags, trails, passives, or melee weapons. As you can see, this looks very similar to SeasonDefinition but also contains information that may only be relevant in Unity (Currency Sprite, for example).

This SeasonData is then referenced in a prefab holding all of the game’s data called DataImporter :

Data Importer Prefab

Data Export

Once the data is ready, the designers can export it from the same prefab using the Export To Server button.

Export To Server from Data Importer

As part of the export process, theSeasonData is mapped to theSeasonDefinition C# code generated using the protocol buffer definition.

public SeasonDefinition Convert()
{
   var seasonDefinition = new SeasonDefinition
   {
      EndTimestamp = new Timestamp{Seconds = endEpochSecond},
      SeasonCurrency = currencyCode,
      SeasonUnlockKey = SeasonUnlocksKey
   };
   

   seasonDefinition.SeasonPassItems.AddRange(seasonPassItemList.Select(x => x.Convert())); //conversion for SeasonPassItem

   return seasonDefinition;
}

Finally, all of the SeasonDefinition information is consolidated into the Seasons data structure and exported to a file as a base64 encoded string.

private void ExportSeasonsData()
{
   var seasons = GetSeasonExportData();
   ExportToFile(Constants.Server.SEASONS_DATA_PATH, seasons);
}

private static void ExportToFile(string path, IMessage proto)
{
   File.WriteAllText(path, proto.ToByteString().ToBase64());
}

The aboveSEASONS_DATA_PATH points to the git submodule that contains all protocol buffer definitions and exported data. After exporting, the designers can commit and push their changes. The data is now ready to be imported by the server.

Data Import

We then update the Java backend’s git repository to point to the same commit/branch for the git submodule we just exported from Unity. This becomes a reliable way of versioning the data as well: both the Unity and backend repositories should be pointing to the same git submodule commit.

Assuming the protocol buffer definition has already been used to generate the Java classes, all we need to do now is read the file.

@Bean
@Primary
public Acquirables.Seasons seasons(
   @Value("${shared.data.seasons}") String seasonsPath,
   ProtoSerializationUtility protoSerializationUtility
) throws Exception {
   return protoSerializationUtility.FromBase64Resource(
      seasonsPath,
      Acquirables.Seasons.parser()
   );
}

shared.data.seasons is the path to the file we exported in the last step, and that’s it — the data is now ready to be used by the server !

Data Validation

To ensure reliability of data, we make sure it’s validated at multiple steps in the process.

Once you click on the export button, we run some validations on the data. For example, there might be a mismatch on the season identifier setup inside SeasonData and the season identifier key in DataImporter. The validations are meant to catch such errors here and flag them to the designers.

We also have data validation tests to make sure the data setup in Unity matches the data exported. If there is a mismatch, a designer probably forgot to export their changes. We run these same validation tests on the backend code too. Any assumptions about the data are verified through these tests. This ensures that we don’t have to wait for a deployment to identify issues. They get caught with our Continuous Integration pipelines that run with every pull request.

Benefits & Considerations

• By relying on protocol buffers, the engineering team is forced to operate a data-first approach, as opposed to logic or UI.
• A single point of ownership — designers are given full control over the game data.
• No more databases for static data. This is a subjective one as your own use case might still require a database. However, in our experience static data updates are not frequent enough to justify a database. This also simplifies our infrastructure, lowers its cost, and facilitates very quick access to static data. There are no database migrations to go wrong. If static data changes are required, we run a new deployment. Since our deployments are blue-green in nature, players don’t face any downtime.
• Extra protection from errors. Validation tests ensure reliability of data. Since the data is packaged in the container images for the backend, there is no scope for accidental changes to the data once the images are deployed.
• Extensibility. It is very easy to add/update data once you implement this process: I only gave an example of seasons here but all of Butter Royale’s data — over 1.5 years’ worth of content — followed the same process.
• Understanding of git submodules is necessary. This was something that some of the team had to educate themselves on. We have since developed more tools to streamline the submodule process on newer projects.

I hope this was a useful insight into how — and why — we share data between our game engine and backend. I would love to hear your thoughts and any further questions in the comments below!