Importance of System Design & Architecture
By Solaman (Soul) Huq
Why LandShuffle?
We wanted to shake up the meta in our 4X game, “War Dragons,” with an exciting new feature for its world map: LandShuffle. It would seasonally generate a new map and shuffle around all players, castles, and troops into the new map. This huge disruption would encourage players to battle and compete for land.
Maintaining such a feature can create a Quality of Life nightmare for the developers without the right architecture. In this post, we will talk about a few ways we tackled those challenges.
Scale It Up — Saving Map Data to the Cloud
Before LandShuffle, our map data was static and defined in the code. While this was easy to maintain, modifying those static files and uploading them for tests and hotfixes could become time-consuming.
Our solution was to instead maintain our metadata and binaries in S3 and distribute it via our Content Delivery Network (CDN). This massively simplified the LandShuffle release pipeline; Once the data was generated and uploaded to S3, our backend instances and client builds could both immediately start using the data.
Multitenancy — Isolating Map Data
Our initial implementation had almost no tech to deal with multiple maps, as there was one map and multiple tutorial maps that rarely changed. For LandShuffle, though, this posed a unique challenge, as we wanted the ability to safely switch back to a prior version of the map if anything went wrong with the shuffle.
We used Google App Engine (GAE) Datastore’s namespacing to separate data into isolated partitions. This provided us with security by making it impossible to access data across different namespaces. Secondly, by generating new maps in a new namespace, the old data remained entirely untouched. Performing a rollback was as easy as pointing to the old data.
Access Control — Kingdom Access
Isolating data alone wasn’t enough. We needed to sort out how client devices and all of our backend services knew which map to point to. To introduce this logic, we were staring at refactoring 10,000+ lines of code!
To navigate this challenge, we designed a Kingdom Access Control mechanism to facilitate pointing servers to the correct map. During the release process, we would set up a staging server to point to the new map, and once it was verified and tested we’d point our main servers to the new map. We integrated this control mechanism into our model classes to automatically modify the datastore key to access the correct map’s data. These strategies combined helped us achieve our goals with minimal refactoring. All of our existing APIs were now always pointing to the right data. We also built web tools to generate diffs between different servers.
The Outcome
All in all, the architectural decisions we made ended up proving extremely beneficial. We could play-test the entire process repeatedly in production without affecting real players.
These changes also allowed us to reduce our overall release process from 20 hours to a 2-hour process that required no engineering support.
