After a few weeks of testing, I’m confident that Crowfall is the game I have been waiting for. Planning for and analysis of the long-term has commenced. Based on the trajectory and what I have observed of the team, there are only a few lingering concerns for me.
The Speed of Light and Campaign Worlds
I tend to play video games with a small group of people, and we’re scattered across the world. Speed of light becomes a real issue, as it can dictate the difference between a 10ms and 300ms latency, which is the difference between supremely playable and unplayable.
Currently, the Campaign Worlds have different rule sets and maps, and their location cycles between US West, US East, Australia, and EU.
My kiwi friend gets great latency on the Australian server, but it’s unplayable for me from US West. EU is unplayable for both of us, and US East is unplayable for him. That means that US West is the only existing location that remotely works for both of us.
When you look at a map, this all makes sense based on geography alone. When you start to consider how fiber has been laid between different locations, it makes even more. Australia has historically had very weak internet connectivity, and the only major population center between New Zealand and the Mainland United States is Hawaii.
What this means is that when you consider the campaign world diagram that shows rule sets and correlation to the center of the world, it needs to be made three dimensional, with server location as the third dimension. The great thing about this design is that for server locations without a significant population, a scaled down set of campaign worlds can be offered; they don’t have to be equal across locations.
Eternal Kingdoms
The same challenges apply to Eternal Kingdoms, but we have significantly more flexibility in tackling them because a single player or group of players has the ability to indicate their own preferences.
Currently, Eternal Kingdoms spin up new servers of some type in US East, but eventually Crowfall will need to support Eternal Kingdoms in additional locations.
There are two primary ideas for how to handle this:
- Automatically determine the lowest latency server location to the person spinning up the EK, similar to Path of Exile
- Allow the owner and anyone given privileges to choose the server location
I would opt for the latter because it gives us more flexibility to accommodate geographically diverse groups and should require less engineering effort.
A use-case where this could be relevant is living in the middle of the United States: an automatic determination could put the server on US East, which would not be playable for kiwis.
Performance and Segmentation
I don’t have a lot of information on this topic currently, but it’s a common issue software development. Scaling servers can be hard, and I expect it’s significantly more difficult when making a video game.
We can draw a few inferences, however. Given that a server is spun up for each Eternal Kingdom, we can presume that the Eternal Kingdoms are virtual servers or containers. These could theoretically be scaled to larger hardware but there will be a cap and it gets progressively more expensive.
The interesting question is at what point clusters can be used. Is a server or container being spun up for each parcel? The Update from The AI Guy post alluded to something like this, but it’s not clear whether the processing for either side of the seam he mentions is done by separate servers, processes, or threads. This is an important distinction and can potentially inhibit scalability.
EpicData and Cloudflare
When I was digging into server locations I noticed that Crowfall uses at least some features of EpicData’s PaaS. This is really awesome because it means that as long as EpicData continues to update and improve, we’ll get all of those benefits.
But it also carries some risk. At a glance, EpicData seems to be a small company, and it’s entirely possible that it could fail. I would encourage early and strong communication with the EpicData team with the insinuation that they could be acqui-hired if things go south. Losing this piece now would likely be very burdensome. However, it’s entirely unclear to me how many features are actually driven by EpicData.
Cloudflare carries less risk because it can easily be swapped out at any point, and it’s sort of a conundrum — you want to defend yourself against DoS but too many gates means higher latency and worse routing for customers. It’s possible that Cloudflare is currently only being used for login servers, and not the actual game traffic, but without some mitigation technique in place, any of the servers will be vulnerable to DoS. I don’t have a whole lot of experience in this niche, but I’m happy to share what I know and resources I have.
Conclusion
While there are some major open questions in my mind, I’m thrilled with where Crowfall is at, and how responsive the team has been. All of my support requests have been responded to promptly and effectively, and while strong-headed, the team is clearly open to feedback. Huge shoutout to vkromas for committing to improving the flow of communication between testers and engineers! This is the kind of stuff that encourages customer loyalty and fanboys.
Followup
A discussion in the Unofficial Crowfall Discord led me to a few pieces of information:
“If smaller is better, why not 10m by 10m?
Now, don’t get me wrong, smaller cells are not simply better than large. This is a tough balancing act — reducing cell size has negative performance implications (e.g. the game must load them separately and stitch them together, causing more intersection points as the player runs around the world and more blending of textures to hide the seams)”
https://www.crowfall.com/en/news/cell-size-change/
Additional Reading
Crafting Questions: Armor and Additives
Capture the Flag: July 9th, 2017
