It’s been half a year since I last updated in Feburary. At the time it felt like I wasn’t making much progress, however looking back I’ve got a lot to share!
The general theme for these months is making the game core and networking robust and evolving the game’s direction. The Sector and Factory management in code is much better. There is a good networking foundation with room to improve. I’ve come up with a new core gameplay idea around Ship Sections. For more… read on!
Action (Sector) vs Lock-Step (Factory) — A Primer
In The Recall Singularity you’ll be flying small factory ships around in space. In space you’re blasting asteroids, fighting off hostile vessels and clipping salvaged ship chunks on to your own vessel. Each region of space which contains vessels, asteroids and battles is called a “sector.” Each ship contains a “factory” (at least in code).
The frenetic action in space is close to a FPS shooter or action game. The clockwork interaction within parts of your ship is instead more like a regular Factory or RTS game. When it comes to networking, these two game types are solved using different approaches.
Action games like the space battles are well suited to a “server authoritative” approach like Quake. The server calculates what happens and tells the clients what to show. To hide the latency (or ping) the client has some client-side prediction. The client can get out of sync or only have partial knowledge of the simulation.
Factory and RTS games are better suited to the clock-like precision of a “Lock-step” approach like Age of Empires. Each player shares actions which change the simulation and the deterministic simulation is run anew on each computer, hopefully remaining in sync. The Client needs to know every detail of the simulation to stay in sync.
I can’t offer the ship physics and fast paced game-play I would like using the Lock-step approach. I also cannot keep many players in sync with a complex factory simulation using the server authoritative approach since there is too much data to send.
So in The Recall Singularity we use both approaches. Each ship (“factory”) is a stand-alone Lock-Step simulation which is synced to a client only if they are in range. Sectors tie everything together with an action based simulation. As well as maintaining both simulations, the server needs to tie them together as ships scoop up rocks, shoot projectiles and take hits.
Specialized Factory vs Sector approaches
In February, the code-base used to have a “Sector” which held nested SyncWorld, LockWorld and specs::World classes in a HashMap and asked each one to send whatever messages it wanted to its matching class in the client. While this kind of worked, it was degenerating into Spaghetti and I was constantly having difficulty remembering where responsibility lied or how specific synchronization was supposed to happen.
Both factory and space simulations were using something like lockstep, though they were not staying in sync. Each individual Factory simulations could get out of sync between client and server also. It was a mess.
Now I’ve moved to SectorClient & SectorServer classes which fit together. There are also FactoryClient & FactoryServer. The SectorServer class maintains some nested FactoryServers along with a specs::World which represents its own space. The server also has data-structures to manage client connections.
While the SectorServer has a full suite of Systems to advance the simulation, the SectorClient currently only has enough to copy a snapshot for presentation to the user. I’ll be iterating on this as I add Client-side Prediction.
Both FactoryServer and FactoryClient have a matching set of systems which advance the simulation in the same way. As discussed below, the client follows the server and each frame it applies the same Actions to the same starting state to calculate a matching next tick.
I’ve also created specs::Systems which copy relevant information from the Factory worlds into the space worlds and others which create Actions to inform the Factory about what is happening in space nearby (like ore recently scooped up). These run on the Server and are sent to the client each frame.
Creating all of these classes and locking down their relationships has given me confidence I have a robust base to build on.
Different FPS for Factory vs Action
Each second is split into individual fixed frames on the server to advance the game’s simulation. For now the Action (Sector) game is updated 60 times per second and the Factory at 20.
Like with LockStep vs Server-Authorative above, the two simulations have different needs. we want the action in space to be smooth and responsive, with somewhat twitch game-play. Inside the factory, things are more sedate. The goal there is to allow large factory ships, where higher frame-rates will just amplify any performance issues.
The core difference high frame rates give us in the action game is a more accurate physics simulation and a shorter latency between a player’s action and the next frame. With 60 FPS, the max time between processing a button press in a network packet and applying it to the simulation is just 16ms. I hope this makes The Recall Singularity seem smooth and responsive.
Lockstep vs Action Networking Basics
The basic versions of both the Lock-step and Action syncing are now in place for Factories and Sectors.
In in the Action Space system each frame (60 times a second) the server processes each clients’ input action for that frame and sends a snapshot of the world to each client. The only input considered here is thumbpads, buttons and whatnot. All the client is sending back for this is those raw input controls and info on which snapshots made it through to the client.
While all communications are going over UDP, the messages relating to the Action game are extra-unreliable. If we lose any, we never retry, we just move on to the next frame. So I need to ensure missed snapshots just mean a minor glitch on the client’s screen. To improve reliability, the client doesn’t just send this frame’s action but actions for each of the past 16 frames. So if a packet is lost hopefully the server can scoop up the missing data in a later packet.
Factory synchronization is split into two main phases. The first is where the server sends a snapshot of the relevant factory’s current layout. This is followed by each frame’s actions (20 times per second). Actions for the factory are gathered from the clients, applied by the server in a later tick and then distributed back to the clients. Once the client gets the relevant actions for tick n+1 they can apply them to their current state n and move to n+1 in sync with the server.
To simplify the sending of Factory actions to the server, I’ve attached them to the same input message used in the action game. That is, 60 times a second you send your button inputs to fly/walk in space and you can also send one action to a factory. Those queued actions might place a module, extend the floor or similar. There is an outgoing queue of actions on the client and you’ll find your actions returning to you in a later lockstep sync message.
Unlike the action snapshots, the factory tick messages from the server must make it through. Inside the UDP communications I’m retrying these messages until they make it through from the server to the client.
Simulation times in networked games like TRS are extra-funky to understand. To explain these, imagine that your Ping is 200ms, so it takes 100ms (latency) for a message from the server to reach you and 100ms for your response to go back.
The Lockstep factories are easy enough — the server applies a frame and 100ms later you find out and apply this change yourself. So the server is always 100ms ahead of the client. If you send an action to the server, it will not apply it for 100ms (during the packet’s travel time) and so you’ll see your change affect the world about 200ms from now. These actions are not labelled for a specific frame, the server will drop them in as soon as it can. Given the 20FPS simulation (one tick every 50 ms) it could take up to 250ms to see your changes applied by the server.
The Action sync is a bit more complicated because we dont’t want the client to notice that 200ms Ping. By the time the client recieves a snapshot, that snapshot is already 100ms old. Any action the client sends the server is going to need to arrive over 200ms ahead of the state shown in that snapshot. So on the client we need to “extrapolate” the snapshot by >200ms, send that action to the server and have it reach the server before the relevant tick.
Server Actual = T
Server state on client = T -100ms (stale)
Prediction on Client = T + 150ms (250 ms ahead of known state)
Client’s actions reach Server = T + 100ms
I’m looking forward to solving this mind-bending problem further as the networking evolves!
Asteroids in-game, Shooting, debris, rocks
It was about here that I started to get anxious that I was making a tech demo (now with networking!) rather than actually working on the core of my game-play. If you’ve been following me for a while, you might have noticed that this is a recurring theme.
I wanted to get a core game-play loop in game:
- Blast up asteroids
- Eat the goodies you get into your ship
- Refine those
- Make your ship bigger with better guns & engines
- Repeat
So I added those asteroids you could mine as actual entities in space. Put in bullets and set up scoops in your ship which would collect any nearby rocks. Everything was going great until I hit an (expected) brick wall of synchronization issues.
Remember how the Action game in Space is now server authoritative and sending out snapshots each tick?
With my simple use of (rust libraries) Serde and Bincode, along with a snapshot for every entity in space … it didn’t take a very complex scene before my snapshots would no longer fit in an outgoing packet. Most entities took up 100 bytes (!) serialized. I could make these entities a LOT smaller but have not done so yet.
So the plan to focus on game-play ground to a halt pretty quickly.
I discovered how I wanted to manage these snapshots and started working on making them more manageable .. then realized I was moving away from game-play programming again. So I shelved that and moved to …
Standalone Ships
Remember how I run a factory simulation for the internals of each Ship?
I realised that I wouldn’t need to worry about a full networking stack if I just set up a FactoryServer and a FactoryClient in a nice little Standalone wrapper struct (which keeps them in sync) and I’d have a great little testbed for rapid iteration and development.
Once that was working, I was all ready to start expanding factory features. To celebrate this I added a GUI which displayed the contents of scoops, crates and more within ships. Then I started considering more complex factory features like larger modules and different power management subsystems.
It didn’t take me long to come up with my next great idea.
Ship Sections
Fans of Factorio know that the world stretches without limit in every direction. Sure, there might be trees, rocks, cliffs or water — but generally the factory can grow on a blank canvas.
To emulate that in space I initially imagined it would be nice to click and drag a floor area for your factory spaceship. That way the pesky spaceship building part would be as easy as building on clear ground is in other games.
But there is a hitch in this — a factory without constraints is going to end up being boring and constant (between ships and game). You won’t need to adapt it to the environment. The ideal spaceship is always going to be a large rectangle. I realized that I was heading down the path of repetitive, cookie-cutter game-play.
Enter Ship Sections. Initially you cannot build these, instead you must salvage them whole from the wreckage around you. Each ship is made from 5+ sections all clipped together. Some are filled with engines, consoles or weapons. Others have empty space for you to build a factory within. They come in a variety of states of disrepair and can be upgraded as you progress.
Since each section has an interesting space to build in, hard points for guns and key structures … they create a basic framework you need to build around. You won’t be finding the same sections repetitively, so I hope you will always be adjusting your designs to fit.
Prototyping meshes and determining pipeline
To create the sections, I’m leaning on the approach used to create Dungeons in ARPG games like Path of Exile. The dungeon generation in PoE connects rooms which are made from Tiles. Those tiles each have several 3D meshes and data about where you can walk (and in my case, build).
Pictured in this article are the outcome of some time spent mocking up suitable 3D assets and modules. It’s important that when the same mesh is used in many tiles it remains “instanced” in Godot (rather than defined anew) as that improves performance. I’m still iterating on this process but I plan to automate some of it using Blender scripts.
So having done a bunch of research and experiments I’ve figured out how I want to generate these sections. Next up is adding them to the game. To do that, I needed to first fix the networking…
Overload of networking system, partial updates and differential syncing
So, back to that networking problem…
We’ve only got 1,000 bytes or so per frame to tell a player about relevant entities. We need to filter them — the close ones, which have changed only. Then we need to send only what has changed. We also need to handle snapshots that got lost, since we won’t be resending them.
I tried to load all the complexities into my head of the following systems, it was all a bit much:
- Which entities are we going to send (those nearby)
- How do we map those to “local-ids” (so we can use small 8-bit numbers)
- For snapshots, how do we track what the client knows and how old their data is?
- Delta encoding and compression of data in flight?
- The continuous feeling that I missed something
I realized I needed to tackle things one at a time, in this case with a bottom-up approach. That’s the option where you build the fiddly stuff first, then you can forget how it works and just use it.
I created a PacketMapping class which worries about the specs::Entity <-> LocalId(u8) mapping. A PacketTracking class deals with the “which snapshot contains which entity” problem. An EnlivenData system which decides which entities go to which client. Each of these modules has plenty of unit tests.
Now I could make my snapshots very slightly smaller (smaller ID), but more importantly I can send just SOME of the info each tick so when the snapshots are getting too large I can reduce the snapshot size.
Still TODO: We can make the snapshots far smaller by doing the full delta encoding and use then use compression to make it smaller still. I’ll be doing that soon!
So, now you’re up to date!
To stay that way, please follow me:
Twitter: https://twitter.com/RecallSingular1
Discord: https://discord.gg/tRCuSNH
Recall November 2019 or read about Rendering a 2D game in 3D
Here’s a video from last night with a demo of everything mentioned here:
