Accurately syncing Unreal’s network clock

When implementing real-time network multiplayer in a game, one very important tool is a reliable network clock.

By “network clock” I mean a representation of the current game time that is shared amongst all network clients, so every machine on the network has the same idea of what time it is. That’s important because, on most internet connections, you can’t afford to replicate all the simulation data — player positions, projectiles, effects, and so on — every single frame, because there’s just not enough available bandwidth for it all. Instead, you have to replicate just the most crucial data and let each network client fill in the gaps with their own local simulation. But you also need to make sure each client is producing the same results so everyone has consensus on the current state of the game. If everyone can agree on something fundamental — like what time it is — it becomes much, much easier to write such a simulation.

For example, let’s say you and I are playing a shooting game over the internet. When I fire a projectile, my computer needs to tell your computer about it. As my projectile moves toward you on my machine, your machine needs to also render the projectile moving toward you, and it’s really important the projectile is in the same place on both machines at any given moment. The naïve way to ensure that is for my machine to tell your machine where the projectile is every single frame:

// pseudocode
void Update(float deltaSeconds)
{
    projectile.position += projectile.velocity * deltaSeconds;
    SendProjectileDataToClient(projectile.position);
}

This uses up a lot of network bandwidth, especially when there are lots of projectiles flying around at once. More importantly, it’s not even necessary: once a projectile is fired, we know it’s going to travel in a straight line at a fixed speed until it hits something. As long as the client knows how fast the projectile is and where it was fired from, the client can simulate the projectile locally after that and it will still match up with the server even though the server isn’t sending any more updates (at least until a collision occurs… but that’s a separate topic).

The only problem here is lag, which is the amount of time it takes data to physically travel from the server to the client. This isn’t instant; transmission may take anywhere from a few milliseconds to several seconds, depending on the quality of both the server’s and client’s network connections. The effect of lag is that the server spawns the projectile and sends a message to the client; while that message is en route, the server continues to move the projectile; when the message finally arrives at the client, the client spawns the projectile at the original spawn location (the location indicated in the message), and that now lags slightly behind the server’s idea of where the projectile is because of the time that passed in between.

We can fix this by using the network clock. If both machines have the same idea of what time it is, and the message that tells the client to spawn the projectile is timestamped with the time it was sent, the client can figure out how long the message took to arrive (the duration of network lag) and move the projectile forward from its initial spawn point by the distance it would’ve covered in that time:

// pseudocode (on the server)
void Update(float deltaSeconds)
{
    projectile.position += projectile.velocity * deltaSeconds;
    SendProjectileDataToClient(
        projectile.position,
        projectile.velocity,
        networkTime
    );
}

// pseudocode (on the client)
void ReceiveProjectileDataFromServer(
    vector position,
    vector velocity,
    float timestamp
)
{
    float lagDuration = networkTime - timestamp;
    projectile.position = position + (velocity * lagDuration);
}

Now the projectile is in the exact same position on both the server and the client at the exact same time; we’ve effectively eliminated network lag! But for it to work, we are absolutely reliant on the server and client having the same value for networkTime.

Unreal’s network clock

(The following analysis is based on the 4.18.2 release.)

Unreal Engine 4 implements a synchronized network clock which can be used for just this kind of thing. You can get its value from AGameStateBase::GetServerWorldTimeSeconds(), and that value is replicated to all network clients, so in theory, everybody should have the same idea of what time it is.

Unfortunately, Unreal’s network clock sync is… not all that accurate.

I brought up a PIE network session with a listen server and one client and had each render the result of AGameStateBase::GetServerWorldTimeSeconds() to the screen. Here's what that looked like (watch the blue text in the upper-left corner):

(Both times are rendered in the listen server’s window due to a quirk of how onscreen debug messages are rendered in PIE: they always render to the first viewport, regardless of which client actually invoked them.)

Notice how the server and client clocks drift apart, and how the degree of difference seems to change almost randomly? Sometimes they’re pretty close to dead-on, and other times they’re a second or more apart.

This happens because of the way Unreal’s network clock is synchronized. To get into this, we’re going to need to dive into the engine code:

• AGameStateBase has a replicated variable ReplicatedWorldTimeSeconds which is updated with the current result of GetWorld()->GetTimeSeconds() every five seconds.
• Each time ReplicatedWorldTimeSeconds is updated, the new value is sent across the network to all clients, whose local copies of that variable are updated in turn.
• Each client compares ReplicatedWorldTimeSeconds with its own local GetWorld()->GetTimeSeconds() and stores the difference in ServerWorldTimeSecondsDelta.
• Finally, whenever you call GetServerWorldTimeSeconds(), what you get back is your local GetWorld()->GetTimeSeconds() + ServerWorldTimeSecondsDelta.

So basically, the server reports the current time every five seconds, and each client maintains an offset between their current time and the last time they heard from the server in order to derive the server’s current time whenever it’s requested. This saves bandwidth because the time is only replicated once every five seconds instead of every frame, and the client can fill in the gaps accurately because time advances linearly and at a fixed rate by definition (leaving engine time dilation aside for now).

And yet, we still have this drift! If you watch the above video again, you’ll notice the degree of drift changes every five seconds, which matches the replication frequency of the network clock. But why isn’t it dead-on, and why does the degree of error appear to be random?

Making the network clock accurate

The answer is, of course, lag. Every time the server replicates the current time, it takes a little bit of time for the client to receive that message. By the time the client hears that it’s time T, it’s actually time T plus “a little bit”. And the client has no idea how much “a little bit” is, because it has no idea when the message was dispatched. And sure, the server could include a timestamp with the message, but what good is the timestamp when we don’t yet agree on what time it is in the first place?

Fortunately, there’s a very easy way to compensate for this: instead of the server sending the time to clients, we have clients ask the server what time it is, then record how long it takes to receive a response and offset the result by that time. Once we’ve done that handshake just once, the client’s local time is now virtually identical to the server’s time — in real time! — and we never need to sync our times ever again.

To illustrate: say we started the server a minute ago, so its local time is 60 (seconds), and we just now started a client, whose local time is 10.

If the server sends the client a message saying the time is 60, and the message takes 5 seconds to arrive — far longer than reality, but useful for this illustration — then by the time the client hears the time is 60, the time will actually be 65, and the client will be running 5 seconds behind.

However, say the client sends the server a message requesting the current time, and writes down that it dispatched that request at client time 10 (server time 60).

The message takes 5 seconds to reach the server, which responds with time 65, and the server’s response takes a further 5 seconds to get back to the client.

The client receives the response of 65 at client time 20, 10 seconds after it sent the initial request. Because the server’s response took 5 seconds to get back to the client, by the time the client hears the server time is 65, the actual server time will be 70.

But the client now knows it took 10 seconds between sending its request and receiving the server’s response, and since half of that is the transit time to the server and the other half is the transit time back from the server, the client can safely assume the server advanced a further 5 seconds while its response was in transit. The client can therefore adjust the reported server time of 65 up to 70, which is accurate.

From here on out, the server and client stay near-perfectly synced, with no further network traffic required:

Implementing it in Unreal

This simplest and least-invasive way to do this is with custom GameState and PlayerController classes.

(We use the PlayerController class because it’s one of the few types for which clients own the network connection, meaning server RPCs can be called. We rely on a server RPC for the client to request the current server time, as you’ll see in a moment.)

Here’s what we add to our custom PlayerController class header:

public:

/** Returns the network-synced time from the server.
  * Corresponds to GetWorld()->GetTimeSeconds()
  * on the server. This doesn't actually make a network
  * request; it just returns the cached, locally-simulated
  * and lag-corrected ServerTime value which was synced
  * with the server at the time of this PlayerController's
  * last restart. */
virtual float GetServerTime() { return ServerTime; }

virtual void ReceivedPlayer() override;

protected:

/** Reports the current server time to clients in response
  * to ServerRequestServerTime */
UFUNCTION(Client, Reliable)
void ClientReportServerTime(
    float requestWorldTime,
    float serverTime
);

/** Requests current server time so accurate lag
  * compensation can be performed in ClientReportServerTime
  * based on the round-trip duration */
UFUNCTION(Server, Reliable, WithValidation)
void ServerRequestServerTime(
    APlayerController* requester,
    float requestWorldTime
);

float ServerTime = 0.0f;

ServerTime keeps track of the client's idea of the server's current time. It is not a replicated variable; it's just a plain ol’ float.

We also declare two network RPCs: ServerRequestServerTime for making the initial request, and ClientReportServerTime for the response. We pass requestWorldTime to both; this is how the client tracks the roundtrip time.

Finally, there’s the public GetServerTime() which will shortly be used by our custom GameState.

In our PlayerController class’s .cpp file, we implement the RPCs:

void APlayerController::ServerRequestServerTime_Implementation(
    APlayerController* requester,
    float requestWorldTime
)
{
    float serverTime = GetWorld()->GetGameState()->
        GetServerWorldTimeSeconds();
    ClientReportServerTime(requestWorldTime, serverTime);
}

bool APlayerController::ServerRequestServerTime_Validate(
    APlayerController* requester,
    float requestWorldTime
)
{
    return true;
}

void APlayerController::ClientReportServerTime_Implementation(
    float requestWorldTime,
    float serverTime
)
{
    // Apply the round-trip request time to the server's         
    // reported time to get the up-to-date server time
    float roundTripTime = GetWorld()->GetTimeSeconds() - 
        requestWorldTime;
    float adjustedTime = serverTime + (roundTripTime * 0.5f);
    ServerTime = adjustedTime;
}

And we override ReceivedPlayer() to add in the network clock request at the earliest point we have a valid network connection:

void APlayerController::ReceivedPlayer()
{
    Super::ReceivedPlayer();

    if(IsLocalController())
    {
        ServerRequestServerTime(
            this,
            GetWorld()->GetTimeSeconds()
        );
    }
}

Finally, in our custom GameState class, we override GetServerWorldTimeSeconds() which is how everything else will actually access the network clock:

float ACustomGameState::GetServerWorldTimeSeconds() const
{
    if(APlayerController* pc = GetGameInstance()->
        GetFirstLocalPlayerController(GetWorld())
    )
    {
        return pc->GetServerTime();
    }
    else
    {
        return GetWorld()->GetTimeSeconds();
    }
}

(We do not call Super:: in this implementation; we don't want any of the old, inaccurate behavior at all!)

Here we’re just grabbing our first local PlayerController and returning its idea of the server time. We do this because the PlayerController is the one that’s actually synchronized the clock (because PlayerControllers have locally-owned network connections which allows them to call server RPCs). In the event we have no PlayerControllers we fall back to the local world’s time, but we don't expect that to actually happen in a real game (and even if it does, we don’t care because in that case we obviously don't have any other network players we need to be syncing with right now).

And that’s all there is to it!

You could also implement this same concept in Blueprints (which I’ll leave as an exercise for the reader) or you could patch the engine source directly (which I’ve done in this pull request).

Caveats

This network clock sync is far more accurate than Unreal’s default method, and should be more than accurate enough for games, but it’s not absolutely perfect. There is a very tiny bit of potential error in the way we correct the time received from the server: we offset it by half the roundtrip time, but we don’t actually know (or, to my knowledge, have any way to know) that the outbound and inbound transit times were an exact 50/50 split. If it took longer for our request to reach the server than for the server’s response to reach us (or vice-versa) then our clock will be very slightly off, proportionate to the difference between those two transit times. (It’s actually possible for the client to be very slightly ahead of the server in this scenario.)

I’ve spent a lot of time and energy trying to figure out a truly perfect solution. Part of me thinks it’s unsolvable, but another part of me thinks the solution is almost visible. I’ve been exploring an idea where the client asks for the time twice, and the server and client compare how much time passed on each in between those two requests and do some math to derive the one-way transit time of a packet in the past, which is then used to absolutely correct the reported time now. I haven’t been able to make the math work yet (and it might never) but perhaps there’s an exercise for the reader there.

Also, if you look at my pull request there’s some additional stuff in there not covered in this article which has to do with replay recording for network sessions. This post is long and complicated enough and I decided to de-scope that stuff; a lot of people probably wouldn’t use it anyway.

Finally, Unreal supports pausing and time dilation, and in the implementation described here I use a timer that respects those features. If time dilation is in effect when you try to synchronize your clocks, you may get different results. I would expect it to work if both machines have the same time dilation values — which of course is a separate network replication problem you’ll probably need to solve on a per-game basis — but if time dilation is changing in realtime (e.g. animating up or down) at the same time you’re trying to synchronize clocks, you’re probably gonna have a bad time.