Time Synchronisation problem and GCP products

Published in

Google Cloud - Community

22 min readJan 1, 2024

Almost always it seems reasonable that you have believed in your machine’s clocks without questioning its accuracy — after all, who wouldn’t? However, it’s essential to recognise that clocks, particularly in distributed systems, are not highly reliable.

Recently, I encountered a situation where I needed to ensure that Google Cloud Platform (GCP) products comply with MiFID regulations. During my investigation, my curiosity led me to delve deeply into the intricacies of time. As a result, I am now focused on verifying that time in distributed systems operates as expected, particularly within the context of GCP. I’ve now fired your curiosity a bit; now let’s delve into the intricacies of clocks and explore why they’re unreliable.

The Crucial Role of Clocks and Hardware Oscillators

I am going to provide some background information on the importance of clocks in this paragraph. While I personally tend to skip article introductions it is crucial to include introductory notes for a broader audience. It has always been important to figure out when to send a reminder email, assess user time on our site, compute the 99th percentile response time of services and so on… Time-related calculations have a wider range of applications, particularly when working with multiple machines. Each machine possesses its own hardware clock, and despite the presence of a central synchronization mechanism (which we will discuss later), these individual hardware clocks play vital roles in the system.

But hold on, what exactly are those hardware clocks? Well, typically, they are quartz crystal oscillators — sounds fancy, right? Let’s break it down! The quartz crystal, a renowned component, is well-known for maintaining a stable frequency during oscillation. Essentially, it’s a mineral made of silicon dioxide (SiO2), and it’s one of the most common minerals on Earth.

Now, what does oscillation mean? In simpler terms, when something oscillates, it regularly moves back and forth around a central point, following a repeated pattern, like the swing of a pendulum or the vibration of a guitar string.

And here comes the next question: why is this important at all? In order to understand it we need to look into how clocks are actually working. Here I have prepared a simple yet descriptive diagram in which we will understand general concepts of them.

The Mechanics of Clocks

in Figure 1 I have drawn overall summarised diagram of what indeed happens in the background when your clock is working. Let’s start explaining each part of it.

Everything started when someone has invented the Piezoelectric Effect. Before you ask chatGPT what is that effect about, let me paste response here: “The piezoelectric effect is a phenomenon where certain materials generate an electric charge in response to applied mechanical stress or deformation. Conversely, these materials can also undergo mechanical deformation when subjected to an electric field”. So those materials can produce electricity from mechanical stress (Direct Piezoelectric Effect) and they can go into mechanical deformation when they receive electricity (Inverse Piezoelectric Effect). Ok what do i do with that information? There are a lot of places where the piezoelectric effect has been used, to make it more clear, I will give an example. When you apply pressure on a lighter, it generates an electric charge, leading to the ignition of gas and the initiation of a flame. That ignite is happening because it uses the direct piezoelectric effect of crystal. And our lovely clocks are a great example of the inverse Piezoelectric Effect. Breaking down the workings of our clock into components, let’s begin with the initial stage where we will also understand the inverse Piezoelectric Effect.

As electrons flow through the battery, they impart an electric charge to the quartz crystal. When these crystals receive the electricity, they initiate oscillations (while ‘vibrate’ is a suitable term, ‘oscillate’ adds a touch of sophistication), a phenomenon which is called the Inverse Piezoelectric Effect. Quartz crystals are known for their stable frequency, and when provided with voltage, they produce signals at 32,768 Hz ( 2¹⁵)

Now, why precisely 32,768 Hz? First of all, they are calibrated during manufacturing to achieve this frequency. But why this specific frequency? Because human hearing generally ceases at frequencies beyond 20,000 Hz. So, the crystals are calibrated to a frequency just above this threshold in order to avoid noise that they can cause. Now, why 32,768 Hz instead of, say, 32,000 Hz because it is also greater than 20000 Hz? The reason lies in the fact that 32,768 (2¹⁵) is the smallest power of 2 greater than 20,000 Hz. Why is the power of 2 significant in this context? Let’s explore this further in the subsequent section, where we delve into logical circuits, which constitutes the second part of our diagram.

The diagram above represents a series of D flip-flops. While you may encounter slightly varied schematics in different sources, any circuit identified as a D Flip-Flop (DFF) will consist of below logical gates.

Figure 4. logical gate representation of DFF

The squares depicted in Figure 3 comprise multiple NAND gates working together to transform the 32,768 Hz signal into a single electrical pulse. Each D flip-flop successively divides the frequency by two. Since quartz crystals provide stable frequencies, their output frequency serves as the CLK input to the D Flip-Flops. The initial 32,768 Hz frequency entering the microchip undergoes successive halving (dividing by 2), resulting in a final frequency of 2⁰ Hz (1 Hz) after 15 consecutive divisions (hence 2¹⁵), which corresponds to a single electrical pulse.

This circuit is commonly referred to as a ‘divide by 2’ counter. Observe the input and output diagram for four consecutive D Flip-Flops (DFF). Envision clock pulses as a stable input source, such as a quartz crystal. This helps to understand how the DFF divides incoming frequencies to produce a single signal. The 4-bit counter shown counts up to 15 (binary 1111) when provided with a stable clock pulse. Notably, if you examine the final output on the QD line, it appears to have converted a 2⁴ input to a 2⁰ output, effectively demonstrating a single signal alteration.

Now that we have delved into these calculations, let’s explore the practical applications of that specific single electrical pulse.

**Figure 6. General concept of stepping motor**

The generated pulse reaches to the stepping motor, as illustrated in the diagram above. Stepping motors play a crucial role in numerous products, and our DC stepping motor, while relatively simple, performs a fundamental task. Its primary function is to rotate the gear consistently in one direction upon receiving a signal. While the actual stepping motor is more complicated than depicted in the above diagram, the essence of its operation can be grasped from the illustration.

Recall that when current flows through a solenoid, it generates a magnetic field around it and altering the current direction changes the North and South poles of the magnetic field. This phenomenon occurs because certain materials exhibit magnet-like behavior in response to an electrical pulse. Using that information, we induce rotation in the actual magnet situated on the left side of the diagram. Considering that the same poles repel each other in magnets, when the solenoid receives an electric current, its South pole and the magnet’s South pole push against each other. The non motorized magnet, not fixed in place, begins to rotate. This rotation, in turn, causes the gear to rotate as well, leading to the creation of a single clock second!!!

From Quartz to Atomic Clocks

After going through the “comprehensive” explanation of how quartz clocks operate, you might find yourself questioning the absolute reliability of your clock. Indeed, your skepticism is justified, as the seamless collaboration of these components is not always guaranteed. Quartz clocks can be influenced by various factors, including temperature fluctuations, magnetic interferences, and potential mechanical gear issues. These variables contribute to the imperfection of these devices, resulting in less-than-perfect accuracy. Consequently, each clock in a network may develop its own timekeeping nuances, running slightly faster or slower than its counterparts.

In our distributed world, such discrepancies necessitate a solution for synchronization. This is where the Network Time Protocol (NTP) comes to the rescue. NTP, the most widely used protocol, which enables computers to adjust their clocks based on time reports from a cluster of servers. These servers, in turn, derive their time from more accurate sources, such as GPS receivers.

Notably, there is an alternative to quartz clocks known as atomic clocks. While they also utilize quartz crystals, they incorporate a self-correction methodology to address any deviations in frequency caused by external interferences. Atomic clocks, commonly paired with GPS receivers, offer enhanced precision in timekeeping.

Despite using highly synchronised clocks, certain issues may still arise, including:

Clock Drift: If your quartz clock experiences drift within your server, even with sync requests occurring every 30 seconds from an NTP server, inaccuracies may occur between those intervals.
Significant Clock Discrepancies: When the computer clock deviates substantially from NTP servers, it might refuse to synchronize, as part of the synchronization algorithm.
Firewall Misconfiguration: Accidental firewalling of your node from NTP servers can go unnoticed for an extended period. Identifying the root cause of server time discrepancies becomes challenging, leading the server to rely solely on its internal hardware clock.
Leap Seconds: The concept of leap seconds involves occasional adjustments to Coordinated Universal Time (UTC) to accommodate irregularities in the Earth’s rotation. During these adjustments, time synchronization with servers may encounter errors, resulting in unexpected jumps backward or forward in time.

The issue of synchronization has not escaped the attention of governments, particularly as they recognize the potential for companies to manipulate timing data, creating market anomalies. For instance, MiFID, a draft European regulation for financial institutions, mandates that high-frequency trading funds synchronize their clocks within 100 microseconds of Coordinated Universal Time (UTC). This requirement aims to aid in troubleshooting market anomalies such as “flash crashes” and detecting instances of market manipulation. Achieving such precision often involves the utilization of GPS receivers in conjunction with atomic clocks.

Moreover, a specialized protocol, the Precision Time Protocol (PTP), has been developed for systems demanding extremely accurate timekeeping. PTP, however, demands substantial effort and expertise, and the equipment involved must support the PTP protocol, including routers. Nevertheless, even with PTP, there is no absolute guarantee of identical time across all servers in a network at any given moment. So, in light of these challenges, what solutions does Google Cloud Platform (GCP) provide, as hinted at in the title of this article?

Google’s NTP server

In the realm of time synchronization on Google Cloud Platform (GCP), there are two crucial concepts to understand: Google’s handling of leap seconds through its Network Time Protocol (NTP) servers, and the TrueTime API used predominantly by Spanner. Let’s start with leap seconds. Coordinated Universal Time (UTC) is periodically adjusted with leap seconds to stay aligned with Earth’s rotation. This adjustment is necessary because UTC, based on atomic clocks, does not inherently account for variations in Earth’s rotation, which can be influenced by factors like lunar and solar interactions, and even global warming.

Twice a year, on June 30 and December 31, the International Earth Rotation and Reference Systems Service decides whether to add or subtract a second from UTC. This adjustment is managed differently by various NTP servers, and Google has its unique approach. Normally, a leap second in UTC would be added as an extra second at the end of the day, resulting in a 61-second minute. But Google uses a ‘time smearing’ method, distributing the leap second across a 24-hour period.

This smearing starts 12 hours before the leap second. Google’s clocks run slightly slower, making each second marginally longer than a standard second. By the time of the leap second event, Google’s time is half a second ahead of UTC. The clocks continue to run slowly for another 12 hours, cumulatively adding another half second, thus realigning Google’s time with UTC. For instance, during smearing, a second might be lengthened to 1.00001157 units, so over 43,200 seconds (12 hours before and after), it adds up to one second.

Google’s documentation provides additional commands for configuring virtual machines to use Google’s NTP servers. It’s crucial not to mix different NTP servers, as this can lead to unpredictable time management. Google strongly advises against such practices. So below output is highly discouraged by Google.

**Figure 7. Sample NTP server list from compute engine.**

Cloud Spanner

Before we explore the inner workings of Spanner, its ability to ensure serialization and external consistency in globally distributed systems, and its relationship with the clock concepts I explained earlier, we must first grasp some fundamental engineering definitions.

Serializable
Atomicity
Linearizabliliy

Let’s delve into the concepts mentioned above and gain an understanding of the problem at hand, as well as explore possible solutions.

Serializable and Atomic transactions

When it comes to transactions, it’s essential to discuss the Atomicity, Consistency, Isolation, and Durability (ACID) principles governing them. Even in a distributed environment, a relational database must find a way to align with these principles. Serializable transactions pertain to the isolation levels in database systems, which essentially mean that concurrently executing transactions should behave as if they are the only ones in the entire database. In contrast, Atomicity ensures that a transaction is either fully executed or completely aborted in case of an error. In other words, success or failure of a transaction is an all-or-nothing affair.

The real challenge lies in achieving these principles in a distributed environment. Let’s begin by discussing isolation. The strictness of isolation levels in a database addresses several common issues as described below.

You can read about all above read pheomena here. However since our primary focus is on the Serializable isolation level, which is the most strong, let’s delve into that. Spanner, on the other hand, asserts that it achieves Strict Serializability, so it’s worth exploring how it accomplishes this.

There are several approaches to achieving the Serializable isolation level, with one of the most well-known and commonly used methods being Two-Phase Locking (2PL). This locking mechanism is actually quite straightforward. It involves two types of locks: shared locks and exclusive locks. Shared locks can be held by multiple readers simultaneously, whereas exclusive locks are exclusive to a single transaction and are primarily used for updating an object.

For example

if transaction A has acquired a shared lock on an object and transaction B wishes to write to that same object, B must wait until A either commits or aborts before proceeding. This ensures that while transaction A is reading the data, it cannot be changed by transaction B. However, the shared lock held by transaction A does not block transaction C from reading the same object hence its shared.
if transaction A has obtained an exclusive lock (for writing) on an object, and transaction B attempts to read that object, B must wait until A commits or aborts before it can proceed. As reading an old version of data while its being changed could lead to wrong results.

We are discussing Two-Phase Locking (2PL) because Spanner, unsurprisingly, utilizes it as part of its approach. However, there are a couple of disadvantages associated with 2PL. The system might be prone to deadlock issues which is not indeed deal breaker. However consider the scenario where you want to take a backup of your database or run fancy complex analytical queries on it. The shared locks held for reading objects in 2PL can block all write operations that require exclusive locks. This can result in a situation where your users experience huge delays and may not be particularly pleased with the performance impact.

Luckily there is a solution for that, multi-version concurrency control MVCC which is also used by spanner as described here. So let’s go through an example in order to understand how MVCC is working.

**Figure 9. Seeing old version of data when it got updated**

In Figure 9, Alice initially has a total of $1000 split between her Account1 and Account2. She queries her Account1 and receives a response indicating she has $500. Subsequently, an asynchronous Transfer operation is initiated, transferring $100 from Alice’s Account2 to Account1, and this transfer operation is committed. However, when Alice queries her Account2, she is astonished to find that she now has only $400 left. This means she now has a total of $500 + $400 = $900, which understandably frustrates her. While one could argue that Alice might eventually see consistent results after a few seconds or upon restarting her internet banking account, in use cases like running analytics or backup processes on a database, encountering such transient inconsistent states is simply unacceptable. Imagine restoring your database from a backup and encountering such an inconsistent state. This is where Multi-Version Concurrency Control (MVCC) comes to the rescue. Let’s continue with the updated version of the problem using MVCC.

We have converted Alice to Txid = 12 and Transfer transaction to txid = 13. You might think this transaction ids are random but actually they are quite important. Lets bring some rules to table and then repeat Alice’s scenario. Here are the rules:

At the start of each transaction, the database makes a list of all the other transactions that are in progress (not yet committed or aborted) at that time. Any writes that those transactions have made are ignored, even if the transactions subsequently commit.
Any writes made by aborted transactions are ignored.
Any writes made by transactions with a later transaction ID (i.e., which started after the current transaction started) are ignored, regardless of whether those transactions have committed.

The most crucial rule in this scenario is the third one (I think), which will also help us understand the TrueTime API. Additionally, we’ve included additional metadata information for objects, such as ‘created by’ and ‘deleted by.’

Let’s consider Alice (Txid = 12) once again querying her Account1. In this case, the value she queries for is not deleted by anyone, and it was created by a transaction with a lower Txid than hers (12 > 5). Consequently, she happily sees that she has $500.

Simultaneously, a Transfer transaction (Txid = 13) begins and creates multiple versions of Account1 and Account2 while transferring $100 between them. As illustrated in the diagram above, the ‘deleted by’ field in the old version is updated with 13, and the ‘created by’ field in the new version is filled with Txid = 13. Later, when Alice queries the database, she retrieves the old record containing $500. This occurs because the new version of the Account2 object has a ‘created by’ field with a higher value than Alice’s transaction (13 > 12), causing the database not to return that record. Which makes Alice happy again as in total she now has 1000$. Moreover here we don’t even need to use locks and it is incredibly helpful in the context of long running queries.

Ok, That works, but only if you provide monotonically increasing transaction ids. In a single-node database or a centrally managed database, a single counter is typically sufficient for achieving this functionality. However, the situation becomes more complex in a distributed environment where nodes operate with a greater degree of independence. In such a scenario, with numerous small and rapid transactions, generating unique transaction IDs in a distributed system can indeed become an untenable bottleneck.

Here is where the TrueTime API becomes invaluable. Instead of relying solely on time as monotonically increasing transaction IDs, which can be problematic due to various time-related issues as mentioned earlier with quartz clocks, TrueTime offers a solution to make clocks highly trustworthy. TrueTime is the API that Cloud Spanner leverages to achieve this level of trustworthiness in its timestamp generation and synchronization. But how? Let’s look into some details.

TrueTime API relies on either GPS or atomic clocks to ensure accurate time. While these time sources are not commonly used due to their cost and management complexity, Google, with its extensive data centers and private network infrastructure (check the premium tier), can effectively deploy and maintain them.

However, it’s important to note that the nodes within the network still use quartz clocks, and they synchronize their clocks with central atomic GPS clocks every 30 seconds. During the 30-second interval between synchronizations, the clock uncertainty of these nodes can increase at a rate of 200 parts per million (ppm), which roughly translates to a 7ms drift in the worst-case scenario.

Let’s look at how TrueTime Api is actually working. When you perform TT.now(), the method provides a pair of timestamps, [earliest, latest], and assures that the exact time when TT.now() was called (according to Google’s highly accurate time reference, as explained above) falls within the interval defined by these timestamps. Cloud Spanner’s backend servers rely on this library to generate commit timestamps and make reasoned decisions based on them.

How? so basically say we have 2 transactions T1.now() and T2.now(),

If T1(earliest) < T1(latest) < T2(earliest) < T2(latest) then T2 is definitely happened T1
If T1 earliest and latest is somehow overlapping with T2 earliest and latest then we just need to wait couple of milliseconds to make sure time synces between those transaction.

Now we have managed understand isolation levels and possible solutions to fix issues there.

How do we achieve Atomicity in distributed environment. in single-node database atomicity is somewhat easy to implement however when it comes to distributed system it is bit challenging. To most well known approach to that is using Two-Phase Commit (don’t mix it with Two-phase lock). 2PC is mechanism to ensure that either all nodes commit a transaction or all of them aborts it. Let’s look at how it is working.

Two-Phase Commit (2PC) employs a component that isn’t typically present in single-node transactions: a coordinator, also known as a transaction manager. In a 2PC transaction, the application initially reads and writes data across multiple database nodes, which are referred to as participants in the transaction. When the application is prepared to commit, the coordinator initiates phase 1. During this phase, it sends a prepare request to each of the nodes, inquiring if they are ready to commit. The coordinator then keeps track of the responses from these participants:

If all participants respond with a “yes,” signifying their readiness to commit, the coordinator proceeds to phase 2 and issues a commit request, resulting in the actual commit taking place.
In the event that any participant responds with a “no,” the coordinator sends an abort request to all nodes during phase 2, leading to the transaction’s cancellation.

As illustrated in the diagram above, it’s evident that the main bottleneck in this setup is the coordinator. But what happens if the coordinator were to fail? To address this question, we first need to delve into the concept of consensus algorithms, with Paxos being one of them as Spanner is based on that. As Paxos has a bit of relation with Linearizabliliy I will explain it under that topic and then come back to problem of coordinator failure and how spanner uses Paxos to fix it.

Linearizabliliy

also known as strong consistency is a concept in distributed systems that ensures the system behaves as if there was only one copy of the data and that all operations on it are atomic. This means that reads always include the most recent writes. It provides the strongest consistency guarantee among various models. However it is quite challenging task in distributed systems if you have multiple copy of data distruted over nodes.

As evident from the diagram above, the number of replicas does not significantly affect the client’s experience. When the client sets the value of ‘x’ to ‘v1,’ the client’s concern about the number of replicas is minimal. Suppose, for instance, that our replica A fails to obtain ‘v1’ due to a network issue during the set operation. Later, when the client performs a ‘get’ operation, let’s assume his request did not reach replica C, resulting in values being fetched from both Replica A and B. Even though Replica A did not successfully complete the set operation, it is important to note that the client still needs to observe ‘v1’ because ‘t1’ is greater than ‘t0.’ Consequently, the client obtains ‘v1,’ demonstrating that the system maintains linearizability.

However, linearizability comes with significant trade-offs:

Replication Mechanisms: Not every kind of replication mechanism can be made linearizable. For instance, multi-leader replication is not linearizable due to asynchronous propagation of writes that could lead to conflicts. Single-leader replication may or may not be linearizable depending on certain conditions like the use of snapshot isolation. Leaderless replication usually isn’t linearizable due to the internal techniques used, like sloppy quorums.
Availability and Performance: Linearizable systems lose availability in the event of a network partition. The response time for read/write requests in these systems is at least proportional to the uncertainty of delays in the network. This implies poor performance, especially in unreliable networks like the Internet.

Spanner ensures high availability and linearizability by employing the Paxos consensus algorithm for synchronous replication (quorum alone does not guarantee linearzibilty). This means that whenever data is written to a Spanner database, it is replicated across multiple geographic locations. This redundancy ensures that the data remains accessible even if a certain number of replica copies experience failures.

To elaborate on how Paxos operates within Spanner: when a write operation is initiated, Paxos plays a crucial role in managing the process. Initially, Paxos designates a leader responsible for coordinating the write. This leader then disseminates a write proposal to all the replicas in the system. For the write to be considered accepted and for the system to progress, a majority of the replicas must acknowledge and accept the proposal. This mechanism optimizes the throughput of write operations and enables them to proceed smoothly, even in the presence of replica failures.

Furthermore, it’s important to note that data within a Spanner instance is divided into distinct groups. These groups are referred to as Paxos groups because each group operates with its independent Paxos protocol engine, facilitating data replication within that specific group. This decentralized approach ensures efficient management of data replication and fault tolerance within the Spanner system.

I learned while reading spanner’s white paper that an early Spanner incarnation supported multiple Paxos state machines per tablet, which allowed for more flexible replication configurations. The complexity of that design led them to abandon it.

In scenarios where transactions span multiple Paxos groups, Spanner employs the two-phase commit protocol alongside long-held locks to ensure that read-write transactions maintain external consistency. Previously, we discussed the limitations of the two-phase commit protocol, particularly regarding the potential bottleneck created by the coordinator. As highlighted earlier, the failure of the coordinator can lead to transaction participants getting stuck in a waiting state and I said we will come to that point so here we are: To address that concern, Spanner combines the two-phase commit protocol with the Paxos algorithm. This integration involves replicating both the transaction coordinator and the transaction participants within Paxos groups. This approach offers several advantages:

Automatic Coordinator Election: By replicating the transaction coordinator across Paxos groups, Spanner ensures that in the event of a coordinator failure, a new coordinator can be automatically elected. This prevents transaction participants from being stranded in a waiting state and ensures the continued progress of transactions.
Enhanced Fault Tolerance: The combination of two-phase commit and Paxos makes the entire protocol more fault-tolerant. It reduces the system’s vulnerability to failures and enhances its resilience in the face of unexpected issues.

In summary, Spanner mitigates the coordinator bottleneck problem by integrating the two-phase commit protocol with the Paxos algorithm. This design choice not only enables the automatic election of a new coordinator upon failure but also boosts the overall fault tolerance of the system, ensuring the reliability of read-write transactions.

Final example

This example is great because it makes the concept underlying all of that complicating reasoning very clear. So the Paxos leader follows these procedures while processing writes related to a transaction in a Paxos group.

Commit Timestamp Selection: A commit timestamp is chosen by the Paxos leader and is indicated as commit_ts := TT.now().latest.
Sends Paxos write proposals to the replicas and waits for a quorum of replicas to agree on them. It is the prepare state of 2PC.
Commit Wait: A critical step is the commit wait, where the system waits until TT.now().earliest > commit_ts. This ensures that no client can observe the effects of the transaction until its commit timestamp is in the past. Consequently, if there is a causal relationship between two write transactions, such as transaction t2 being dependent on the outcome of transaction t1, t2's commit timestamp is guaranteed to be greater than t1's commit timestamp. It's important to note that the commit timestamp is determined in step #1, and the commit wait occurs in step #3. Effectively, the wait happens concurrently with the transmission of Paxos proposals and the waiting for a quorum. Due to the precise uncertainty window provided by TrueTime, the commit wait step typically involves little to no actual waiting. This holds true even when transactions span multiple Paxos groups, where the fundamental effect remains the same.

In essence, this approach is how Spanner ensures external consistency. Irrespective of the timestamp used by a transaction to read the database, it will consistently perceive a version of the database that is fully synchronized up to that specific point in time. This means that even when multiple independent actors are concurrently operating on the database, there is no necessity for explicit coordination among these actors to arrange causally related events in a manner that guarantees perfect consistency. Spanner’s utilization of two-phase locking and commit wait also enables it to assure perfect isolation. Coupled with the precision offered by TrueTime, Spanner rarely encounters performance degradation due to the need to wait for clock uncertainty to resolve.

Conclusion

In summary, the world of time synchronization in distributed systems reveals the fragility of clocks. While they may seem reliable, various factors can affect their accuracy. This precision is essential in diverse applications, from email reminders to service response times. Quartz crystals, with their stable frequencies like 32,768 Hz, play a central role in clocks. The choice of this frequency considers human hearing limitations. However, imperfections arise due to temperature, magnetic interference, and mechanical issues, leading to clock discrepancies across a network. To address these issues, the Network Time Protocol (NTP) comes to the rescue, allowing computers to adjust their clocks based on highly accurate sources like GPS receivers. Atomic clocks, another alternative, offer enhanced precision.

Challenges persist, such as clock drift, firewall misconfigurations, and leap seconds. Regulatory bodies like MiFID mandate microsecond-level synchronization for financial institutions.

On the other hand, GCP offers solutions in this area:

Google’s NTP Servers: GCP handles leap seconds uniquely through a “time smearing” method, ensuring smoother transitions.
Cloud Spanner: GCP’s Cloud Spanner uses the Paxos consensus algorithm, the two-phase commit protocol, and the TrueTime API to address the challenges of maintaining strong consistency and atomicity in distributed systems.

In conclusion, time synchronization remains crucial in distributed systems, impacting various industries. GCP’s innovative approach, especially through Cloud Spanner, provides robust solutions for maintaining precise time across networks.