How much latency is there between Google Cloud Storage locations?
Google Cloud Storage is one of the multiple storage options offered by the Google Cloud Platform but the only that provides global access scope. It means that all your applications deployed to any zone and any region can access any Bucket in GCP. Other options have zonal, or even local access so they must be attached in the same zone or the same VM as your application.
Although the access scope is global, it doesn’t mean that performance is the same across the World!
The performance of Cloud Storage buckets depends on the storage class that you select and the location of the bucket relative to your instance.
Source: https://cloud.google.com/about/locations/#network
However, GCS buckets are limited to a location so files are not automatically distributed across the World. The biggest locations you can choose is the United States, Europe, or Asia. If you try to access a bucket in the USA from the application deployed in Europe, the object will have to be transferred through the Atlantic Ocean. What latency does it cause? Can it cause performance issues? How to make the correct choice? Let’s find out.
The test application will upload three files with sizes 1 MB, 5 MB, 10 MB and calculate average upload time for selected buckets and file sizes. The application will be deployed in a VM located in the us-west2.
You can find the source code of the application in my Github: https://github.com/RyszardSzwajlik/gcs-performance-test
Results
The lowest upload time is for the us-west2 region — the region where the VM with test application was deployed. We are mostly interested in latency between regions. To calculate it, I will assume that us-west2 has zero latency and is the minimal time required to save data in the storage. I will use it to calculate the absolute latency participation. Every column in the next tables are the result of the calculation:
Where:
- cell is the cell with measurement for given file and region
- usWest2 is the measurement for us-west2 region for the given file
Region vs latency
When rows were sorted by average latency impact, great relationship between region, storage classes, and latency have emerged:
The lowest latency impacts are for US Region buckets. Then, US Dual Region, then Multi-Region and Regions for non-US continents at the end. It’s worth to notice that Region buckets are the fastest for locations close to the upload source and the slowest for Region buckets that are located far away from the upload source!
File size vs latency
The second goal of the test is to check the impact on latency based on the file size. It turns out, that the latency impact is higher on smaller files and lower for bigger files.
It may be useful to have in mind that relationship when choosing the best option for your use case, so…
Which location and which storage class should you choose?
There is one general rule provided on Google Cloud Docs:
Generally, you should store your data in a location that is convenient or contains the majority of the users of your data.
Based on the performed test, the correct location choice can boost your upload up to 70%!
The answer is not that simple however in case of global application. A tempting compromise in this situation seems to be choosing a Multi-Region storage class. It has not the worst latency impact and offers high data availability. If you need a relatively constant latency impact for each region in the World, you can consider having just one Multi-Region storage class. The expected file size can help you make the decision. It’s also a quick and easy solution to implement.
What if Multi-Regions are too slow?
Well, if you need a globally-available, highly-performant application then you are faced with a challenge. The configuration may be complex so think twice if you really need to do it. You will need to invent your custom solution based on your domain knowledge. To find it, let’s get inspired by the ideas that come from databases world: sharding, caching, eventual consistency and replication.
Sharding
If you expect that your users will access resources in local regions, you can shard the data into multiple Region classes so your users will access files relatively quickly:
We need to remember about clients from other regions so the application should be able to request other buckets. They still will be able to get the data but with worse performance.
Caching and eventual consistency
To improve the above solution so clients from other regions have quicker access to files stored in distant locations we can add a cache for reading files. The idea makes sense only if your clients read files frequently. Remember that the data may be eventually consistent sometimes, which means that clients in Region 1 can potentially get data different than clients in Region 2.
Replication
If you are unable to shard your data, you can consider replication on the pattern of mirror servers. What’s more, you can replicate your data to other Storage Products like i.e. Persistent Disk so we could mix benefits that come from those ideas. Bear in mind however that this solution may affect the total storage cost.
The solution may be implemented in many different forms so instead of diagrams I will just list ideas:
- Synchronize the content of buckets/directories with rsync tool.
- Synchronize the content using the application responsible for file upload
- Synchronize the content using Google Cloud Functions that would be executed under specified circumstances (i.e. a file was uploaded)
Summary
The main goal of the article was to depict how much the bucket location affects the speed of file access. During performance tests, it turned out that storage class and file size makes matter.
Regional buckets have the best performance if they are localized close to the user and the worst when the bucket is far away from the user. The reasonable compromise may be Multi-Region class but if the performance is very important, there may be implemented some more complex solutions.
