Google Cloud Storage Performance

Google cloud storage is listed as a “a unified object storage solution” … or in layman’s terms, it’s a place in the cloud to host and serve files.

The uses for GCS are pretty large. For example, some people host their static websites there, some ship game content from there, others have used it to power their back-up services, and let’s not forget, it can host and serve content just like a CDN.

That’s all fine and dandy, but what I’m concerned with is : how would you test performance of it?

Direct test — Read performance

As we looked at in previous posts, it’s really easy to just throw a curl command around to see how GCS performs when fetching an asset.

First, place an asset on a server, say, in North America, (maybe near a BBQ restaurant) and then fetch that asset from a bunch of machines located around the world.

for ((i=0;i<500;i++)); do (time curl http://examplecdn.cdn/asset.png) 2>&1; done | grep real

The graph below shows our asset being fetched from a machine in Asia, Australia and Europe.

Note : Caching was not enabled for these assets, so their numbers are higher than what you’d see from the CDN article. AND this bucket was intentionally set to Regional (instead of multiregional) We can talk about the perf differences, but that a topic for another article.

Now this bucket was set up to be Regional, which means it doesn’t scale properly to being accessed across regions, so you can expect that the farther away the request is, the slower the response can be.

Direct test — Write performance

But we’ve seen read performance be really strong with GCS in the past; Write performance, however, is much trickier to get correct. In some following articles, I’ll highlight a few gotchas that some developer friends ran into, but for now, let’s just upload the same file, under different names, to the US-CENTRAL-1 bucket from various places in the world.

Now, if you run this test yourself, be careful about how you do your timings. There’s two main ways to do upload data to GCS: via `gsutil cp` and via a language-specific API. I’ve charted the performance of both below.

Note : These tests were done on a bucket was intentionally set to Regional (instead of multiregional) We can talk about the perf differences, but that’s another article.

We can see that write performance, for a 600k file is really good over time, however we see a significant difference between the GSUTIL and Python versions. The reason for this is that our scripting process for GSUTIL will cause a new process to be created for each uploaded asset, which isn’t ideal in terms of processor usage, and may not directly be GSUTIL’s fault.

Note : There’s also a multithreaded upload flag, but we’ll talk about that in a future article.

Using perfdiag

Direct read/write performance is easy to test. But in honesty, there’s lots of variables here that influence performance, that’s a pain to write proper benchmarking for. As we will talk about later, performance can greatly vary depending on the size of your objects, or if you’re creating, deleting, uploading or downloading. Thankfully, you don’t have to write all those tests yourself. The gsutil perfdiag command runs a suite of diagnostic tests for a given Google Storage bucket:

  • Read/Write throughput tests (1MiB file(s))
  • Latency tests for Delete, Download, Metadata & Upload (for 0b,1KiB and 100KiB)

Simply running `gsutil perfdiag -o test.json gs://<your bucket name>` can get you something that looks like this :

The perfdiag command is provided so that customers can run a known measurement suite when troubleshooting performance problems.

What’s next?

Now that we have a clear picture of the proper way to test GCS read/write performance, we can get on with helping some of our developers and their performance problems. Stay tuned to Google Cloud Performance Atlas (YT Video, Medium Blog), and subscribe / follow to get more great content!




DA @ Google; | | | |

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Introduction to Apache Hadoop Ecosystem & Cluster in 2021

Evolution of our Engineering Team’s Structure @ Appsynth

Logging in Java

Guide to Lock Interface in Java

Unit Testing in Python

Unit Testing in Python

The EOSIO StackExchange is Live in Private Beta — Time to get the Q&A going!

Using the Java library Chapter — 6

CSS Animations: A Complete Guide in 7 Levels of Difficulty

CSS Animations: A Complete Guide in 7 Levels of Difficulty

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Colt McAnlis

Colt McAnlis

DA @ Google; | | | |

More from Medium

[CI/CD] Github Deploy to Cloud Run

New Google Cloud Platform cert for database engineers

How to use Copilot to deploy projects on ECS

Google App Engine