Codebrace
Published in

Codebrace

Object Versioning for Google Cloud Storage!

Using Object Versioning in Google Cloud Storage

Usecase

Suppose we have a lot of data in our Cloud Storage bucket and somehow by mistake someone runs

gsutil rm gs://my_bucket/*,

we will lose all our data and won’t be able to recover it easily or may never be able to recover it.

How does Object Versioning Help?

By Design, every storage object (file) in Cloud Storage is assigned 2 sequence numbers

  • generation number
  • meta-generation number

we will talk about them in detail later,

In a Nutshell, a generation number will be assigned each time we replace an object or modify it. similarly, a meta-generation number will be assigned to an object each time we modify the meta-data.

By default, object versioning is disabled as it incurs more cost as we store multiple versions of the same object with different generation and meta-generation numbers, but if we need the ability to recover old data we can leverage object versioning.

Enabling Object versioning

  • We have just created a new bucket ashish_vtest having 2 files log.txt and Main.java
  • we can check if the Object versioning is enabled on a bucket/folder
    status can be Suspended or Enabled.
gsutil versioning get gs://ashish_vtest
checking object versioning status

Now Let’s enable the versioning

gsutil versioning set on gs://ashish_vtest
enabling object versioning

Checking Object Versions

  • We can use gsutil ls -a gs://<path> to check all the files object versions. ( all the files including old version and current ones )
checking all object versions
  • we can see there is number after file names prefixed by #, this number is called generation number.
  • we can access any non-current file( old versions ), using full name of the files ( name + generation number )

Deleting and recovering a file

  • Now, let's do some real work first we will delete log.txt and then recover it using the generation number.
Deleting a file
  • as we can see there is only one current file, which we can check using gsutil ls gs://<path>
  • but if we check all the Object versions we will still see 2 files
  • Now, let's recover the log.txt and put it inside the same location.
recovering deleted Object version
  • As you can see, we are just copying files and putting them inside the same directory, note that for using the non-current file we will have to use the file name and generation number together.
  • we can see there are 2 current files but there will be 3 versions as a new version will be created when we copy.

Generations and Meta-Generation Number

  • Even without Object Versioning enabled, all Cloud Storage objects have generation numbers and meta-generation numbers. The generation number changes each time the object is replaced, and the meta-generation number changes each time the object’s metadata is updated.
  • Buckets maintain a meta-generation number enabling users to uniquely identify a bucket metadata state.
  • we can check meta-generation number using -la flags in gsutil ls -la gs://<path>
checking metageneration Numbers
  • meta-generation number starts from 1 and increases as we update the metadata state of an Object.
  • let’s update meta-data for log.txt and check the meta-generation number.
  • We can edit metadata directly via UI or we can use CLI refer — https://cloud.google.com/storage/docs/viewing-editing-metadata#view
  • from 3 dots on right side of an object on GCP UI, we can edit metadata
Editing meta-data
  • Now, if we check, we will see 2 meta-generation numbers for log.txt
    If we want to access file with specific meta-data, we will have to use the generation number if that version is not the current one.

Disabling Object versioning

  • we can disable object versioning using gsutil command
gsutil versioning set off gs://ashish_vtest

If you like this article, please follow me and this publication for more interesting articles, a clap will be really appreciated..

#codebrace #happy_coding

--

--

--

Coding blog to help people get going with Competetive programming, Big Data and other technologies, visit http://medium.com/codebrace

Recommended from Medium

Leetcode 2133: Check if Every Row and Column Contains All Numbers

Deploying Sapper PWA using Github Pages: Step by Step Tutorial ( Part 1)

Kubernetes Logging with Fluent Bit, Elasticsearch and Kibana

Design a simple JAX — WS (RPC) java web service

FIBONACCI PRIME COMPOSITE USING C++!!!

The 3 pillars of modern web development: Node.js, PHP, or Java?

October 14th

Editing Images In 8-bit vs. 16-bit Color

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ashish Patel

Ashish Patel

Big Data Engineer at Skyscanner , loves Competitive programming, Big Data.

More from Medium

Configure notification for new/updates available for AWS EKS Add-ons

Manually Installing an ElasticSearch Cluster on GCP

What to remember if you decide to ingest logs using logging agent in Google Cloud

Create your own API in Google Cloud with Terraform