Using the gsutil command

This page shows you how to perform basic tasks in Cloud Storage using the gsutil command-line tool.

Photo by Nils Schirmer on Unsplash

acl

ACLs (Access Control Lists) grant read or write access to users for buckets or objects. ACL has three commands: GET, SET, or CH.

GET : To lists the permission on a given buckets or objects.

$ gsutil acl get gs://<bucket>
$ gsutil acl get gs://data/path/acl.png

SET : Set the permission on a given buckets or objects.

If you want to define more fine-grained control over your data and avoid mistakes. You can using the get command, export them to a file.

$ gsutil acl get gs://<bucket1>/file1 acl.txt

Next, modify the acl.txt file and then use the set command to set the new permissions that ACL on the buckets and/or objects.

$ gsutil acl set acl.txt gs://<bucket2>/file2
$ gsutil acl set acl.txt gs://<bucket3>

Using the -m option to perform parallel synchronization (multi-threaded).

$ gustil -m acl set acl.txt gs://<bucket4>/*.png

CH : Change the current permission on a given buckets or objects.

To grant all users read permission for the object in bucket1.

$ gsutil acl ch -u AllUsers:R gs://<bucket1>/data/path.png

To grant all users read permission to all objects under the specified URL.

$ gsutil acl -r ch -u AllUsers:R gs://<bucket1>/data/path/

If the bucket have a large number of objects to update. Using the -m option to perform parallel synchronization (multi-threaded).

$ gsutil -m acl -r ch -u Allusers:R gs://<bucket1>

ls

List providers, buckets, or objects.

List all of the Cloud Storage buckets under your project.

$ gsutil ls

List the specified objects.

$ gsutil ls gs://<bucket>/*.png

du

The du command displays the amount of space in the bucket.

To list the size of all objects in a bucket.

$ gsutil du gs://<bucket>

To list a summary of the total bytes in the two given buckets.

$ gsutil du -s gs://<bucket1> gs://<bucket2>

To list the size of all objects in a bucket.

$ gsutil du -a gs://<bucket>

mb

Make buckets. Cloud Storage has a single namespace and must not be already in use by another user.

The mb command creates a new bucket.

$ gsutil mv gs://<bucket>

mv

Move or rename objects.

move : To move data between local file system and Cloud Storage, and move data between Cloud Storage.

To move all objects from bucket to local directory.

$ gsutil mv gs://<bucket>/* /local/dir

To move all objects from local directory to bucket.

$ gsutil mv ./local/dir gs://<bucket>

rename : To rename all objects with a given prefix to have a new prefix.

Cloud Storage namespace is flat, that doesn’t support true folders. The meaning of / is really imposed by users. When you rename a folder, it’s accomplished by copying each object to a new object with the desired name while deleting the old one.

$ gsutil mv gs://<bucket>/<old_prefix> gs://<bucket>/<new_prefix>

Preserving ACL permissions for mv, you should use the -p option.

$ gsutil mv -p gs://<bucket>/<old_prefix> gs://<bucket>/<new_prefix>

cp

The cp command is used to copy data between local file system and Cloud Storage.

Upload file from local file system.

$ gsutil cp /local/file gs://<bucket>/

Download file from Cloud Storage.

$ gsutil cp gs://<bucket>/file /local/path/

Transfer a file between buckets.

$ gsutil cp gs://<bucket1>/file gs://<bucket2>/

Create a new folder in a bucket.

$ gsutil cp <folder_name> gs://<bucket>/

rsync

Synchronize content of two buckets/directories.

To copy only new/changed files without deleting extra files.

$ gsutil rsync -r /data/path gs://<bucket>

To make the contents of bucket1 the same as bucket2.

$ gsutil rsync -d -r gs://<bucket1> gs://<bucket2>

To perform parallel synchronization (multi-threaded).

$ gsutil -m rsync -r -d /data/path gs://<bucket>

Please exercise caution when you use -d option. This option will make the contents of /data/path the same as bucket. If you erroneously reverse source and destination. It’s possible to delete large amounts of data accidentally.

lifecycle

Get or set lifecycle configuration allow you to automatically delete or change the storage class.

get : Gets the lifecycle configuration for a given bucket.

$ gsutil lifecycle get gs://<bucket>

set : Sets the lifecycle configuration for one or more buckets with setting defined in the *.json file.

In this config.json file, more than 30 days old will be deleted automatically.

$ cat config.json
{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "Delete"},
        "condition": {"age": 30}
      }
    ]
  }
}

To enable lifecycle for a bucket.

$ gsutil lifecycle set config.json gs://<bucket>

Changing storage class of objects to Nearline after one year. If Nearline older than two years that changing to Coldline.

$ cat config.json
{
  "lifecycle": {
    "rule": [
      {
        "action": {
          "type": "SetStorageClass",
          "storageClass": "NEARLINE"
        },
        "condition": {
          "age": 365,
          "matchesStorageClass": ["MULTI_REGIONAL", "STANDARD", "DURABLE_REDUCED_AVAILABILITY"]
        }
      },
      {
        "action": {
          "type": "SetStorageClass",
          "storageClass": "COLDLINE"
        },
        "condition": {
          "age": 730,
          "matchesStorageClass": ["NEARLINE"]
        }
      }
    ]
  }
}