Golang | Copy to GCS & Check Bucket

Are you trying to use Go to get files into Google Cloud Storage without pulling them onto the computer that is running the code or open them and read the contents into a new file? If yes, so was I not long ago. Here is a quick PSA to share the code I put together to solve this.

Copy Files to GCS

The following code example can get files from one url location into GCS without downloading it on the server running the code.

import (
    "cloud.google.com/go/storage"
    "fmt"
    "io"
    "net/http"
)
func storeGCS(url, bucketName, fileName string) error {
    // Create GCS connection
    ctx := context.Background()
    client, err := storage.NewClient(ctx)
    // Connect to bucket
    bucket = client.Bucket(bucketName)
    // Get the url response
    if response, err := http.Get(url); err != nil {
        return fmt.Errorf("HTTP response error: %v", err)
    }
    Defer response.Body.Close()
    if response.StatusCode == http.StatusOK {
        // Setup the GCS object with the filename to write to
        obj := bucket.Object(fileName)

        // w implements io.Writer.
        w := obj.NewWriter(ctx)

       // Copy file into GCS
       if _, err := io.Copy(w, response.Body); err != nil {
           return fmt.Errorf("Failed to copy to bucket: %v", err)
       }

       // Close, just like writing a file. File appears in GCS after
       if err := w.Close(); err != nil {
           return fmt.Errorf("Failed to close: %v", err)
       }
    }
    return nil
}

The key line includes io.Copy which is where it will take what is in the HTTP response and copy directly into the bucket. It doesn’t matter what format this file is in because it doesn’t need to read it. It only needs to grab what is in the response and copy it to GCS. I found this useful especially working with zipped files. This is also good for image files but really any file.

This assumes credentials are setup on the machine you are using so it will be able to log into your GCP and load into GCS. Also, the code assumes the bucket exists. Otherwise you need to create a bucket.

Golang Example Create GCS Bucket If Doesn’t Exist

Below is sample code to check if a bucket exists and create it if it doesn’t.

import (
    "cloud.google.com/go/storage"
    "context"
    "fmt"
    "google.golang.org/api/iterator"
    "log"
)
func CreateGCSBucket(bucketName, projectID string) error {
   // Setup context and client
    ctx := context.Background()
    client := storage.NewClient(ctx)
   
    // Setup client bucket to work from
    bucket = client.Bucket(bucketName)

    buckets := client.Buckets(ctx, projectID)
    for {
        if bucketName == "" {
            return fmt.Errorf("BucketName entered is empty %v.", bucketName)
          }
       attrs, err := buckets.Next()
       // Assume bucket not found if at Iterator end and create 
       if err == iterator.Done {
           // Create bucket 
           if err := bucket.Create(ctx, projectID, &storage.BucketAttrs{
            Location: "US",
           }); err != nil {
               return fmt.Errorf("Failed to create bucket: %v", err)
           }
           log.Printf("Bucket %v created.\n", bucketName)
           return nil
        }
        if err != nil {
            return fmt.Errorf("Issues setting up Bucket(%q).Objects(): %v. Double check project id.", attrs.Name, err)
        }
        if attrs.Name == bucketName {
            log.Printf("Bucket %v exists.\n", bucketName)
            return nil
         }
   }
}

The key lines to note is grabbing all the buckets for that project with client.Buckets, the for loop that loops over each bucket name using buckets.Next and confirming the iterator is not at the end, iterator.Done. If it is then create a bucket with bucket.Create but if you find the bucket name provided in the bucket list with attrs.Name == bucketName then you don’t need to create it.

Wrap Up

Above provides two Go code examples which are focused on how to copy files into GCS using Go which boils down to using io.Copy. Also, how to check if a bucket exists already and if not then to create it. An important package to checkout is the Go Storage package for more details on how to interact with GCS as well as Stiface on how to test interacting with GCS.

These code snippets are from a project I’m working on called Project OCEAN (Open Source Community Ecosystem Focus) that is working to model common structures and impacts of technical open source communities. You can view more details about the above code in our GitHub repo.