Using pure Golang for Google cloud

Frikky
Frikky
Jun 4 · 9 min read

For the past half year I’ve been playing around with Google cloud’s options for development and deployment using pure Golang. I soon realized it’s actually pretty hard to use these API’s, at least as a first time user, and want to share parts of why and with what I’ve been struggling. I’ll be showing some examples and talk about the annoyances of building with Golang for Cloud run (new), Cloud tasks (new ish) and Datastore.

These issues have been experienced every time I want to try a new GCP API, which when you use any cloud platform in general, is quite often. My struggles have been due to a lack of examples and godoc understanding, and to make matters worse, the code feels really convoluted. Before moving on, I expect you to have already set up authentication properly.

To get started, here’s a simple PUT and GET from Datastore, with the bare essentials for both. Since Datastore is a document storage solution (dictionaries, JSON), we’ll just upload a simple JSON object. For all the following examples you’ll have to define the projectID variable, which is your GCP project name. Examples here!

package main

import (
"cloud.google.com/go/datastore"
"context"
"encoding/json"
"log"
)

// Create an item we're gonna put in and remove
// Uses datastore and json tags to map it directly
type Data struct {
Key string `datastore:"key" json:"key"`
}

// Puts some data from the struct Data into the database
func putDatastore(ctx context.Context, client *datastore.Client, dbname string, data Data) error {
// Make a key to map to datastore
datastoreKey := datastore.NameKey(dbname, data.Key, nil)

// Adds the key described above with the
// data from datastoreKey
if _, err := client.Put(ctx, datastoreKey, &data); err != nil {
log.Fatalf("Error adding testdata to %s: %s", dbname, err)
return err
}

return nil
}

// Gets the data back from the datastore
func getDatastore(ctx context.Context, client *datastore.Client, dbname string, identifier string) (*Data, error) {
// Defines the key
datastoreKey := datastore.NameKey(dbname, identifier, nil)

// Creates an empty variable of struct Data, which we map the data back to
newdata := &Data{}
if err := client.Get(ctx, datastoreKey, newdata); err != nil {
return &Data{}, err
}

return newdata, nil
}

func main() {
// Describe the project
projectID := "yourprojectnamehere"
dbname := "medium-test"

ctx := context.Background()

// Create a client
client, err := datastore.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Failed setting up client")
}

// Create som json data to map to struct
jsondata := `{
"key": "qwertyuiopasdfghjkl"
}`

// Map the jsondata to the struct Data
var structData Data
if err := json.Unmarshal([]byte(jsondata), &structData); err != nil {
log.Fatalf("Failed unmarshalling: %s", err)
}

// Puts the data described above in the datastore
if err := putDatastore(ctx, client, dbname, structData); err != nil {
log.Fatalf("Failed putting in datastore: %s", err)
}

// Gets the same data back from the datastore
returnData, err := getDatastore(ctx, client, dbname, structData.Key)
if err != nil {
log.Fatalf("Failed getting from datastore: %s", err)
}

// Print with some extra value
log.Printf("%#v", returnData)
}

As for datastore, it has an ok API and is quite simple to use. You create a client, make a struct to map the data in, and you’re essentially there. This sample has most of whatever you’ll need.

Moving on, lets have a look at a little more annoying example: Cloud tasks. I developed these functions before any examples were available, meaning it might not be 100% accurate with live. (SEE: apiv2beta3). I used way more time than I feel I should’ve been understanding this API, but the good part is that it taught me how to read and trace godoc pretty better, which has been handy in general.

The code makes a client, defines a project location (which task name to target), creates a single task, and then counts the amount of tasks. PS: Tasks are autodeleted after the hook, meaning you might have to stop it to test the iterator.

package main

import (
cloudtasks "cloud.google.com/go/cloudtasks/apiv2beta3"
"context"
"fmt"
"google.golang.org/api/iterator"
taskspb "google.golang.org/genproto/googleapis/cloud/tasks/v2beta3"
"log"
)

func createTask(ctx context.Context, client *cloudtasks.Client, parent string) {
// Define some endpoint you want the data to hit from
url := "/api/test"

// Nested structs. Just mapped them like this so it's actually readable
var appEngineHttpRequest *taskspb.AppEngineHttpRequest = &taskspb.AppEngineHttpRequest{
HttpMethod: taskspb.HttpMethod_GET,
RelativeUri: url,
}

var appeng *taskspb.Task_AppEngineHttpRequest = &taskspb.Task_AppEngineHttpRequest{
AppEngineHttpRequest: appEngineHttpRequest,
}

var task *taskspb.Task = &taskspb.Task{
PayloadType: appeng,
}

// Structs added into the last struct which creates the task
req := &taskspb.CreateTaskRequest{
Parent: parent,
Task: task,
}

ret, err := client.CreateTask(ctx, req)
if err != nil {
log.Printf("Error creating task: %s", err)
return
}

log.Printf("%#v", ret)

}

func listAllTasks(ctx context.Context, client *cloudtasks.Client, parent string) {
// Makes a struct to map
req := &taskspb.ListTasksRequest{
Parent: parent,
}

// Returns an iterator over the parent tasks and counts
ret := client.ListTasks(ctx, req)
cnt := 0
for {
_, err := ret.Next()

if err == iterator.Done {
break
}

if err != nil {
log.Printf("Error in iterator: %s", err)
break
}

cnt += 1
}

log.Printf("Current amount of tasks: %d", cnt)
}

func main() {
// Define the client
ctx := context.Background()
client, err := cloudtasks.NewClient(ctx)
if err != nil {
log.Fatalf("Error creating cloudtask client: %s", err)
}

// Set the projectId, location and queuename for the specific request
projectID := "yourprojectnamehere"
location := "europe-west3"
queuename := "myqueue"
var formattedParent string = fmt.Sprintf("projects/%s/locations/%s/queues/myqueue", projectID, location, queuename)

// Creates a task
createTask(ctx, client, formattedParent)
listAllTasks(ctx, client, formattedParent)
}

Now, wasn’t that easy? Well.. Not really. This is where the hard_to_grasp internals of the google cloud code structure is really making it a hassle for firsttimers. An example of something I personally found really stupid would be directed at the four nested structs, that are there simply to define a GET request and an endpoint. I understand that there is some depth to the appengine integration here, but come on..

Before moving on, let’s add a Docker image to the Container Registry on GCP. Skip this step if you have a webserver docker image ready which listens based on the enviornment variable “PORT”. (Yes, I’m aware of the hypocrisy, but I don’t really want to use another day just to push a Docker image.. (I’ll get back to this eventually)

git clone https://github.com/frikky/medium-examples
cd medium-examples/gcloud/webhook
# Set the "projectname" variable in gcp_run.sh
vim gcp_run.sh # ..
# Run the script to build and deploy the webserver
./gcp_run.sh
# run.sh can will run the same file locally

As for the last part, Cloud Run, I was excited to see Jaana’s blogpost after the release a little while back, hoping for some real Golang specific examples. I was sad to see (like most blog posts out there) the fallback is the use of gcloud CLI, and not just native Go code (I understand this is for a broader audience :)). I like gcloud as much as the next person, but I personally don’t make my platform integrations in bash. Anyway, here is a snippet that creates a new Cloud Run service for an already existing docker image. It also has a function to get a service. Explanation of how I got here is below the code. I want to emphasize that way more time than a task like this should’ve.

package main

import (
"context"
"fmt"
cloudrun "google.golang.org/api/run/v1alpha1"
"log"
)

func getAllLocations(projectsLocationsService *cloudrun.ProjectsLocationsService) ([]string, error) {
// List locations
// Make a request, then do the request
list := projectsLocationsService.List(fmt.Sprintf("projects/shuffle-241517"))
ret, err := list.Do()

if err != nil {
log.Println(err)
return []string{}, err
}

locationNames := []string{}
for _, item := range ret.Locations {
locationNames = append(locationNames, item.Name)
}

return locationNames, nil

}

// https://cloud.google.com/run/docs/reference/rest/
func main() {
ctx := context.Background()

// Defines a the projectname, the servicename to use and an existing image to use
projectId := "yourprojectnamehere"
imagename := "yourimagenamehere"
servicename := "webhook2"

// Create a service client like anywhere else
apiservice, err := cloudrun.NewService(ctx)
if err != nil {
log.Fatalf("Error creating cloudrun service client: %s", err)
}

// Gets all available locations
projectsLocationsService := cloudrun.NewProjectsLocationsService(apiservice)
allLocations, err := getAllLocations(projectsLocationsService)
if err != nil {
log.Fatalf("Error getting locations: %s", err)
}

// Define the service to deploy
// Wtf even is this
// Metadata initializers
// SOO MANY LAYERS OF BULLSHIT (:
tmpservice := &cloudrun.Service{
ApiVersion: "serving.knative.dev/v1alpha1",
Kind: "Service",
Metadata: &cloudrun.ObjectMeta{
Name: servicename,
Namespace: projectId,
},
Spec: &cloudrun.ServiceSpec{
RunLatest: &cloudrun.ServiceSpecRunLatest{
Configuration: &cloudrun.ConfigurationSpec{
RevisionTemplate: &cloudrun.RevisionTemplate{
Metadata: &cloudrun.ObjectMeta{
DeletionGracePeriodSeconds: 0,
},
Spec: &cloudrun.RevisionSpec{
Container: &cloudrun.Container{
Image: imagename,
Resources: &cloudrun.ResourceRequirements{
Limits: map[string]string{"memory": "256Mi"},
},
Stdin: false,
StdinOnce: false,
Tty: false,
},

ContainerConcurrency: 80,
TimeoutSeconds: 300,
},
},
},
},
},
}
//Env: []*cloudrun.EnvVar{
// &cloudrun.EnvVar{
// Name: "PORT",
// Value: "8080",
// },
// },

// Deploy the previously described service to all locations
// Locations are the same as "parent" in other API calls, AKA:
// projects/{projectname}/locations/{locationName}
for _, location := range allLocations {
getService(projectsLocationsService, location)
createService(projectsLocationsService, location, tmpservice)
}
}

func getService(projectsLocationsService *cloudrun.ProjectsLocationsService, location string) {
projectsLocationsServicesGetCall := projectsLocationsService.Services.Get(fmt.Sprintf("%s/services/webhook", location))

service, err := projectsLocationsServicesGetCall.Do()
if err != nil {
log.Fatalf("Error creating new locationservice: %s", err)
}

_ = service
}

func createService(projectsLocationsService *cloudrun.ProjectsLocationsService, location string, service *cloudrun.Service) {
projectsLocationsServicesCreateCall := projectsLocationsService.Services.Create(location, service)
service, err := projectsLocationsServicesCreateCall.Do()
log.Println(service, err)
if err != nil {
log.Fatalf("Error creating new locationservice: %s", err)
}

log.Printf("%#v", service.Spec)
}

Now, that might not seem too bad, but to manage the creation of this monstrosity, I had to make it backwards by creating the Get call first. I initially started building it backwards from the godocs, looking for references to “locations.services.create” found here (the REST api doc is _actually_ pretty neat). After building the structure and testing the API calls, I had to build the “Service” struct, which seemed doable (it’s large and horribly convoluted) until I found that the API didn’t tell me WHAT was wrong when my APIcalls failed. This means I have to go on a witch hunt for invalid or missing variables within seven (yes, really) nested structs to find the required or missing variables.

Horrible error code
Notes while trying to understand and build the structure

So to further debug what was required, I had a look at the logging utility in Google cloud, and was happy to find that they have ok audit logging. The real problem though, is that the audit logs don’t tell you anything either.

My issues and anger kept piling up, and I even went to the point of reverse engineering the frontend API calls, walking through required fields when creating a new cloud run service, but to no avail.

The CREATE function calls when creating in the frontend.

I soon remembered that I totally forgot about a “simple” APIcall that I should’ve thought of earlier. There exists a GET statement for webhooks as well. I built the GET statement, and soon developed the following struct based on walking all fields of run.Service returned from the Get Call.

The finished cloudrun example struct

And finally, after all that struggle, I got the long awaited 200 response. Here’s how it looks in the GUI (Yes, I know I’m amazing at censoring).

Now, what’s the real issue here? Is it that I’m too dumb to understand the API, the lack of examples, or that it’s in alpha? Well.. no, not really. I think there are multiple reasons to my struggles here, with the main one being that it’s simply too convoluted to understand first hand, which in and of itself is it’s own pitfall. I haven’t done this in other languages, but the JSON behind all that has to be built in any language, so it depends on the wrappers, but with code generation being a big theme in their libaries, I can’t imagine them being much better.

I find it troubling that I used a full day of work just to understand how a Struct should be built.

So there it is, some fully working examples with some context to them. One easily usable, one a little more tricky with really weird definitions without good samples, and one stupidly complex because of the layered code. Here’s the source for all three of them.

Again, the point here isn’t to shit on Google cloud as I love their services and use them daily, but I would like to share my frustration and learnings for features and services that I feel like are missing required documentation for proper implementations. I’m part of the problem as well, as don’t usually take the time to do pull requests every time I finish something I haven’t found samples for either. (They don’t accept entirely new samples in the GoogleCloudPlatform/golang-samples repo anyway)

Happy serverless coding :)

The Startup

Medium's largest active publication, followed by +481K people. Follow to join our community.

Frikky

Written by

Frikky

automating all the things

The Startup

Medium's largest active publication, followed by +481K people. Follow to join our community.