Set up Serverless Store: Part 3 (Computing, Cloud Build, Cron Jobs, and Management Tools)

This document is a part of the Serverless on Google Cloud Platform: an Introduction with Serverless Store Demo How-to Guide. It discusses the computing, cron jobs, and management tool solutions Serverless Store uses.

Before you begin

• Create a Google Cloud Platform project with billing enabled and install Google Cloud SDK. See the opening piece for instructions.
• Download the project.

Computing and Cloud Build

Now with all the products and services set up you are ready to deploy the app and the functions.They are deployed to Google App Engine and Google Cloud Functions respectively. App Engine is a managed serverless platform for highly scalable applications, and Cloud Functions is event-driven serverless platform for small, independent units of code. To learn more, see App Engine Documentation and Cloud Functions Documentation.

Serverless Store consists of one App Engine app and 6 Cloud Functions. The easiest way to deploy them is to use Google Cloud Build. Cloud Build lets you build software quickly across all languages; you can set up a custom workflows for building, testing, and deploying across multiple environments such as VMs, serverless, Kubernetes, or Firebase.

If you prefer not using Cloud Build, you may use Cloud SDK to deploy the artifacts one by one instead; for more information, see Deploying Serverless Store without using Cloud Build.

First, enable Cloud Build API in your project:

• Open Cloud Build panel in Google Cloud Console.
• Click Enable API.

At this moment, the Serverless Store project uses placeholder configurations. Cloud Build can help you replace them with your project-specific values; in this tutorial, you will upload those custom configurations in a Cloud Storage bucket:

• Open extras/cloudbuild.
• Edit app_main.yaml and replace YOUR-PROJECT, YOUR-NEW-PRODUCT-TOPIC, YOUR-PAYMENT-TOPIC, YOUR-GCS-BUCKET with values of your own. This is the App Engine configuration file for the main app. If you have followed the naming patterns in this setup guide, your app.yaml should look this:

• Edit the app_stream_events.yaml file. This is the App Engine configuration file for event streaming service. Replace YOUR-DATASET and YOUR-TABLE with values of your own; if you have followed the naming patterns in this setup guide, the two values should be sample_data and sample_table respectively.
• Edit the function_env_vars.yaml file. This YAML file specifies a collection of environment variables used in your Cloud Function deployments. Replace YOUR-AUTOML-MODEL-ID, YOUR-GCS-BUCKET, YOUR-STRIPE-API-KEY, YOUR-PAYMENT-COMPLETION-TOPIC, and YOUR-SENDGRID-API-KEY with values of your own. If you have followed the naming patterns in this setup guide, YOUR-PAYMENT-COMPLETION-TOPIC should be payment-completion.
• Copy static/initFirebase.js and static/stripe.js to extras/cloudbuild. The two static files have been updated in previous steps.
• Create a new Cloud Storage bucket named YOUR-PROJECT-ID-build. Replace YOUR-PROJECT-ID with the value of your own.
• Open gcsFetcherManifest.yaml. Find and replace all occurrences of YOUR-PROJECT-ID with the value of your own. gcsFetcherManifest.yaml essentially tells Cloud Build where to fetch your project specific configurations.
• Now, open the bucket and click Upload folder. Select extras/cloudbuild. Folder cloudbuild will be uploaded to the bucket, which includes all of your project specific configurations.

Go back to the root folder. cloudbuild.yaml specifies each step Cloud Build takes to build and deploy your app. In this tutorial, it takes Cloud Build 10 steps to deploy everything:

1. Step #1: Clones the project from GitHub using git.
2. Step #2: Replaces placeholder configurations with those of your own using gcs-fetcher.
3. Step #3: Deploys the main App Engine app using gcloud.
4. Step #4: Deploys the sidecar App Engine service for streaming events to Google BigQuery using gcloud.
5. Step #5–10: Deploys the Cloud Functions with environment variables specified in function_env_vars.yaml using gcloud.

To trigger a build, run command

gcloud builds submit --config cloudbuild.yaml

Give Cloud Build a few minutes to get things done. You can view the progress via the link in the output.

Your app is now available at YOUR-PROJECT-ID.appspot.com. You can view all the App Engine services via App Engine Services page in Google Cloud Console. Before you open the app, there are two more things to do:

• Open Firebase Console and go to your project.
• Go to the Authentication page.
• Go to the Sign-in method tab.
• Click Add domain and whitelist YOUR-PROJECT-ID.appspot.com.
• Open the Cloud Pub/Sub page in Google Cloud Console.
• Click the new-product Pub/Sub topic.
• Click Create Subscription.
• Type in the name of the subscription.
• Pick the Push into an endpoint URL delivery type.
• Use the URL stream-events-dot-YOUR-PROJECT-ID.appspot.com/stream as the URL. Replace YOUR-PROJECT-ID with the value of your own.
• Click Create. The App Engine service you deploy earlier is now a subscriber of the new-product topic and will stream all the incoming events to Cloud BigQuery.
• Repeat the steps above for topics payment-process and payment-completion.

Now, open the app and try signing in with your Google account, adding some items, purchasing an item, and watch the events flow via Google Data Studio.

Cron Jobs

Google Cloud Platform offers a product, Google Cloud Scheduler, for scheduling tasks to run periodically, either on Google Cloud Platform or on a local server. You can, for example, ask Cloud Scheduler to run a Cloud Function every week to remind people of the items they left in the cart in Serverless Store.

To set up this task with Cloud Scheduler:

• Open a terminal and switch to functions/sendReminder. Function sendReminder in index.js checks Cloud Firestore, finds all the items added more than one week ago, and send reminder emails to the associated accounts. If you use Cloud Build in the earlier step, the Cloud Function has been deployed; to deploy this function manually, run command:

• Open Cloud Scheduler page in Google Cloud Console. Enable the API if prompted.
• Click Create Job.
• Type in the name of the job, and 0 9 * * 1 as the frequency. This job will run on every Monday at 9:00 in the specified timezone.
• Choose HTTP for target.
• Type in the URL of the deployed Cloud Function as the URL. It should be available in the output of the terminal; you can also view it on the Cloud Functions page.
• Choose GET for HTTP method.
• Click Create.
• Click Run now to try the task out. You may want to edit the data in Cloud Firestore and manually create some out-of-date cart items.

Management Tools

Google Cloud Platform offers a variety of tools to help you manage your deployments inside and outside Google Cloud Platform. You can use Stackdriver Logging to collect logs, Stackdriver Monitoring to get real-time updates about your app, Stackdriver Trace to identify performance bottlenecks in your code, and many more. See here for a full list of Google Cloud Platform Management Tools.

Stackdriver Logging

Logs from Serverless Store are automatically collected in Stackdriver Logging.

To view the logs from the flask app:

• Open the App Engine Services page in Google Cloud Console.
• Click Tools > Logs.

To view the logs from the Cloud Functions:

• Open the Cloud Functions page in Google Cloud Console.
• Press the context button (three vertical dots) in the row of the Cloud Function you would like to manage and click View Logs.

You can view, search and group the logs. They can also be exported to Cloud Storage and Google BigQuery. See Stackdriver Logging Documentation for more information.

Stackdriver Monitoring

To view the status of your deployments, click Monitoring in the left side bar of the Cloud Console. You will be redirected to Stackdriver Console, where you can check the metrics of the Serverless Store app and related functions.

See App Engine Metrics and Cloud Functions Metrics for a complete list of metrics you can monitor.

Stackdriver Trace

Stackdriver Trace allows developers the check how long one specific block of code takes to run, and helps identify the slowest link in an execution path. App Engine deployments have Stackdriver Trace enabled by default; you can view the times App Engine takes to serve each request in the Stackdriver Trace page of the Cloud Console. Alternatively, you can also create custom traces non-specific to requests; Serverless Store, for example, includes a custom trace that tracks the three steps of payment processing: order preparation, event streaming, and payment processing:

See app/blueprints/charge/blueprint.py for the full source code.

See functions/pay_with_stripe/main.py for the full source code.

Note that the custom trace spans across two different products (payment processing happens in a Cloud Function) as if it happens inside the same deployment. Every time a customer makes a payment, you can view the trace in Cloud Console and see which step takes the most time to finish, and make optimizations accordingly to create a better user experience.

To view the custom trace:

• Open the Stackdriver Trace page. Click Trace List.
• There might be many traces available. Type in prepare_order_info, the name of the first step in the custom trace, to filter the list. If no data is available, make sure you have chosen the right time window on the top right of the screen.
• Click any data point in the graph. You should be able to see the length of each step and the length of the entire trace.

What’s next

See Further Discussion for some tips and notes about Serverless Store.