Set up Serverless Store: Part 2 (Machine Learning/API, Data Analytics, and Data Visualization)

This document is a part of the Serverless on Google Cloud Platform: an Introduction with Serverless Store Demo How-to Guide. It discusses the machine learning/AI, data analytics, and data visualization solutions Serverless Store uses.

Before you begin

• Create a Google Cloud Platform project with billing enabled and install Google Cloud SDK. See the opening piece for instructions.
• Download the project.

Machine Learning/AI

Serverless Store uses three Google Cloud Platform machine learning/AI solutions:

• Google Cloud Vision API
• Google Cloud AutoML Vision
• DialogFlow for Google Assistant integration (Actions on Google)

Google Cloud Vision API

Google Cloud Vision API is an image analysis tool using a machine learning model pre-trained by Google. You can try the API out here. Serverless Store uses Cloud Vision API to label product images.

To use Cloud Vision API:

• Enable the API.

Understanding the code

When a new product is added, the flask app publishes an event to the new-product topic. Cloud Function detect_labels (functions/detect_labels/main.py) subscribes to the topic, listens to the events, and call the Cloud Vision API to analyze the image of new products. Returned labels from Cloud Vision API are then written to Cloud Firestore.

See functions/detect_labels/main.py for the full source code.

Google Cloud AutoML Vision

Cloud Vision API uses a pre-trained model; for labeling tasks, the API always return a subset of labels predefined by Google. Useful as it is, many use cases require a more precise, or specific, solution, hence the creating of Cloud AutoML Vision. Cloud AutoML Vision allows developers to train a custom model for image classification and integrate it in their apps. Little machine expertise is required; Cloud AutoML Vision takes labeled images and handles everything else automatically behind the scenes.

Serverless Store uses Cloud AutoML Vision to train a model that tells pet products apart from other products, and promotes all the pet products automatically on the front page. To set up AutoML Vision for Serverless Store:

• Find some pet product and regular product images. AutoML Vision recommends using at least 100 images for each category; however you may use a much smaller dataset just to try things out.
• Click Get started with AutoML.
• Click Set up now.
• Click New Dataset.
• Type in pet_recommendations as the dataset name.
• Choose Import images later.
• Click Create Dataset.
• Click Add label. Add two labels, not_for_pets and pets.
• Click Add Images > Upload from your computer.
• Upload all the pet product images. It may take a while for AutoML to process the data.
• Click category Unlabeled. Select all images.
• In the Label drop-down menu, select pets.
• Repeat the steps and upload regular product images. Assign them the label not_for_pets.
• Now AutoML has all the information to train the model. Select the Train tab and click Start Training. It takes a while to train the model.
• Now you have a model capable of identifying pet products. You can check the details of the model in the Evaluate tab and try the model in the Predict tab. Write down the model ID in the Predict tab. It looks like this: ICN000000000000000000.

Understanding the code

Cloud Function automl (functions/automl/main.py) is responsible for predicting images using the model. It also subscribes to the new-product topic, and analyze the image of the new product when events arrive.

See functions/automl/main.py for the full source code.

DialogFlow (for Google Assistant integration)

DialogFlow is a Google provided solution for building natural and rich conversational experiences. In Serverless Store you will use DialogFlow to add a simple custom action to Google Assistant so that people can check all the trending items on sale interactively via conversation.

To set up the trending items Google Assistant action for Serverless Store:

• Go to the Activity Control page. Sign in with your Google account and make sure you have enabled the following permissions: Web & App Activity, Device Information, and Voice & Audio Activity.
• Go to the Actions on Google Console. Actions on Google allows you to register an action for Google Assistant and other Google solutions.
• Click Add/Import Project.
• Type in a project name.
• Click Create Project.
• Skip the welcome page.
• In the left side bar, click Build > Actions. Then click Add Your First Action.
• Select Custom Intent and click Build. This tells Actions on Google console that you would like to use DialogFlow to create the action.
• DialogFlow Console will open in a new window/tab. Type in the name of the action (Agent name) . Click Create. You will be redirected to the Intents page.
• Intent is, loosely speaking, a conversational round. It specifies an event or a sentence that starts a round (“What’s up, my assistant?”), and a response that concludes the conversation (“Everything’s good, thanks!”) or triggers the next round (“I’m good, what can I help you today?”). DialogFlow also allows developers to extract information automatically from sentences (“I want to book a flight from New York to Shanghai.”). At this moment there should be two intents on screen. Default Welcome Intent is the default starting point of a conversation and Default Fallback Intent is what Google Assistant will use when it does not know what to say. Click Default Welcome Intent.
• You will see, in the new page, a collection of training phrases. DialogFlow uses Natural Language Processing technologies to learn from these phrases; users can speak any sentence close to the training phrases in meaning to start a conversation. If no training phrase is present, Google Assistant will lead the conversation instead. For this demo, we would like Google Assistant to speak up first; as a result, you will have to remove all the training phrases.
• Next, add a custom response. Remove all the defaults ones, and add “Hi Serverless Store here. What can I help you today?”
• Click Save.
• Go back to the Intents page. Click Create Intent to add another round of conversation to follow up with the “Hi! Serverless Store here. What can I help you today? prompt.
• Type in Trending Items as the intent name.
• Click Add Training Phrases. Type in “What’s trending in Serverless Store?”. Press Enter.
• Since trending items are changing all the time, you will need a dynamic response. Click Fulfillment, then Enable Fulfillment. Turn on Enable webhook call for this intent. This settings allows DialogFlow to prepare a response using a webhook, such as a Cloud Function.
• Click Save.
• Open the left side bar and go to the Fulfillment page.
• Enable Inline Editor. You will deploy a Cloud Function directly from the DialogFlow Console. Alternately, you may also deploy it manually and set it up in the Webhook section.
• Delete the sample code in the editor. Copy all the lines in functions/dialogFlow/index.js here.
• Click Deploy.
• Now, speak to the Google Assistant app on your phone or your Google Home devices: “Talk to my test app.” If you don’t have Google Assistant available, use the Simulator in Actions on Google Console. The conversation should sound like this:

Understanding the code

Cloud Function functions/dialogFlow/index.js is a simple code snippet using the DialogFlow SDK to prepare responses. At this moment it returns nothing but one fixed response; you, however, can easily modify it to prepare sentences using product names in your Cloud Firestore.

See functions/dialogFlow/index.js for the full source code.

Data Analytics and Data Visualization

As introduced in the opening piece, events delivered via message queues such as Cloud Pub/Sub are perfect sources for real-time data analytics. You can, for example, stream these events to Cloud Dataflow, where event contexts are extracted and analyzed in real-time, and save the insights to Google BigQuery, a data warehousing solution for future reference at ease without ever interrupting the normal operations of your app.

In Serverless Store, for simplicity reasons, you will send these events directly to BigQuery instead and use Google Data Studio to create a dynamically updated data report. The report prepared can help you easily track the number of orders and payments processed by the app.

Google BigQuery

Capable of storing hundreds of PBs of data and more, Google BigQuery allows developers and data analysts to query massively large dataset fast and efficiently. For this demo, however, you will only use a small subset of its features. For more information on BigQuery, see Google BigQuery Documentation.

To set up Google BigQuery for Serverless Store:

• Open the BigQuery page in Google Cloud Console. Enable the API if prompted.
• Click Create Dataset.
• Type in sample_data as the Dataset ID.
• In the left side bar, expand your project and select the newly created dataset.
• Click Create Dataset.
• Click Create Table.
• Type in sample_table as the Table Name.
• Edit the schema and add the following fields:

• Click Create Table.

Understanding the code

An App Engine service, extras/streamEventsApp/server.js accepts events from all the Pub/Sub topics you create earlier and save them in BigQuery:

See extras/streamEventsApp/server.js for the full source code.

If you have you own domain, it is also possible to use a Cloud Function for streaming events into BigQuery. See functions/streamEvents for details.

Google Data Studio

Google Data Studio gathers data from a variety of sources, including Google BigQuery, and helps developers and data analysts build interactive visual data report/dashboard. With all the events saved to BigQuery, in Serverless Store you can create a dashboard updated in real-time with Google Data Studio.

To set up Google Data Studio with Serverless Store:

• Open Google Data Studio.
• Click the Add Button (+) at the bottom right of the screen.
• Click Create New Data Source.
• Select BigQuery.
• Find your Google Cloud Platform project, select the sample_data dataset, and the sample_table table.
• Click Connect.
• You will see a number of fields you create in the last step. Click Add to Report and confirm.
• Click Add a Chart. For this demo you can pick a standard bar chart.
• By default it shows the record count, which is 0 at the moment, with the context dimension.
• In the right panel, remove the context dimension and add the eventType dimension. Now the bar chart will display the number of events in each event type.

What’s next

Continue the setup in Set up Serverless Store: Part 3.