Share and Secure your Cloud Services: An Introduction to Cloud Endpoints

You are a data scientist or a developer and you want to securely share a model or a Web Service, let’s get started!

Photo by Eric Ward on Unsplash

Solution Overview

To demonstrate how to share and secure a cloud service, let’s take an example of a solution to extract data from RSS News Feeds (see previous post) that we want to expose to end users. Basically, we want to:

• Allow authorized users to start a search-and-extract background process using a secured API,
• Allow authorized users to get access to articles using a second secured API.

These functionalities are implemented by two Cloud Functions :

1. publish_rss_feed_search_message(): triggers a Search-and-Extract Process by publishing an event message to a Cloud Pubsub Topic that will start a background process involving multiple steps.
2. read_news_articles(): returns news articles from Cloud Firestore database.

Overview Solution to Share and Secure Cloud Services

Cloud Endpoints: How does it work?

“Endpoints is an API management system that helps you secure, monitor, analyze, and set quotas on your APIs using the same infrastructure Google uses for its own APIs. After you deploy your API to Endpoints, you can use the Cloud Endpoints Portal to create a developer portal, a website that users of your API can access to view documentation and interact with your API.” [Source]

Cloud Endpoints can be used to expose different Cloud Services like Cloud Function, App Engine, Kubernetes, Compute Engine (Docker), etc.

We will present here how to create a secure API Endpoint in order to expose Cloud Functions to end users. The following steps are mostly the same if applied to other Cloud Services.

Cloud Endpoints: Step-by-Step

Reference: https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions

1. Deploy Cloud Function
2. Deploy ESP to Cloud Run
3. Create OpenAPI document to describe the API specifications
4. Deploy Endpoints configuration
5. Configure ESP so it can find the configuration for your Endpoints service
6. Grant ESP permissions to invoke the target Cloud Function

Deploy Cloud Function

In order to deploy a Cloud Function for our solution, we need to create these two files:

1. main.py file that contains our python function,
2. requirements.txt file that lists the python libraries to install. In this case, it will contain the following libraries:

google-cloud-pubsub== 0.41.0
pandas==0.25.1
Flask==1.0.2

main.py

The private package mylib.gcp_cloud_util.gcp_cloud_util encapsulates the interaction with google-cloud packages (storage, pubsub, firestore etc.) in order to simplify its reusable usage. You can refer to this public Github repository to install the package:

edemnati/public

Let‘s’ deploy this Cloud Function using Google Cloud SDK:

gcloud functions deploy publish_rss_feed_search_message --runtime python37 --trigger-http

Deploy ESP to Cloud Run

Reference: https://cloud.google.com/sdk/gcloud/reference/beta/run/deploy

Note: Currently, only us-central1 is supported for Cloud Run.

gcloud beta run deploy rss-news-search --image=”gcr.io/endpoints-release/endpoints-runtime-serverless:1" --[YOUR GCP PROJECT ID]  --platform managed

Explanation:

• gcloud beta: We are using a beta version of glcoud commands, so we should migrate to a future stable release.
• rss-news-search: This is the service name that will be created in Cloud Run. It will also be part of Cloud Endpoints service hostname.
• --platform managed: this option means that we will use a fully managed version of Cloud Run. If omitted, gcloud will ask you to choose a target platform ([1] Cloud Run (fully managed), [2] Cloud Run on GKE, [3] cancel).

A successful result should look like:

Deploying container to Cloud Run service [rss-news-search] in project [YOUR GCP PROJECT ID] region [us-central1]
 Deploying new service…
 Setting IAM Policy…done
 Creating Revision…done
 Routing traffic…\
 Done.
 Service [rss-news-search] revision [rss-news-search-00001] has been deployed and is serving traffic at https://[GENERATED SERVICE HOSTNAME]

Note that [GENERATED SERVICE HOSTNAME] will be used to set the host parameter in the OpenAPI document.

In Google Cloud Console you should see that:

• A new service was created in Cloud Run:

Cloud Run — New Service Added

• A new service was created in Cloud Endpoints:

Cloud Endpoints — New service added

Create OpenAPI document to describe the API specifications

Reference: https://swagger.io/docs/specification/describing-parameters/

We should create a file named openapi-functions.yaml. Each function that we want to expose must be listed in the paths section.

RSS Search Process — openapi-functions.yaml

Explanation:

In the host section, host: [GENERATED SERVICE HOSTNAME], you should copy the URL generated when we deployed the ESP to Cloud Run. You can also get is from the Cloud Endpoints console interface.

Parameters

In the paths section, /rsssearchprocess/start is the unique API path name. It will be concatenated to the host URL to get the full API request URL such as https://[host][api path].

There are many ways to pass parameters to an API. If we create an API using a GET operation, we can pass parameters directly in the URL: path parameters such as /users/{id}, or query parameter such as /users?role=admin.

In our case, we will use a POST operation, so we need to specify a request body to pass parameters to the backend using a json format. This is represented in the yaml file by the highlighted options bellow:

Then we need to describe the body schema object:

• The required section identifies the mandatory properties. In this case, url, source and keyword must be nonempty, but the limit property remains optional.
• For each property we specified a type, which refer to a Data type and a description, which provides a descriptive information about the property. We can also add other specifications such as minimum, which we use in this case to specify a minimum value of 1 for the limit property.

The previous information will be used in the Endpoints Portal as part of the API documentation.

Security

In this example we used an API Key to restrict the API usage but other authentication methods can be used.

The security definition is described at the end of the yaml file:

in: “query” means that the API key will be passed as part of the URL as a query parameter like /start?key=[YOUR_API_KEY]

Finally the x-google-backend section is used to locate the Cloud Function URL: address: https://us-central1-[YOUR GCP PROJECT ID].cloudfunctions.net/[Cloud Function Name].

[YOUR GCP PROJECT ID]: Should be the Project ID where the Cloud Function was deployed.

[Cloud Function Name]: Should be the Cloud Function name, is this example publish_rss_feed_search_message.

Deploy Endpoints configuration

Once the OpenAPI document is filled out, we should deploy it using the following command:

gcloud endpoints services deploy openapi-functions.yaml --[YOUR GCP PROJECT ID]

Configure ESP so it can find the configuration for your Endpoints service

The following command will configure the ESP using:

• rss-news-search: The Cloud Run service name
• [GENERATED SERVICE HOSTNAME]: The Cloud Endpoints service hostname
• [YOUR GCP PROJECT ID]: The Project ID where the Endpoints was deployed

gcloud beta run services update rss-news-search -- set-env-vars ENDPOINTS_SERVICE_NAME=[GENERATED SERVICE HOSTNAME] --[YOUR GCP PROJECT ID] -- platform managed

If you get an error such as: gcloud crashed (AttributeError): ‘NoneType’ object has no attribute 'revisionTemplate', updating gcloud components might fix the problem. Run this command to do so: gcloud components update

A successful result should look like:

Deploying…
 Creating Revision…done
 Routing traffic…done
 Done.
 Service [rss-news-search] revision [rss-news-search-00001] is active and serving traffic at https://[GENERATED SERVICE HOSTNAME]

Grant ESP permissions to invoke the target Cloud Function

The following command will grant permissions to ESP to invoke a Cloud Function using:

publish_rss_feed_search_message: Cloud Function name

[Default compute service account]: Go to the IAM console to find the Default compute service account

[YOUR GCP PROJECT ID]: The Project ID where the Cloud Function was deployed

gcloud beta functions add-iam-policy-binding publish_rss_feed_search_message --member "serviceAccount:[Default compute service account] --role "roles/cloudfunctions.invoker" --project [YOUR GCP PROJECT ID]

A successful result should look like:

bindings:
 - members:
   - serviceAccount:[Default compute service account]
   role: roles/cloudfunctions.invoker
 etag: xxxxxxxxxxx
 version: 1

Cloud Endpoints: How to…

Delete a Cloud Endpoints service

• List services: gcloud endpoints services list
• Delete a service: gcloud endpoints services delete [GENERATED SERVICE HOSTNAME]

Granting access to Enpoints Portal

Reference: Ref: https://cloud.google.com/endpoints/docs/grpc/control-api-access

gcloud beta run services add-iam-policy-binding [GENERATED SERVICE HOSTNAME] --member='[User email]'  --role='roles/run.invoker'

Configure quotas

Reference: https://cloud.google.com/endpoints/docs/openapi/quotas-configure

Deploy a second secured API

The previous step-by-step process is used to deploy a second API to allow authorized users to get access to articles. Here is a short summary of these steps:

Deploy Cloud Function

gcloud functions deploy read_articles_firestore --runtime python37 --trigger-http

Deploy ESP to Cloud Run

gcloud beta run deploy read-news-articles --image="gcr.io/endpoints-release/endpoints-runtime-serverless:1" --allow-unauthenticated --project=[YOUR GCP PROJECT ID] --platform managed

Create OpenAPI document to describe the API specifications

Read News Articles — openapi-functions.yaml

In this case, we deployed two access points to test different security models:

• /readnewsarticlesapi/start: will be secured using an API key as shown previously.
• /readnewsarticlesuser/start: will be secured using Firebase authentication method. The securityDefintion section should be configured as:

Firebase authentication method

Deploy Endpoints configuration

gcloud endpoints services deploy openapi-functions.yaml --project [YOUR GCP PROJECT ID]

Configure ESP so it can find the configuration for your Endpoints service

gcloud beta run services update read-news-articles --set-env-vars ENDPOINTS_SERVICE_NAME=[GENERATED SERVICE HOSTNAME] --project [YOUR GCP PROJECT ID] --platform managed

Grant ESP permissions to invoke the target Cloud Function

gcloud beta functions add-iam-policy-binding read_articles_firestore --member "serviceAccount:[Default compute service account] --role "roles/cloudfunctions.invoker" --project [YOUR GCP PROJECT ID]

Test your API

Endpoint Portal

You can create a portal from the Cloud Endpoints console in the Developer Portal section. Once created you should see a URL to your Endpoints Portal:

If you go to the Portal you should see your API services:

Endpoints Developer Portal

RSS Search Process

You must configure the API Key to be able APIs secured by an API Key. Go to Settings and enter an authorized API key.

If you don’t already have an API Key, follow these steps to create one:

1. Log to GCP console and go to APIs & Services > Credentials

Create a New API Key

2. Click on Create Credentials > API Key

API Key Created

3.Click Restrict Key option to restrict the API Key to a specific scope. In this case, we restricte the API Key to only access our two APIs create before:

Restrict API Key

Let’s test this out!

Test Endpoints Portal APIs

Test your API using Python

This sample code comes from Endpoints Portal. Other sample codes are available for cURL, HTTP, NODE.JS, Javascript, JAVA and PHP.

# Demo code sample. Not indended for production use.
# See instructions for installing Requests module for Python
# http://docs.python-requests.org/en/master/user/install/
import requests
def execute():
  requestUrl = "https://[GENERATED SERVICE HOSTNAME]/readnewsarticlesuser/start"
  requestBody = {
    "entity": "Apple"
  }
  requestHeaders = {
    "Authorization": "Bearer [YOUR_API_KEY]",
    "Accept": "application/json",
    "Content-Type": "application/json"
  }
request = requests.post(requestUrl, headers=requestHeaders, json=requestBody)
print request.content
if __name__ == "__main__":
  execute()

Share your work

Since we have created secure APIs, we can choose to use them in Web applications or to share them with other users.

• For API Keys authentication, we might create different API Keys to track users or applications calls.
• For Firebase authentication method, you should grant users access to your services.
• You can also allow someone to enable your services in their own Cloud project to invoke the APIs. You need to assign them the Service Consumer role. You can use gcloud commands or you can go to Cloud Endpoints Console, select a service and click on Permissions to add members and give them Service Consumer role.

Last words

Your APIs are also visible in the APIs & Services library. You can filter by Private tag to see your APIs. Click on an API to be able to manage it. You will have the ability to disable it, to see some metrics about the traffic, errors, latency, and to see which API Keys are granted access to it.

APIs & Services Screen

Manage API Screen

I hope you enjoyed this post. Please share your comments, feelings, improvements...