Getting Started w/ Python on GCP
The “Missing Tutorials” series
Writing a short series of ‘getting started’ posts for those of you, like me, who may get to the point of wanting to write code against a Google service, having a language chosen but then, having not written code for a week or two, I’m stalled by “How exactly do I get started?”
Python
Everyone loves Python!
Google’s first Google Cloud Platform service, App Engine was launched in 2008 and supported Python. Python is a much-loved and much-used language at Google. Guido van Rossum — Python’s BDFL spent time at Google. Alex Martelli still does.
Python is one of the languages for which Google provides Libraries and code samples.
Let’s get started.
Setup
To avoid supporting the myriad ways you will arrive at this point, I’m just going to tell you what I’m running (Linux), Python (2.7.13), pip (9.0.1) and virtualenv (15.1.0).
A compelling advantage in using virtualenv is that it creates an isolated Python environment for you. This yields reproducibility; e.g. what works here for me, should work same-same for you since we’re both using virtualenv; what works for you should work for your customers; what works for you today should work for you tomorrow since you won’t introduce conflicting packages.
Please get yourself to a working Python runtime. Use Python to install pip. Use pip to install virtualenv.
PROJECT_ID=[[YOUR-PROJECT-ID]]
LANG=pythonmkdir -p ${HOME}/${PROJECT_ID}/${LANG}
cd ${HOME}/${PROJECT_ID}/${LANG}virtualenv venv
source venv/bin/activatepip freeze
The convention is to call the virtualenv environment ‘venv’ but you may use whichever name you prefer. This is the directory where virtualenv installs a copy of Python and a copy of the Python packages used by this (!) environment. You must activate the virtualenv environment to use it, this is done by source’ing the “activate” script under bin in the “venv” (or you preferred alternative) directory that was just created. “pip freeze” lists installed packages. The list will likely be empty because we have created a clean virtualenv. However the “pip” package will be installed and possibly other key packages.
Google Python packages
Where to find Google’s Python packages? Mostly we will be looking either for the API Client Libraries or for the Cloud Client Libraries. You can determine the library name and current version in various places.
API Client Libraries
As I write, the current version is 1.6.2
https://developers.google.com/api-client-library/python/start/installationPyPi
https://pypi.python.org/pypi/google-api-python-client
https://pypi.python.org/pypi/google-api-python-client/1.6.2GitHub
https://github.com/google/google-api-python-client
https://github.com/google/google-api-python-client/releasesrequirements.txt:
google-api-python-clientCloud Client Libraries
NB #1 Google has decided to decompose the Cloud Client Libraries from one package containing all the Google Cloud services into one package for each Google Cloud service. To minimize the distribution size of your solutions and to potentially manage package versions by service, you are encouraged to install only those packages that you need for the services you will use.
NB #2 Cloud Client Libraries are released on a cycle that lags the underlying service. A service may be GA but the Cloud Client Library *may* be Alpha.
https://cloud.google.com/apis/docs/cloud-client-librariesPyPi
https://pypi.org/project/google-cloud/
https://pypi.python.org/pypi/google-cloudGitHub
https://googlecloudplatform.github.io/google-cloud-python/
https://github.com/GoogleCloudPlatform/google-cloud-pythonrequirements.txt:
google-cloud-bigquery
google-cloud-datastore
google-cloud-runtimeconfigNB Google recommends using only the packages needed for the services that you will use. This list is for example only. The complete list is at the bottom of the PyPi page for google-cloud.
Google Runtime Configurator
We’re going to use the Runtime Configurator service. I’ve not written against this service using Python until this post. I have accessed the service using Node.js (more in that post) and I noticed that there is also a Cloud Client Library for Runtime Configurator so we will write this service two-ways (here: API Client Library; later: Cloud Client Library).
https://cloud.google.com/deployment-manager/runtime-configurator/Although part of Deployment Manager, this service is useful elsewhere. It provides a central location in your GCP project to store (and monitor, “watch”) runtime configuration variables. Many distributed systems require, for example, an initial set of variable (values) that may be used to bootstrap a cluster. Runtime Configurator could be the service to help with this. It’s also a straightforward API that does not require particular knowledge of GCP services to be understood. We will use 4 methods:
1+2: creating and deleting config resources:
https://cloud.google.com/deployment-manager/runtime-configurator/create-and-delete-runtimeconfig-resources3+4: creating and deleting variable(s) within the config:
https://cloud.google.com/deployment-manager/runtime-configurator/set-and-get-variablesLooking in the documentation, we can see that these methods are available with the “v1beta1” API:
https://cloud.google.com/deployment-manager/runtime-configurator/reference/rest/Solution #1: Using API Client Libraries
For the API Client Libraries, there is only one package that must be installed. You may install the library directly using pip install but it is a better practice to create a file called (conventionally) requirements.txt and listing the package names and minimum versions therein, e.g.
requirements.txt:
google-api-python-client==1.6.2You need only provide the package name (google-api-python-client) but, for precision in the above file, we’re requiring that the current version (i.e. specifically version 1.6.2) be used.
Assuming you followed the setup instructions, we should be operating within a virtualenv environment and, issuing the following command(s) will install the API Client Library (and its dependencies) into the environment
pip install --requirement requirements.txtpip freeze
google-api-python-client==1.6.2
httplib2==0.10.3
oauth2client==4.1.2
pyasn1==0.3.3
pyasn1-modules==0.1.1
rsa==3.4.2
six==1.10.0
uritemplate==3.0.0
NB one of the dependencies was oauth2client and this is the package that provides us the ability to use Application Default Credentials.
Let’s write some Python
What import statements do we need to use Runtime Configurator from Python? Good question. It’s not immediately clear and, because this is a Beta API, the documentation does not include the customary multi-language examples:
https://cloud.google.com/deployment-manager/runtime-configurator/create-and-delete-runtimeconfig-resourcesSo, how we know what to include? Well, there’s a ‘pattern’. All services are referenced by API name and version, like so:
from apiclient.discovery import build
service = build('api_name', 'api_version', ...)This is explained here:
https://developers.google.com/api-client-library/python/start/get_startedOK…. So what is the API name? From the documentation we know that we need “v1beta1”. This page lists all the Google services by version. So, open it and find “Runtime”:
https://developers.google.com/api-client-library/python/apis/
So, it’s called “runtimeconfig” and we want “v1beta1”. We can verify the API using API Explorer. Using API Explorer to find “runtimeconfig” returns the 2 versions of the API as expected. Selecting “v1” does not include the methods that we need:
https://developers.google.com/apis-explorer/#search/runtimeconfig/runtimeconfig/v1/
Using “v1beta” includes the projects.configs resource with the create and delete methods (1+2) and the projects.configs.variables resource with the create and delete methods (3+4):
https://developers.google.com/apis-explorer/#search/runtimeconfig/runtimeconfig/v1beta1/
So, we need the following boilerplate for the runtimeconfig API and to use Application Default Credentials
runtimeconfig.api.py:
from apiclient.discovery import build
from oauth2client import GoogleCredentialsSERVICE_NAME = 'runtimeconfig'
SERVICE_VERSION = 'v1beta1'credentials = GoogleCredentials.get_application_default()service = build(
SERVICE_NAME,
SERVICE_BETA,
credentials=credentials
)
Let’s do the construction in Python and the destruction using the Cloud SDK (aka “gcloud”). How to create a configs (sic.). One nice consequence of the machine-generated API Client Libraries is that they are all very formulaic. Most resources in GCP are owned by project resources and so we have a Python class structure that’s of the form:
projects.[resources].[resources].method(request-body).execute()The Python API documents (and API Explorer) are great ways to investigate how to code against the API. Let’s start with the API Explorer as it’s the easiest…. runtimeconfig.configs.create. You can navigate your way to this method or construct the URL:
https://developers.google.com/apis-explorer/#search/runtimeconfig/runtimeconfig/v1beta1/runtimeconfig.projects.configs.create
We’re required to provide a “parent” and told this must be of the form “project/[YOUR-PROJECT-ID]”. We ought create a “requestId” and, expanding the request body there are only 2 properties: name and description. These are self explanatory. If we were familiar with coding the API Client Libraries in Python, this would be sufficient. But, to explore, let’s check the API documentation. The consistent (all services) way to do this is to (return to the) Google “API Client Libraries” page:
https://developers.google.com/api-client-library/Choose “Python”, then “APIs”, find “runtimeconfig” and then from this templated page, select PyDoc:
https://developers.google.com/api-client-library/python/apis/runtimeconfig/v1beta1Bringing us to:
https://developers.google.com/resources/api-libraries/documentation/runtimeconfig/v1beta1/python/latest/
You’ll recall we’re confident the methods are under “projects()”, clicking it you’ll be taken to “configs()” (good) and this provides our create method:
https://developers.google.com/resources/api-libraries/documentation/runtimeconfig/v1beta1/python/latest/runtimeconfig_v1beta1.projects.configs.html
And there’s the clarity for the method “create”. Aha! The “name” property isn’t as straightforward as I’d assumed. Scrolling through the PyDoc we’re told that the config name must be of the form:
projects/[PROJECT_ID]/configs/[CONFIG_NAME]That’s obviously important. OK, here’s one way to code this call in Python assuming the prior code and the nuance of the config name property:
import uuidPROJECT_ID = "[YOUR-PROJECT-ID]"
NAME = "dev"body = dict()
body["name"] = "projects/{}/configs/{}".format(PROJECT_ID, NAME)
body["description"] = "Dev Environment Config"params = dict()
params["body"] = bodyrequest = service.projects().configs().create(
parent="projects/{}".format(PROJECT_ID),
requestId=str(uuid.uuid4()),
**params
)
Assuming a constant PROJECT_ID, we’ll set the parent to be of the form projects/[PROJECT_ID] per documentation and API Explorer guidance.
I’m using a (random) GUID for the requestId because we were recommended to use a unique value.
We’ll create a dictionary called “body” and set name=”dev” and description=”Dev Environment Config”. Then we’ll create a dictionary called “params” (arbitary) and set a property on it, called “body” to the value of the body dictionary. We pass ‘params’ unpacked as the body ;-)
We’re set. Now to issue the call. Again, convention in the API Client Libraries makes this formulaic:
response = request.execute()And, to confirm it worked, let’s dump the (JSON) result:
import jsonjson.dump(
response,
sort_keys=True,
indent=2
)
Jump ahead to “Testing” if you’d like to test the code.
We have more methods to write. Hopefully, this second method will give you sufficient knowledge that you may complete the rest for yourself. If you become bored, the “Testing” section shows how to use the Cloud SDK to create and delete configs and variables too ;-)
So, for the 3rd method, we want to create variables in our newly-created config “dev”. PyDocs describes the method:
https://developers.google.com/resources/api-libraries/documentation/runtimeconfig/v1beta1/python/latest/runtimeconfig_v1beta1.projects.configs.variables.html
Reviewing the docs, the variables name has a structure similar to the configs name, so we’ll format that. The value of the variable may be set at creation time. It can either be “text” or a base-64 encoded “value”. To keep it simple, I will set the text property. Here’s the code:
VARIABLE = "my-variable"body = dict()
body["name"] = "projects/{}/configs/{}/variables/{}".format(
PROJECT_ID,
NAME,
VARIABLE
)
body["text"] = "my-value"params = dict()
params["body"] = bodyrequest = service.projects().configs().variables().create(
parent="projects/{}/configs/{}".format(PROJECT_ID, NAME),
requestId=str(uuid.uuid4()),
**params
)response = request.execute()json.dumps(
response,
sort_keys=True,
indent=2
)
Jump ahead to “Testing” to test your code now.
When you’re done with this sample, you can deactivate the virtualenv with:
deactivateThis preserves the virtualenv enviroment, if you wish to delete it, delete the “venv” directory with your code. If you wish to re-enter this virtualenv (though I would recommend you create separate virtualenv environments for each distinct project), you may repeat the command:
source venv/bin/activateSolution #2: Using Cloud Client Libraries
For the Google Cloud Client Libraries you have two alternatives: not recommended: install *all* the Google Cloud Client Libraries (for all the available services) using one package; recommended: install the specific Cloud Client Libraries packages for the services you will use.
In this case, we will use the Runtime Configurator. But, how do we know whether this service is one the services supported by the Cloud Client Libraries for this service and for Python?
This page lists the Libraries documentation by language:
https://cloud.google.com/apis/docs/cloud-client-librariesChoosing Python “Library Reference” brings us to:
https://googlecloudplatform.github.io/google-cloud-python/
And, selecting “Read the Docs” presents a summary of the services that are supported. You should be able to see “Runtimeconfig” listed on the left hand side:
Unfortunately (!) the documentation is misleading here. It suggests that you should install “google-cloud” which is counter to the Google guidance. So, please tread carefully. You are better place reviewing the PyPi libraries:
https://pypi.python.org/pypi/google-cloud
And “Google Cloud Runtime Configuration” provides:
NB here, we’re correctly provided the specific pip install for the runtimeconfig package:
pip install --upgrade google-cloud-runtimeconfigAs before though, we will use requirements.txt to manage the packages. You may continue using the virtualenv environment that we created for the API Client Library, or you may create a new environment.
I’m honestly unclear in finding the current version through this path but there are two places where you can find the current version for the specific service library:
https://pypi.org/project/google-cloud-runtimeconfig/
or via the GitHub Releases page and search for the library:
https://github.com/GoogleCloudPlatform/google-cloud-python/releases
requirement.txt
google-cloud-runtimeconfig==0.27.0And, as before:
pip install --requirement requirements.txtpip freeze
cachetools==2.0.1
certifi==2017.7.27.1
chardet==3.0.4
futures==3.1.1
google-auth==1.0.2
google-cloud-core==0.27.0
google-cloud-runtimeconfig==0.27.0
googleapis-common-protos==1.5.2
idna==2.6
protobuf==3.4.0
pyasn1==0.3.3
pyasn1-modules==0.1.1
requests==2.18.4
rsa==3.4.2
six==1.10.0
urllib3==1.22
The Cloud Client Library documentation is useful and mostly all you will need. There’s a section called “Core” which is the foundational content including using Application Default Credentials (remember you will need to use a service account to run code locally and you will need to set APPLICATION_DEFAULT_CREDENTIALS=/path/to/your/key.json). Then, we’ll use the documentation specifically for the Runtime Configurator:
https://googlecloudplatform.github.io/google-cloud-python/latest/core/index.html
https://googlecloudplatform.github.io/google-cloud-python/latest/runtimeconfig/usage.htmlI’m going to create a file called runtimeconfig.cld.py:
from google.cloud import runtimeconfigclient = runtimeconfig.Client()
That’s all that’s needed to use Application Default Credentials from our code and to instantiate a client.
Then, my problems begin. I’m looking through the Runtime Configurator resources expecting to find create methods on Configs and Variables but not finding these even though the documentation suggests that these should exist (for this library) and we know that they exist on the service. Being familiar with these libraries with other services, the pattern is:
client.[resources](...).create()So, I filed some GitHub issues and will have to defer until these issues are resolved:
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3897
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3898The getters for Configs and Variables do work so… If you still have the “dev” config and “my-variable” variable created by the API Client, we will reuse these. If you do not have the config and variable created, you can create them using the Cloud SDK:
NAME=devgcloud beta runtime-config configs create ${NAME} \
--project=${PROJECT_ID}gcloud beta runtime-config configs variables \
set my-variable "my-value" \
--is-text \
--config-name=${NAME} \
--project=${PROJECT_ID}
Then, the following code will get the Configs resource called “dev” and then get the Variables resource called “my-variable” within it:
NAME="dev"config = client.config(NAME)
print(config.exists())
print(config.get_variable("my-variable").exists())
print(config.get_variable("my-variable").full_name)
print(config.get_variable("my-variable").state)
print(config.get_variable("my-variable").value)
All being well, this should return (albeit unexpectedly):
True
True
projects/${PROJECT_ID}/configs/${NAME}/variables/my-variable
None
NoneUnfortunately, there appears to be another bug in the library. We’d expect “state” to be VARIABLE_STATE_UNSPECIFIED and “value” to be “my-value”.
I’ll update this post when responses have been published to my issues to determine whether this is not the library but me ;-)
Testing
You’ll need a Google Cloud Platform project in order to test your code. You’ll also need to ensure the “RuntimeConfig API” is enabled in it. You can do this either via the Cloud SDK:
gcloud service-management enable runtimeconfig.googleapis.com \
--project=${PROJECT_ID}Or, using Google Cloud Console and a corrected version of this URL:
https://pantheon.corp.google.com/apis/api/runtimeconfig.googleapis.com/overview?project=${PROJECT_ID}Then, from within the virtualenv environment (or your preferred editor’s debugger) run:
python runtimeconfig.api.py
python runtimeconfig.cld.pyYou’ll likely receive an error if you jump ahead and try to run the code using Application Default Credentials (ADCs) as you may expect:
gcloud auth application-default loginThis is because, this beta version of the service has not been fully enabled for ADCs. You may receive an error of the form:
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://runtimeconfig.googleapis.com/v1beta1/projects/${PROJECT_ID}/configs?alt=json&requestId=... returned "Google Cloud Runtime Configuration API has not been used in project usable-auth-library before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/runtimeconfig.googleapis.com/overview?project=usable-auth-library then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.">In this case, you must create and use a service account to authenticate your code and, you will thereby need to set:
GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/key.jsonThen, running the code you may get:
Nothing :-( It’s a bug. The service doesn’t return anything (beside a 200) in the response. It worked though. You may check by using the Cloud SDK. We list the configs:
gcloud beta runtime-config configs list \
--project=${PROJECT_ID}NAME DESCRIPTION
dev Dev Environment Config
And then the variables in the “dev” configs:
gcloud beta runtime-config configs variables list \
--config-name=dev \
--project=${PROJECT_ID}NAME UPDATE_TIME
my-variable 2017-08-29T01:32:33.789003860Z
And then the “my-variable” value:
gcloud beta runtime-config configs variables get-value my-variable \
--config-name=dev \
--project=${PROJECT_ID}my-value
If we’d not used the “text” property on the variables create, we’d have received a base-64 encoded value that would need be to decoded.
When you’re done, you may delete this config (which will also delete all its variables :-) [2+4] using:
gcloud beta runtime-config configs delete dev \
--project=dazwilkin-170828-mediumDeleted [https://runtimeconfig.googleapis.com/v1beta1/projects/${PROJECT_ID}/configs/dev].
Aside: Google Documentation
Because Python has been so widely used by Google, there is a rich history of Python libraries for Google services. This can trip you up when Googling for help. Please see here and here for a recap. Fundamentally (and this is true for all the languages), there are 2 types of library: API Client Libraries; and Cloud Client Libraries. You should endeavor to limit your choice to these Libraries. Because Python was available in Google App Engine in 2008, there are Python libraries built specifically for App Engine services (e.g. Python DB Client Library for Cloud Datastore). Use these specific libraries if you are unable to use the Client Libraries. When Googling try to include the precision of “API Client Library” e.g. for “Datastore”.