Setup and Invoke Cloud Functions using Python

Oh the fun I’ve had learning Cloud Functions. It has actually be fun while also frustrating and definitely enlightening to explore serverless services that are supposed to make things easy. Granted I’ve been around long enough to know there is always ramp up, even when it is simple.

Cloud Functions is a service that allows you to run code on Google Cloud servers without needing to deal with server configuration or scaling. It is a pay as you go approach that means you pay for what you use and that can help optimize costs. I’ve started using it for a couple functions that I need to run multiple times and that is where it can be very valuable. There was a learning curve especially when connecting different services and this post covers key learnings on these subjects.

• Testing code locally before running in Cloud Functions
• Setting up Cloud Functions with an HTTP trigger and Python
• Invoking Cloud Functions from your laptop
• Troubleshooting errors and authentication
• Cost

This post assumes you have the gcloud SDK already installed locally and that you have a Google Cloud project setup. I’ve got previous posts that cover these topics. I used Python 3.7 in the function and a Mac for local development.

Emulating Cloud Functions Locally

If you want to test your code before running in Cloud Functions then you can do that with Functions Framework for Python.

Simple Example | No Parameters Passed

Install functions-framework with pip on your machine.

pip install functions-framework

Draft up a simple example that you can test like the following and label it main.py.

def my_function(request):
    return 'Hello World'

To simulate Cloud Functions, run functions-framework command in your terminal.

functions-framework --target my_function

There are flags you can pass in to make adjustments like the name of the file ( — source) and the port it points to ( — port) for example. And use the — debug flag for more detailed help when developing.

Then call curl to invoke it locally to see how the HTTP trigger works or run it in a browser window.

curl http://localhost:8080
OR for the browser
http://localhost:8080

The top Stack Overflow response in the link is where I pulled the above code. Emulating Cloud Functions is especially valuable when you pull in packages and want to see how those packages will or will not work as well as checking connections to other services. Also, this is a good framework to use to test Cloud Run locally as well.

Passing Parameters

The following is an example of how to pass in parameters to a Cloud Function.

Use this starter code to experiment and put that in a main.py file.

def hello(request): 
    if request.args:
        return f'Hello' + request.args.get('first') + request.args.get('last')+'\n'
    elif request.get_json():
        request_json = request.get_json()
        return  f'Hello' + request_json['first'] + request_json['last']+'\n'
    elif request.values:
        request_data = request.values
        return  f'Hello' + request_data['first'] + request_data['last']+'\n'
    else:
        return f'Hello World!'+'\n'

In order to test it you can use the curl command.

curl http://localhost:8080?first=My&last=Name

Or you can pass the parameters with the following for Linux or Mac.

curl http://localhost:8080 -d ‘{“first”:”My”, “last”:”Name”}’

Or you can call it in the browser with the same url.

http://localhost:8080?first=My&last=Name

Setup Cloud Function

If you want to run code in an actual Cloud Function you can use the Google Cloud Console to setup your function to run. You can also post up the code using gcloud SDK. For this post, I’m sticking to the UI but feel free to checkout this link for more info on launching from your terminal with gcloud.

Go to the Google Cloud Console and the Cloud Functions dashboard. Select Create Function.

Fill out the name of the function to use in the URL for the HTTP Trigger. Also choose the Trigger to use. There are other trigger options but for this example I stuck with HTTP.

The url to invoke the function will post under URL in the image above. You can test it directly in the UI, curl it in your local terminal, run it through a browser or post a request through code.

The Authentication box is the tricky bit. If selected then anyone can make a call to the function. This is a way to test it and not deal with authentication, but be careful because it is open for anyone who finds the url (including bots) to call it and that can get costly since it is pay as you go. More on cost further down in the post. Note, it automatically is unselected and if you forget about it then you may go down the rabbit hole I did trying to figure out why I couldn’t invoke without authentication.

In Runtime select the language you want to code in for the function and code up your function accordingly under MAIN.PY. When using HTTP and Python, you have to accept a request object so you can pass in parameters. You also need to fill out the REQUIREMENTS.TXT tab with any requirements that you import that are not already part of Python’s standard package.

In Function to execute, put the name of the function in your code that the program needs to execute when it runs. There is sample code already there to help get you started.

When you expand the Environment Variables, Networking, Timeouts and More, you will find under Networking the Ingress settings that state Allow all traffic. If you miss checking the box above for Allow unauthenticated invocations but make sure Allow all traffic is selected (which is the default) then it will still require authentication. This threw me off for a bit which I reiterate below.

Also a note about Timeouts is that the program defaults to 60 seconds, and you can get at most 9 minutes before the function will time out if it is waiting for a response to finish processing.

Once the function is setup, choose to Create and give it a few minutes to get the function launched on a server. When there is a green checked circle, it is ready to be run.

You can always go back and edit the function and Deploy it again if there are errors.

Invoke Cloud Function

After the function is setup, you can invoke/run the function. This step was more challenging for me than it needed to be because of the Authentication check box as mentioned and how Python requests handle the data vs json terms.

All User Access

When granting anyone access to the Cloud Function, you can leave off passing authentication parameters in your code and the following provide invoke examples.

Terminal
Curl an existing Cloud Function without parameters.

curl https://[MYPROJECT].cloudfunctions.net/[FUNC NAME]

Curl an existing Cloud Function that takes in parameters.

curl https://[MYPROJECT].cloudfunctions.net/[FUNC NAME] -H “Content-Type:application/json” -d ‘{“first”:”Mae Carol”, “last”:”Jemison"}’

Replace MYPROJECT with your Google Cloud project id and FUNC NAME with the Cloud Function name as noted in setup.

Python Code
When calling the url inside a Python function, I used the requests package to apply the HTTP trigger and pass in parameters through the request.

Create these variables.

url = "https://[MYPROJECT].cloudfunctions.net/[FUNC NAME]
param = {“first”:”Mae Carol”, “last”:”Jemison"}

Make the request call to the Cloud Functions url.

r = requests.post(url, json=param)

If you are getting an error like “Your client does not have permission to the requested URL” it may not necessarily be the authorization despite what the error says. I found that the real issue above was that I was using the keyword data and not converting my param into json format. When I changed the input parameter from data to json it worked.

This is the code I was stuck on that did NOT work before I switched to json.

r = requests.post(url, data=param)

The reason data was not working was because my original code left off the ability to parse parameters passed in under the data flag. This was the code I was missing before I added it into the example at the start of the post.

request_data = request.values
return  f'Hello' + request_data['first'] + request_data['last']+'\n'

I can also leave off the above code out of my example and continue to use data=param if I convert the parameters into json with json.dump or if I pass in headers that clarify the content type as json.

import json
r = requests.post(url, data=json.dump(param))

OR

newHeaders = {'Content-type': 'application/json', 'Accept': 'text/plain'}
headers=newHeaders
r = requests.post(url, data=param, headers=newHeaders)

Authorized Access

As mentioned, I spent a good chunk of time down the auth rabbit hole because I thought for a while I did not have permission. A couple things on this point are that I didn’t actually have permission initially since I missed that Authentication checkbox but I also had some challenges getting the token to load in my Python code.

Terminal
I found pretty quickly the following example code that works to invoke my Cloud Function from my laptop and pulls in my default identity token since I have gcloud configured.

curl -H “Authorization: Bearer $(gcloud auth print-identity-token)” https://[MYPROJECT].cloudfunctions.net/[FUNC NAME] -H “Content-Type:application/json” -d ‘{“first”:”Mae Carol”, “last”:”Jemison"}’

Python Code
When I went to try this out in Python that’s where it got challenging. Since my bash command worked, it helped me identify that loading the token into my code was part of the challenge.

I experimented with google-auth and google-oauth packages. I even used os and subprocess packages to simply pull the token directly from the bash call. What I learned from trying all these things was I was not able to get a token to load with google-auth and google-oauth which is a problem for another time.

One of the errors I was getting as noted above was because I needed to swap data with json in the request call or fix how my parameters were configured.

I was able to parse out my token with os and subprocess, and this is the subprocess command I ended up using.

import subprocess
token = '{}'.format(subprocess.Popen(args="gcloud auth print-identity-token", stdout=subprocess.PIPE, shell=True).communicate()[0])[2:-3]

Is this a good way to do it? Debatable and most likely no. But it worked and I needed to move on to get other things done. Feedback always welcome and if I find a better way I’ll try to come back here and update.

Use this code to make a request to the Cloud Function url using an authentication token and without params.

r = requests.post(url, headers={"Authorization":"Bearer {}".format(token)})

Use this code to make a request to the Cloud Function url using an authentication token and with params.

r = requests.post(url, json=param, headers={"Authorization":"Bearer {}".format(token)})

After requests runs you can check how it did. The following commands give insights on whether the token is incorrect and if the request responded with a 200 or a 500 or something in the 400 range codes.

r.headers
r.status_code

Troubleshooting Side Notes

Authentication | All User Access

If you missed that Authentication checkbox the first go around when setting up your function, you use these directions from the Managing Access via IAM to make it available for anyone to use it. Remember to be careful on this access and consider using it cases like testing the integration.

1. Go to Google Cloud Console
2. Click the checkbox next to the function on which you want to grant access.
3. Click Show Info Panel in the top right corner to show the Permissions tab.
4. Click Add member.
5. In the New members field, type allUsers.
6. Select the role Cloud Functions > Cloud Functions Invoker from the Select a role drop-down menu.
7. Click Save.

Authentication | Metadata Tokens

If you are running Cloud Functions requests from a Google Cloud service, there are directions that show how to get a token by calling a metadata service. The docs under Authenticating Developers, Functions and End-users goes into more details. Below is one sample of some code you can use in Python.

REGION = 'us-central1'
PROJECT_ID = [PROJECT ID]
RECEIVING_FUNCTION = [FUNC NAME]
function_url = f'https://{REGION}-{PROJECT_ID}.cloudfunctions.net/{RECEIVING_FUNCTION}'
metadata_server_url = \
    'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience='
token_full_url = metadata_server_url + function_url
token_headers = {'Metadata-Flavor': 'Google'}
token_response = requests.get(token_full_url, headers=token_headers)
jwt = token_response.text
function_headers = {'Authorization': f'bearer {jwt}'}
r = requests.get(function_url, headers=function_headers)

When I was trying to figure out the token locally, I tried metadata and had other errors that threw me off. “Max retries exceeded with url” and “Failed to establish a new connection”. This led me down a confused path thinking I needed to change a configuration in Cloud Functions. Bottom line, you have to be on a service that you can call it from like Cloud Compute.

Troubleshooting and Logging

When building out something with multiple integrated services, it is good to break down the components and test them directly on each individual service when you can. I’ve shared above that Cloud Functions allows testing directly in the UI or from the command line. Use that if you are getting errors that aren’t clear. Also, use logging where you can. Google Cloud provides logging and you can see more about how to set that up to record into the Operations Logging service of the platform. Detailed logging is your friend and can help track down weird errors.

If you’ve read any of my previous posts, you may know that I like to start small and expand and do a little testing of integration locally whenever possible. That can be challenging when dealing with authentication, differences in configurations and not enough logging details. Start small, add logs, test each service individually and grow.

Reduce Time & Costs

Last but not least, remember to use return in your Cloud Function when using HTTP trigger. This will ensure that the function ends vs running until it timeout. You don’t want to pay for the time for it to timeout if it can finish the function more quickly.

Cost

The total cost of using Cloud Functions include how many times it is called, how long it runs, how many resources are provisioned and if any outbound network requests are made. So the equation is invocations + compute time + networking. The free tier gives 2M invocations per month, 1M seconds of free compute per month and 5GB free Internet egress traffic per month. After that, invocation is a flat rate of $0.4 per million, networking is a flat rate of $0.12 per GB and compute time is in tiers of $0.0000025 per GB-second or $0.00001 per GHz-second for tier 1. To learn more you can checkout the Pricing docs which provides more specific example and breakdown of cost.

Wrap Up

I stepped through setting up and invoking a Cloud Function using Python 3.7. I shared pitfalls that I experienced in the hopes that others who may hit these errors can find this and see how to navigate out of the problem faster than I did when searching. A key thing to keep in mind when using Cloud Functions is that it’s good to use when you need to make many, many calls to that function and the function can complete in less than 9 minutes; otherwise, you should look at something like Cloud Run.

Have fun exploring.