How to Develop, Debug and Test your Python Google Cloud Functions on Your Local Development Environment

Using the Functions Framework for Python to debug and test how your HTTP- and Pub/Sub-triggered functions behave before deploying them to the Google Cloud Platform.

Source: FullHD Wallpapers

One of the main questions we face when starting developing our Google Cloud Functions is:

“Ok. This sounds very nice, but how can I develop and test my code before deploying it?”

In this tutorial, I’ll show how you can develop and test your Python-based Cloud Functions on your local development environment, giving examples for how to debug and test HTTP- and Pub/Sub- triggered functions.

If you prefer a more practical approach, feel free to reach directly to the source code repository:

ivamluz/cloud-build-cloud-functions-pipeline

Disclaimer: this sample project will also serve as the basis for a future tutorial I intend to publish on how to automate the deployment of Google Cloud Functions using Google Cloud Build, hence the name.

Functions Framework for Python

To achieve our objective, we’ll make use of the Functions Framework for Python, an open source FaaS (Function as a service) framework for writing portable Python functions — brought to you by the Google Cloud Functions team.

GoogleCloudPlatform/functions-framework-python

The framework allows us to run a simple HTTP server that acts as a kind of proxy in front of our functions, making it really simple to test our functions with different payloads:

The local development test flow

A Quick Look Into our Sample Code

To follow along, please check the instructions described in the README file of the sample repository. The README guides you through the process of:

1. Cloning the source code
2. Configuring your environment with VirtualEnv and Python dependencies.
3. Running and testing the application locally with some helper shell scripts.

To what concerns the application, these are the main files:

requirements-dev.txt
Defines the Python dependencies to be installed locally. The reason it’s prefixed with -dev is because we don’t want the functions-framework package to be installed when our functions are deployed, something Cloud Functions would do automatically if our file was called requirement.txt, as stated in the docs:

Dependencies in Python are managed with pip and expressed in a metadata file called requirements.txt. This file must be in the same directory as the main.py file that contains your function code.

When you deploy your function, Cloud Functions downloads and installs dependencies declared in the requirements.txt file using pip.

Reference: https://cloud.google.com/functions/docs/writing/specifying-dependencies-python

src/main.py
This is the Python file where our Cloud Functions are implemented:

src/main.py with HTTP- and Pub/Sub-triggered functions

As you can we see, we have two sample functions:

• sample_http is a very simple function that reads the subject query string argument from the request parameter (a flask.Request object) and returns a hello message.
• sample_pubsub decodes the message received and logs it along with the event and context parameter objects. For more information about the information contained in the event and context parameters, please refer to the following pages of the official documentation:

Google Cloud Pub/Sub Triggers | Cloud Functions Documentation

Background Functions | Cloud Functions Documentation | Google Cloud

PubsubMessage | Cloud Pub/Sub Documentation | Google Cloud

Testing our Functions

The sample repository provides some helper scripts and files for running and testing our functions locally:

.env.local

Defines some environment variables, like function names and ports, so we reduce the amount of duplicated values and avoid running into the risk of forgetting something, in case anything is changed (keep it DRY):

Exported environment variables

run-local-http.sh

Executes the sample_http function defined at src/main.py using the functions-framework command that is made available after we install the functions-framework package with pip:

run-local-http.sh script contents

From the command above, notice how we:

1. Load the environment variables from the .env.local file;
2. Pass ../src/main.py as the --source argument;
3. Make use of the FUNCTION_NAME_HTTP environment variable to define what’s the target function to be served (sample_http, in this case) with the --target argument;
4. Specify the --signature-type is http. This parameter controls unmarshalling rules and determines which arguments are used to invoke your function. Check the functions-framework-python README file for more information about this parameter.
5. Make use of the FUNCTION_PORT_HTTP environment variable to define at which port it should listen to with the -- port argument;
6. Enable debugging with --debug.

Once the command is executed, this is what you are expected to see in your console output:

test-local-http.sh console output

Notice how the function is being served at the port 4000 defined by FUNCTION_PORT_HTTP environment variable sourced from the .env.local file.

test-local-http.sh

This helper script is really simple. All it does is to load the environment variables defined at .env.local and issues a GET HTTP request to localhost at the port identified by the FUNCTION_PORT_HTTP environment variable, passing FooBar as the value to the subject query string parameter:

test-local-http.sh script contents

To make development easier, it’s best to fire this command from a new console window, instead of running run-local-http.sh in background, so you can see its logs more easily.

Once the command is executed, you should:

• See the Hello, Foobar! message printed to the screen:

test-local-http.sh expected output

• See the GET request being logged on the console where the run-local-http.sh script was executed:

The GET request is logged in the console output

run-local-pubsub.sh

This is very similar to the run-local-http.sh script defined above. The differences are:

1. We use different variables for the --target and --port command arguments;
2. --signature-type is set to event instead of http.

run-local-pubsub.sh script contents

Once the command is executed, this what you are expected to see in your console output:

run-local-pubsub.sh expected console output

Notice how the function is being served at the port 5000 defined by FUNCTION_PORT_PUBSUB environment variable sourced from the .env.local file.

test-local-pubsub.sh

This test command is a bit more ellaborated than the test-local-http.sh described previously. It:

1. Reads the message passed as an argument to the command;
2. Encodes this message in Base64 format;
3. Loads the contents from the /payloads/test-local-pubsub-payload.json template file;
4. Replaces the __DATA_BASE64_PLACEHOLDER__ placeholder with the Base64 encoded message;
5. Sources the environment variables defined on the .env.local file;
6. Performs an HTTP POST request to localhost (at the port defined by the FUNCTION_PORT_PUBSUB environment variable), with the payload built on the previous steps.

test-local-pusub.sh script contents

Once the command is executed, you should:

• See an OK printed to the screen:

test-local-pubsub.sh expected output

• See the POST HTTP request being logged on the console where the run-local-pubsub.sh script was executed, along with the contents of the event and context parameters:

sample_pubsub function parameters are logged to the console output

payloads/test-local-pubsub-payload.json

A JSON template file for building the payload to test the Pub/Sub-triggered function. Notice how we have the __DATA_BASE64_PLACEHOLDER__ placeholder value defined, which, as described previously, is replaced with the Base64 encoded message.

Notice also that we have an attributes argument, which you can change according to your needs, depending on the payload that your Pub/Sub-triggered function expects.

As for the format of the JSON payload, I confess it was a bit challenging to find out how it should look like. It involved checking the docs for the PusubMessage object, as well as digging into the source code of the function-framework-python package and playing around with the values, until I was able to get the application to receive both the event and context arguments with the expected format and data.

Debugging Your Functions with PyCharm

After some research about how to debug the functions using IDEs, I found this open issue on the functions-framework-python repository.

The procedures described below were tested successfully on my PyCharm Community Edition. To do so, I followed this great tutorial from the GitHub user joelgerard.

Debugging the sample_http function

To avoid duplicating the values already defined on the .env.local file, launch PyCharm from the command line with the command source .env.local && pycharm-community.

With your project opened on PyCharm, follow these steps:

• On the toolbar, click Add Configuration…

Adding a new Run Configuration to PyCharm

• Add a new Python Configuration:

Adding a new Python Run Configuration

• Fill the fields as follows (make sure to adjust the values according to your local environment):

Run Configuration params for debugging the sample_http function

Configuring the Run Configuration for the sample_http function

• Apply the changes, select the newly created Run Configuration and hit the Debug button:

Starting a debugging session with the run-http-local configuration

• Open src/main.py, add a breakpoint to the sample_http function and run the scripts/test-local-http.sh script from a different terminal window. You should now be able to debug and inspect the state of your function:

Inspecting the state of the sample_http function with a breakpoint on PyCharm

Debugging the sample_pubsub function

The steps for configuring the PyCharm debugger for the Pub/Sub-triggered function are very similar to the described above. The only difference are the parameters set on the Run Configuration:

Run Configuration params for debugging the sample_pubsub function

Configuring the Run Configuration for the sample_pubsub function

Inspecting the state of the sample_pubsub function with a breakpoint on PyCharm

Really cool, right?

Final Thoughts

In this tutorial, we went through the process of setting up our local environment to develop, debug and test our functions locally, before deploying them to the Google Cloud Platform.

We saw how to we can debug both HTTP- and Pub/Sub-triggered functions, made use of some helper scripts that helped us during this process, and also saw how to configure PyCharm, so can have access to breakpoints and other great debugging features offered by the IDE.

Though I didn’t test it on VS Code, I found this sample repository from GitHub user pengelbrecht2627, which looks very promising. In case you test it, please share your experience in the comments.

It’s important to mention that, for the sake of not getting too long, we didn’t cover the process for testing Google Cloud Storage-triggered functions. Though I didn’t test it, I believe we could implement a similar process to what we did for testing the Pub/Sub-triggered function, but replacing the data attribute of the payload with what is described in the Google Cloud Storage Objects documentation.

What about you? What are the strategies you use for developing and testing your functions locally? I’d love to learn from you.

I hope you had a nice experience reading this tutorial and that you learned something along the way. If you think this tutorial was helpful, feel free to share your experience in the comments.

Photo by James Pond on Unsplash

References

• https://github.com/GoogleCloudPlatform/functions-framework-python
• https://cloud.google.com/functions/docs/writing/specifying-dependencies-python
• https://flask.palletsprojects.com/en/1.1.x/api/#flask.Request
• https://cloud.google.com/functions/docs/calling/pubsub#event_structure
• https://cloud.google.com/functions/docs/writing/background#function_parameters
• https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
• https://github.com/GoogleCloudPlatform/functions-framework-python/issues/32
• https://github.com/joelgerard/functions-framework-python/blob/pycharm/PYCHARM.md
• https://github.com/pengelbrecht2627/functions-framework-python-vscode