Develop your Cloud Tasks pipeline locally with ngrok!

Task queues allow you to offload background work from your service by adding requests to a queue to be executed in the future. Using a managed service like Cloud Tasks has a lot of advantages, but it can be difficult to mimic the resulting task request coming from the queue when developing your worker services.

Example Cloud Tasks pipeline

ngrok is a tool to expose a local web server to the public internet with a handy UI to inspect requests sent to the server and the server’s response. In this tutorial, we will learn how to use ngrok to locally test a pipeline of adding tasks to a Cloud Tasks queue and processing task requests with a worker service.

** This method has inherent risks associated with exposing your network externally. In this tutorial we will use Google Cloud Shell’s sandbox environment to minimize these risks. This process can be replicated on your local machine with the risks mitigated by not revealing your public URL and minimizing the time your tunnel is exposed.**

ngrok is not the only option for SSH tunneling, there are other similar tools such as localtunnel.me, burrow.io, serveo, or you can set up self-hosted remote port forwarding.

1. Before you begin
2. Create a queue
3. Start your server
4. Start ngrok
5. Create a task
6. Clean up
7. Next steps

Before you begin

1. Create or select a Google Cloud Platform project.
2. Add a Google App Engine application to your project.
3. Enable the Cloud Tasks API and Datastore API.
4. Open Cloud Shell Editor. Cloud Shell provides built-in authorization for access to projects and resources hosted. If you are using a local machine, set up authentication to the API by creating a service account and setting the key as an environment variable.
5. In the Cloud Shell terminal:

• Download and unzip ngrok:

wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
unzip ngrok-stable-linux-amd64.zip

• Install the Python dependencies:

pip install --user flask google-cloud-datastore google-cloud-tasks

Create a queue

First, let’s make a queue to hold and send your task requests. Cloud Shell has pre-installed the Cloud SDK, which gives you access to the gcloud command-line tool.

1. Create a queue named “my-queue” using the following command:

gcloud tasks queues create my-queue

2. Make a note of some information about your queue: the project, the location, and the name. Identify these values by using the gcloud CLI:

gcloud tasks queues describe my-queue
# Output
name: projects/my-project-id/locations/us-central1/queues/my-queue
...

Start your server

Second, we will run a Python Flask app with endpoint /update_counter that will receive Cloud Tasks requests and update a value stored in Cloud Datastore.

1. Create a file named main.py and copy over the following code:

2. In the Cloud Shell terminal, start the app server using the following command:

python main.py

The Flask app is set to run on port 8080. We will also set ngrok to listen to this port.

Start ngrok

We will need to start ngrok before we create our first task.

1. Open another tab in your Cloud Shell terminal (➕) and start ngrok with the following command:

./ngrok http 8080

ngrok console UI

2. Make a note of the forwarding URL. In this example, the URL is https://2da88316.ngrok.io.

3. ngrok provides a real-time web UI, on port 4040, where you can inspect all of the HTTP traffic running over your tunnel. To open the inspection UI, click the Web Preview button in the top right corner, then select “Change port”. Enter 4040 and click “Change and Preview”. This opens http://localhost:4040 in your browser from Cloud Shell.

Create a Task

Now, let’s create and add a task to your queue.

1. Create a file named create_http_task.py and copy the following code:

2. Update variables project, queue, location, and url with the values noted above:

project = ‘my-project-id’
queue = ‘my-queue’
location = ‘us-central1’
url = ‘https://2da88316.ngrok.io/update_counter'

Make sure your URL retains the “https” and the endpoint “/update_counter”, so the request can be properly routed to the handler.

3. Now send a create task request targeted at the ngrok public URL:

python create_http_task.py

4. Look at the inspection UI. If the set up was correct, you will see a 200 OK response

You can also look at the details of the request and response, including the time, duration, headers, query parameters and request payload as well as the raw bytes on the wire.

You can replay the request in the inspection UI in order to reduce API calls.

Customize the request body and headers

The Cloud Tasks API enforces the task request body to be encoded as a byte array. The request body is added to the task dictionary under task['http_request']['body']. In create_http_task.py, the string variable payload has been encoded to a byte array with the method .encodes().

Try changing the Content-Type of the request, from the default “application/octet-stream”, by adding the headers field to the http_request dictionary. The Flask route can parse the request body based on this Content-Type.

Learn more about the Task object in your language of choice: Python, Node.js, .NET, Go, Java, PHP, Ruby.

Using Google App Engine targets

You must use HTTP targets in order to make use of ngrok’s public URL. Though you can easily change http_request to appengine_http_request and url to relative_uri to target an App Engine endpoint.

Using App Engine targets will provide extra features such as the App Engine Routing field to customize your request down to a specific service, version, and instance of your app and special headers.

Clean up

• Shutting down your ngrok instance using CTRL+C in the terminal.
• Shut down your local Python server using CTRL+C in the terminal.
• To avoid incurring charges to your Google Cloud Platform account for the resources used, go to the console and delete your project.

Next Steps

Learn more about the Cloud Tasks service in the Cloud Tasks Overview.