GCP Cloud Run sidecars with shared volume (Part 2) — implementation
If you’ve read Part 1, you have an understanding of why the solution I’m detailing below from an implementation standpoint looks the way it looks.
Let’s get cracking then!
Scripts
I chose to go with Python for writing my container scripts because I found it easier to do the needed work and also to keep a server up and running from the same main process.
Since Cloud Run doesn’t support Init containers (the functionality I needed), I have to run sidecar containers in parallel. And they both have to be alive, even though one can start after the other. To keep the container processes alive, I use the Flask framework to create hosts on the containers and keep small web applications alive.
Render script
My “init” sidecar will run the “Render script” which helps me read a Config file that has regular Shell Environment Variables templates ${MY_ENV_VAR}along with other static key-value pairs and checks the OS Environment Variables for matches and replaces them if they are found (and the expectation is they will be loaded inside the Container OS from Google Secret Manager).
It expects a specific Config file to be present in a specific location within the Container.
from flask import Flask, jsonify
import os
import re
app = Flask(__name__)
# source config file path
FILE_PATH = '/var/www/config.yaml'
# output config file path where the processed content will be saved
# and from where the main container will load it
NEW_FILE_PATH = '/mnt/config.yaml'
# status message for the exposed Flask app endpoint '/status'
status_message = "Processing not started"
def process_and_save_file_content(original_filepath, new_filepath):
global status_message
try:
with open(original_filepath, 'r') as file:
content = file.read()
# checking for ${...} type strings in the config file
pattern = re.compile(r'\$\{([a-zA-Z0-9_]+)\}')
# found hit will be searched in the OS Environment Variables
def replace_with_env(match):
env_var_name = match.group(1)
return os.getenv(env_var_name, match.group(0))
templated_content = pattern.sub(replace_with_env, content)
# output the rendered file in the output config file location
with open(new_filepath, 'w') as new_file:
new_file.write(templated_content)
# update the Endpoint status message
status_message = "Processing done"
except Exception as e:
status_message = f"Error processing and saving the file: {str(e)}"
@app.route('/status', methods=['GET'])
def get_status():
return jsonify({'message': status_message})
if __name__ == '__main__':
# first, process and save the file content
process_and_save_file_content(FILE_PATH, NEW_FILE_PATH)
# then, run the Flask app
# I don't really care about the port, I'll secure it in Google
# but I need a port opened for the mandatory Cloud Run Startup Probe
app.run(host='0.0.0.0', port=8181)Mock “main App” script
My “main App” script is a mock of the actual Application that will eventually be deployed to the Cloud Run instance.
For testing purposes, I want a web app that loads the final “rendered” Config file and displays it when I access the Cloud Run service revision URL. This allows me to check if my entire flow was successful, and see the process update when I make changes to the Config file or scripts.
It’s a placeholder for the Application that will later use this PoC.
from flask import Flask, Response
app = Flask(__name__)
# Path to the processed file
PROCESSED_FILE_PATH = '/mnt/config.yaml'
@app.route('/')
def serve_file():
try:
with open(PROCESSED_FILE_PATH, 'r') as file:
content = file.read()
# I can see my file's content in the browser at the Cloud Run
# service URL
return Response(content, mimetype='text/plain')
except Exception as e:
return Response(f"Error serving the file: {str(e)}", mimetype='text/plain')
if __name__ == '__main__':
# port 8080 is where my final Application will live for this exercise
app.run(host='0.0.0.0', port=8080)Dockerfiles
Each of my scripts will be packaged into an individual container image.
Render Dockerfile
# Use an official lightweight Python image
FROM python:3.9-slim
# Set the working directory in the container
WORKDIR /app
# Copy the dependencies file to the working directory
# The requirements.txt file contains:
# Flask==2.1.2
# Werkzeug==2.2.2
COPY requirements.txt .
# Install any dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy the content of the local src directory to the working directory
# This is where my 'init' script called 'main.py' is
COPY src_init/ .
# Specify the command to run on container start
CMD [ "python", "./main.py" ]I build and push this image like so:
# make sure you are at Dockerfile level
cd /path/to/my/Dockerfiles
bash
# authenticate to Google from the cli (I use Service Account impresonation)
gcloud auth login --impersonate-service-account=developer@<project_id>.iam.gserviceaccount.com
# login to Artifact registry
gcloud auth configure-docker europe-west2-docker.pkg.dev --impersonate-service-account=developer@<project_id>.iam.gserviceaccount.com
# build the Dockerfile (I call it Dockerfile_init)
docker build --platform linux/amd64 -t europe-west2-docker.pkg.dev/<project_id>/my-artifact-registry-instance-name/poc/cldrungcs:1 -f Dockerfile_init .
# push the image to Google's Artifact Registry service
docker push europe-west2-docker.pkg.dev/<project_id>/my-artifact-registry-instance-name//poc/cldrungcs:1Main App Dockerfile
# The contents and the requirements.txt contents are identical with the other
# Dockerfile
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# the only difference is where I keep my python script which is also called
# 'main'
COPY src_main/ .
CMD [ "python", "./main.py" ]And to push the image to Google, I use the same process as before, the only thing that changes is the image name cldrungcs:1 .
Resources
For provisioning the required resources in Google Cloud Run, I chose Terraform (+ Terragrunt) because the project already used these tools to provision Infrastructure.
I’m only going to go through the main.tf file so I won’t be using any variables in my code. You should adapt it to use Terraform variables if you need to use it.
# main.tf
# to make this work, I need to use the v2 Terraform resource for
# Cloud Run
resource "google_cloud_run_v2_service" "default" {
# and because some features are still in Beta, like GCS volume mounting
# or sidecar containers, I need to use the 'google-beta' provider
provider = google-beta
name = "poc-gcs-volumes"
ingress = "INGRESS_TRAFFIC_INTERNAL_ONLY"
location = "europe-west2"
# and the BETA launch stage ties into the 'google-beta' provider
launch_stage = "BETA"
template {
# I also have ti use the Gen2 execution environment for Cloud Run
execution_environment = "EXECUTION_ENVIRONMENT_GEN2"
service_account = "my-cldrun-svc-account@<project_id>.iam.gserviceaccount.com"
# I want to have one instance always on for the purpose of this PoC
scaling {
max_instance_count = 1
min_instance_count = 1
}
vpc_access {
# don't really need egress traffic but meh
egress = "ALL_TRAFFIC"
connector = "my_vpc_connector"
}
# my "init" sidecar container
containers {
image = "europe-west2-docker.pkg.dev/<project_id>/my-artifact-registry-instance-name/poc/cldruninit:1"
name = "init"
# I mount the GCS bucket object - my config file
volume_mounts {
name = "bucket"
mount_path = "/var/www"
}
# and I also mount the in-memory disk which is shared between sidecars
volume_mounts {
name = "empty-dir-volume"
mount_path = "/mnt"
}
# I did not link the Secret Manager instance as it was pointless for
# the PoC, but I needed to add an Env Var to the continer to be rendered
# in my template config file
env {
name = "CLIENT_SECRET"
value = "my_super_special_secret"
}
# this is where it gets tricky, the sidecar container, althoug doing
# a smal processing, needs to pass the Startup probe check and remain
# up and running alongside my main container
startup_probe {
http_get {
port = 8181
path = "/status"
}
}
}
# my "main" Application container
containers {
image = "europe-west2-docker.pkg.dev/<project_id>/my-artifact-registry-instance-name/poc/cldrungcs:1"
# this depends_on parameter allows me to set a container startup order
depends_on = ["init"]
# also mounting the same in-memory volume which will hold my
# templated config file
volume_mounts {
name = "empty-dir-volume"
mount_path = "/mnt"
}
}
# and this is where I refer my mounts to sources
# first source is the GCS template config file object
volumes {
name = "bucket"
gcs {
bucket = google_storage_bucket.default.name
read_only = false
}
}
# and the second is the in-memory destination rendered config file
volumes {
name = "empty-dir-volume"
empty_dir {
medium = "MEMORY"
size_limit = "50Mi"
}
}
}
}
# This policy allows unauthenticated access to my Cloud Run instance
# via the allUsers member
resource "google_cloud_run_service_iam_member" "invoker" {
project = "<project_id>"
service = google_cloud_run_v2_service.default.name
location = google_cloud_run_v2_service.default.location
role = "roles/run.invoker"
member = "allUsers"
}
# minimal bucket provisioning - the Org policy automatically makes and ensures
# my buckets stay private
resource "google_storage_bucket" "default" {
project = "<project_id>"
name = "poc-gcs-volumes-test-bucket-76125312"
location = "EU"
uniform_bucket_level_access = true
versioning {
enabled = true
}
}
# my Cloud Run designated Service Account needs bucket permissions to
# read from the bucket and for Cloud Run to mount volumes to objects
resource "google_storage_bucket_iam_member" "default-sa" {
depends_on = [google_storage_bucket.default]
bucket = google_storage_bucket.default.name
role = "roles/storage.objectAdmin"
member = "serviceAccount:my-cldrun-svc-account@<project_id>.iam.gserviceaccount.com"
}
# the default compute engine service account in my project also needs
# bucket access permissions (Cloud Run uses it)
resource "google_storage_bucket_iam_member" "default-sa-compute" {
depends_on = [google_storage_bucket.default]
bucket = google_storage_bucket.default.name
role = "roles/storage.objectAdmin"
member = "serviceAccount:<11-digit>-compute@developer.gserviceaccount.com"
}Other things
Test config file
I use a test config file that I upload to Google Cloud Storage
application_name: my_application
authentication:
access-token-uri: https://login.microsoftonline.com/azureford.onmicrosoft.com/oauth2/v2.0/token
# the client-secret value will be rendered by my 'init' script from the
# container's OS Environment Variable called the same way
client-secret: ${CLIENT_SECRET}
status:
enabled: true
features:
has_carrots: true
# dummy test
has_sugar: fal${e
has_milk: nullTo push the file to the Bucket
gcloud storage cp config.yaml gs://poc-gcs-volumes-test-bucket-76125312 --impersonate-service-account=developer@<project_id>.iam.gserviceaccount.comAnd really, that’s it, to test everything, I run terraform apply and access my Cloud Run service URL to check if I see my config file correctly rendered based on my Environment Variable value in my browser.
Hope this helps anyone trying to do something similar, and let me know if you have a better way of doing this :)