Deploying Dapr on Google Cloud Run for Efficient Microservices Development
A distributed application run time on GCP serverless
Dapr Adventures on Google Cloud Run
So another writing on a small adventure to get Dapr running on Google Cloud run for a small internal project for our marketing department to get certain operational insights of our webshops (thesting.com, cottonclub.nl and costesfashion.com).
Another project, which should not be complicated in terms of ops but also not in terms of code. These were our first 2 requirements.
As we are a fan of Cloud Run, we wanted to get our hands dirty on a multi container setup as described here:
https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments
This multi container setup fits how Dapr works as well, as a sidecar proxy.
Why we want to play around with Dapr? Taking our requirements into account, Dapr should be a viable option as they mention this on their website.
Dapr increases your developer productivity by 20–40% with out-of-the-box features such as workflow, pub/sub, state management, secret stores, external configuration, bindings, actors, distributed lock, and cryptography. You benefit from the built-in security, reliability, and observability capabilities, so you don’t need to write boilerplate code to achieve production-ready applications.
Dapr? Ok let’s elaborate a bit on that one first
Dapr stands for Distributed Application runtime.
The whole thing can be found on https://dapr.io
In a oneliner:
Dapr provides integrated APIs for communication, state, and workflow. Dapr leverages industry best practices for security, resiliency, and observability, so you can focus on your code.
Ok, nice, that should save me some time on writing boiler plate code.
It gives a nice opinionated way to get an application working with the building blocks we needed: Persistence and PubSub, which by Dapr are provided as components.
Our Little Project
This is the small architecture that we want to provision.
Here’s the basic layout of what we wanted to build:
- Python Flask API: Handling requests and business logic.
- Dapr Sidecar: Bringing that Dapr goodness to the mix.
- Postgres on Cloud SQL: For storing our precious data.
- Google Pub/Sub: Helping processes communicate without getting too attached.
For persistence we will use Postgres Cloud SQL and for messaging Google PubSub. We have some processes running (Cloud functions with puppeteer) which dig through our webshops and we want to get notified if there are inconsistencies or we need to take certain actions. Hence the need for pubsub for a nice decoupling of the processes. In our case we want to know if we have product lister pages on our webshops which are effectively empty.
Although everything seemed straight forward, there were some rough edges in implementing and getting things running.
So in your Google project enable the applicable api’s and setup the things needed.
The Fun Begins
We started with setting up a Python Flask application. This app is our api which handles the requests from our React frontend application and contain the business logic.
Just like in our last article, Duet AI can really jumpstart your project, another timesaver.
Once you have the bare minimal Flask api setup, it is time to integrate Dapr. First download Dapr for the environment you are working on:
Homebrew is used for the installation of Dapr as we develop on Mac’s.
brew install dapr/tap/dapr-cliOnce installed you can verify your installation:
dapr -h
As Dapr is a sidecar process and your application runs in top of it, we need to install a package: The Dapr client SDK. This package will take care of the communication with the Dapr runtime from your Python application.
Just install the client sdk package with pip:
pip install daprBut wait, how does Dapr know where to store stuff. This is where the Dapr components (also referred as building blocks) come into play.
We need state management and as we want to use Postgres, we need a component definition for this.
In your application root create a folder called `dapr/components/`
In this folder create a file called statestore.yaml with the following content:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: empty-category-state
spec:
type: state.postgresql
version: v1
metadata:
- name: connectionString
secretKeyRef:
name: sqlconnection
auth:
secretStore: envvar-secret-storeHere we created a component of type state.postgresql (they have more flavours, but do pick one which fits your needs, as not all state components for instance have the ability to be queried).
The metadata.name is one you need to remember.
Also, as we don’t want to store database connection strings and password’s etc as plain text we use a secretStore.
So we need another component.
Create a file in the dapr/components folder, for instance secret-env-store.yaml, and add the following content:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: envvar-secret-store
spec:
type: secretstores.local.env
version: v1What this component does is it takes care of environment variable handling and exposes them tou your runtime. You could also have Google secret manager as a secret store, but in our case we can set these in the Cloud Run instance. As you can see in the component definition, we have a secretKeyRef called ‘sqlconnection’ which is an environment variable which will contain our connection string to the Postgres database.
Ok, we have Dapr configured , let’s dive into some code which will use the client sdk and connects to the Dapr runtime. In our main application code create a function to save to the state store. DAPR_STORE_NAME is the variable containing the metadata.name of the state storage component.
def save_state(data, key=None):
with DaprClient() as client:
state = json.dumps(data).encode('utf-8')
if key is None:
key = str(uuid.uuid4())
client.save_state(store_name=DAPR_STORE_NAME, key=key, value=state, state_metadata={"contentType": "application/json"})
This function will take some data and use the client to save it.
There was a caveat at this point. My data was stored as a base64 encoded string and flagged as binary. This data is not queryable. We needed to add the contentType to the state_metadata to store it as a key-value pair.
We have some simple data with a url, which webshop it belongs to and timestamped when the empty page was found. Here you see an example of some fake data.
Now we needed to get actual data into our small little system. As mentioned earlier we want to use pubsub for this. So Dapr needs to be enriched with another component.
In the dapr/components folder create a file called pubsub.yaml with the following content:
# This is a component file to configure the pubsub component
# This component is only use when creating a subscription
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: marketing-dashboard-events
spec:
type: pubsub.gcp.pubsub
version: v1
metadata:
- name: projectId
value: xxxxReplace the xxx with your actual GCP project id.
Also by convention Dapr expects a certain GET endpoint at /dapr/subscribe to get the pubsub subscriber info from you Python application.
@app.route('/dapr/subscribe', methods=['GET'])
def subscribe():
subscriptions = [{'pubsubname': 'marketing-dashboard-events', 'topic': 'marketing-dashboard-events', 'route': 'event_handler'}]
return jsonify(subscriptions)The topic to listen to is defined, but also a route.
This route gets invoked when a pubsub message is received, and guess what? Another endpoint is needed in your Python application:
@app.route('/event_handler', methods=['POST'])
def event_handler() -> None:
event = json.loads(request.data)
logging.info('Subscriber received: ' + str(event))
return parse_cloud_event(request)We defined a route named `event_handler` to take care of this.
As we use Flask, the raw data is in request.data property in the form of a CloudEvent. Dapr uses this structure for standard event handling.
We do like structure, so we will not accept every type of events. We use Avro schema’s to validate the incoming data. But it also is a contract for the other process which puts the data on the pubsub topic. We created a folder called schemas and a file within that folder called EmptyCategoryFound.asvc.
{
"name": "EmptyCategoryFound",
"type": "record",
"fields": [
{
"name": "url",
"type": "string"
},
{
"name": "formula",
"type": "string"
},
{
"name": "category_name",
"type": "string"
},
{
"name": "timestamp",
"type": "int",
"default": 0
}
]
}We need some other packages, cloud events and fastavro, so pip it.
pip install cloudevents fastavroTo handle this and verify the data we have another function in our application. This will parse the request with the from_http from the cloudevents package. The event type will match the filename in our schemas folder.
def parse_cloud_event(request): # Assumes Cloud Event in an HTTP request form
event = from_http(request.headers, request.get_data())
event_type = event['type']
schema_path = os.path.join(SCHEMA_BASE_PATH, f"{event_type}.avsc")
if os.path.exists(schema_path):
with open(schema_path, "r") as f:
schema = fastavro.parse_schema(json.load(f))
try:
json_data = json.loads(event.data)
validate(json_data, schema, raise_errors=True)
except Exception as e:
logging.warning("Schema validation failed , invalid payload")
# We log and return a 200 OK as we dont want invalid data
return {"error": "Schema validation failed"}, 200
save_state(CategoryDTO().load(json_data))
return {},200What this effectively does, we parse the cloudevent, check if we have a schema, validate the payload against the schema and if no exception is thrown we invoke the save_date function we mentioned earlier.
We can test the invocation by applying a message on pubsub in the google cloud console, by going to you pubsub page, goto the MESSAGES tab in press the ‘PUBLISH MESSAGE’ button.
{
"specversion" : "1.0",
"type" : "EmptyCategoryFound",
"source" : "/cloudevents/marketing",
"subject" : "123",
"id" : "A234-1234-1234",
"time" : "2018-04-05T17:31:00Z",
"comexampleextension1" : "value",
"comexampleothervalue" : 5,
"datacontenttype" : "application/json",
"data" : "{\"url\":\"https://www.nu.nl\",\"formula\":\"costes\",\"timestamp\":11111111, \"category_name\":\"empty\"}"
}If a message comes in it will be persisted through the client SDK.
Now that was fun. Let’s get some data out.
The Postgres component supports querying. So here is an example where we will get all EmptyCategoryFound’s stored in postgres.
@app.route("/api/formula/<formula_id>/categories")
def get_empty_categories(formula_id):
# return categories for a formula
with DaprClient() as client:
if formula_id not in ['costes', 'cotton', 'sting']:
abort(400, "Formula not found")
query = '{"filter": {"EQ": { "formula": "%s" }}}' % formula_id
result = client.query_state(DAPR_STORE_NAME, query=query)
categories = []
for doc in result.results:
cat = EmptyCategoryResponse().load(json.loads(doc.value)) #
categories.append(cat)
return jsonify(categories)We do some validation, create a query and apply it to our state.
Pretty straight forward.
Now we need to start the whole shabang.
As Dapr is a sidecar you can start it from the command line.
dapr run --app-id APP-NAME --dapr-grpc-port 50001 --dapr-http-port 3500 --components-path ./dapr/components --log-level debug --enable-api-logging --app-port 3000 We do have some important stuff here
— app-id, the name of your app
— dapr-grpc-port , the SDK primarily communicates with Dapr through grpc
— dapr-http-port , but it can also communicate through http defined on this port
— components-path, Dapr needs to know which components to load so you have to provide the folder containing the definitions for this.
— app-port, this is the port where your Python application runs on and this port is used by Dapr to get data from your application (e.g. in the pubsub case)
In your IDE (we use Pycharm) we can still run and debug our application like we are used to (or the way you prefer).
When you have everything working locally the next step is to get it to work on GCP. The logs are your friends , watch over them.
Here are some things we bumped into.
According to the Cloud Run documentation and the following picture we can project this on our setup.
In our case we have 3 containers. Our Python Flask app, the Dapr sidecar and the Cloud SQL proxy.
In this picture there is a shared volume which can also be used to mount with GCS to your Cloud Run Containers according to the documentation.
When playing around we only saw system.exit(255) in the logs and couldn’t get it to work yet, perhaps at this point in time it is still work in progress as it is in preview mode. The thought we had was to store the component configuration files in a GCS bucket, mount it to our cloud run container. This way we easily could deploy configurations during our Cloud Build process.
Bummer for now, but we will get back to that approach when it is out of preview. For now we created a separate Docker file for the Dapr sidecar where we copy the configuration into the custom Dapr sidecar image.
FROM daprio/daprd:edge
# copy the requirements file used for dependencies
COPY dapr/components/envvar-secret-store.yaml /dapr/components/
COPY dapr/components/pubsub.yaml /dapr/components/
COPY dapr/components/statestore.yaml /dapr/components/
COPY dapr/components/subscription.yaml /dapr/components/Downside is that you have to build 2 docker images. One for your Python Flask api and one for the Dapr Side car. But on the other hand, it does work. The Docker file for this app is relatively straight forward.
# Python image to use.
FROM python:3.9-bullseye
# Set the working directory to /app
WORKDIR /app
# copy the requirements file used for dependencies
COPY requirements.txt .
COPY app.py .
# Install any needed packages specified in requirements.
RUN pip install keyrings.google-artifactregistry-auth
RUN pip install --trusted-host pypi.python.org -r requirements.txt
# Copy the rest of the working directory contents into the container at /app
COPY . .
# Run app.py when the container launches
EXPOSE 8080
CMD ["python", "app.py"]Now just build the images, push it to your repository and reference it in a new file: service.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
annotations:
run.googleapis.com/ingress: all
run.googleapis.com/ingress-status: all
labels:
cloud.googleapis.com/location: europe-west8
name: marketing-dashboard-api
namespace: '1234567890'
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/minScale: '1'
autoscaling.knative.dev/maxScale: '3'
run.googleapis.com/client-name: cloud-console
run.googleapis.com/startup-cpu-boost: 'true'
labels:
client.knative.dev/nonce: fa399755-d3ad-4dfc-9319-344c122e7f47
run.googleapis.com/startupProbeType: Default
spec:
containerConcurrency: 80
containers:
- env:
- name: GOOGLE_CLOUD_PROJECT
value: xxx
- name: DAPR_GRPC_PORT
value: '3200'
- name: DAPR_HTTP_PORT
value: '3500'
- name: HTTP_APP_PORT
value: '8080'
- name: ENVIRONMENT
value: 'production'
image: gcr.io/xxxxxx/marketing-dashboard-api@sha256:...4d6b72
imagePullPolicy: Always
name: md-api
ports:
- containerPort: 8080
name: http1
resources:
limits:
cpu: 1000m
memory: 1Gi
startupProbe:
httpGet:
path: /api/health
port: 8080
initialDelaySeconds: 15
timeoutSeconds: 10
failureThreshold: 3
volumeMounts:
- mountPath: /dapr/components
name: DaprSharedVolume
- args:
- -app-id
- 'md-api'
- -components-path
- '/dapr/components'
- -dapr-http-port
- '3500'
- -dapr-grpc-port
- '3200'
- -log-level
- debug
- -app-port
- '8080'
command:
- ./daprd
env:
- name: APP_ID
value: 'md-api'
- name: APP_PORT
value: '8080'
- name: DAPR_HTTP_PORT
value: '3500'
- name: DAPR_GRPC_PORT
value: '3200'
- name: DAPR_HOST_IP
value: '0.0.0.0'
- name: sqlconnection
value: 'host=localhost port=5432 user=myyy password=zzz connect_timeout=10 database=md-db'
image: gcr.io/xxxxxx/dapr-runtime@sha256:...85e821721
imagePullPolicy: Always
name: dapr-sidecar
resources:
limits:
cpu: 1000m
memory: 512Mi
- name: cloud-sql-proxy
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:latest
args:
- "--port=5432"
- "xxxx:europe-west8:marketingdashboard"
serviceAccountName: xxxxx-compute@developer.gserviceaccount.com
timeoutSeconds: 300
volumes:
- emptyDir: {}
name: DaprSharedVolume
traffic:
- latestRevision: true
percent: 100The important thing is that the ports of the containers match. Also Flask normally runs on port 3000, but you can also get these from the Dapr settings in the client SDK, in our code we use the HTTP_APP_PORT to match up with the settings when we deploy.
if __name__ == "__main__":
#copy files to shared disk for cloud run, TODO: workaround, fix properly, GCS as volume doenst work yet
if(os.getenv("ENVIRONMENT") == "production"):
logging.info("** Loading components for cloud run in production")
shutil.copy('dapr/components/statestore.yaml', '/dapr/components/statestore.yaml')
shutil.copy('dapr/components/envvar-secret-store.yaml', '/dapr/components/envvar-secret-store.yaml')
shutil.copy('dapr/components/pubsub.yaml', '/dapr/components/pubsub.yaml')
shutil.copy('dapr/components/subscription.yaml', '/dapr/components/subscription.yaml')
from waitress import serve
serve(app, host="0.0.0.0", port=settings.HTTP_APP_PORT)We also tried to copy the component configuration in our Python application to the mounted disk. But this got us in a race condition.
If the Dapr container started before the Python container, the files were not in the mounted disk and the Dapr sidecar fails to start.
That was the reason why we went for the custom Dapr sidecar Docker container. We just copied the needed component files into a docker container.
In our service.yaml we also have a startupProbe. As we have multiple containers we do want to know if at least both started properly. Todo that we defined an endpoint in our Python Flask application and calls the Dapr sidecar health endpoint.
@app.route('/api/health', methods=['GET'])
def health():
#check the dapr healthz endpoint
try:
result = requests.get(f'http://localhost:{settings.DAPR_HTTP_PORT}/v1.0/healthz')
if result.status_code != 204:
raise Exception("dapr healthz endpoint returned an error")
return jsonify({"status": "ok"}),200
except Exception as e:
logging.error(e)
return jsonify({"status": "error", "message": str(e)}),500Now you can deploy to cloud run
gcloud beta run services replace service.yamlWe got a green deployment after applying , can fetch our data and receive messages from pubsub. Hip Hip Hooray! Now the only thing needed is documentation, unit tests and monitoring but that’s a different story.
Final Thoughts
Dapr was originally a Microsoft incubation project and was donated to the Cloud Native Computing Foundation in 2021. And as its from the Microsoft shed, Azure has a more native integration of Dapr, but this doesn’t mean we can’t run it on GCP. And although it was quite some trial and error we did get it to work!
We do like the concept of Dapr and getting it running on a serverless proposition. Keeps our hands free of writing business logic instead of Ops and boiler plate code. Happy coding and we hope you enjoined this writing and perhaps it even can have you jump started with Dapr on GCP :)
Cheers,
Martijn Schouwe, Tech Lead at the Sting.
