Cloud Native Compliance as Code
Evaluating your GCP resource realtime
How to build a service to validate GCP resource from CAI Feed
There are many ways to enforce cloud infrastructure compliance, Preventive Control targets Infrastructure as Code (IaC) CI/CD pipeline before it gets applied, this is good to stop upfront misconfigurations. Open Policy Agent Evaluate Infrastructure Score is a good example.
Detective Control validates the resource after the changes are made. Why do we need this? Because there are situations when manual or external changes remain undetected if we only scan the IaC.
This article focuses on building Resource Validation service, a realtime Detective Control validator using Go library Config Validator on Cloud Run.
Cloud Asset Inventory Feed (CAI Feed) is used to detect resource changes and send them to Resource Validation service via Pub/Sub. The following diagram shows the high level flow:
If you just need to evaluate your existing GCP resources once, no continuous detection is required, consider a batch detective control approach from this article Evaluating your existing GCP resources.
Detecting resource change
CAI Feed is an asset monitoring with feature to notify GCP resource change to Pub/Sub.
Before configuring CAI Feed, create a Pub/Sub Topic first, you can create its Push Subscription later when you have the Resource Validation service on Cloud Run is ready.
Plan for what type of asset you want to monitor, generally for policy validation you want to monitor all assets. To create feed for all asset types:
gcloud asset feeds create cai_feed_all \
--project=${PROJECT_ID} \
--content-type=resource \
--asset-types=".*.googleapis.com.*" \
--pubsub-topic="projects/${PROJECT_ID}/topics/${PUBSUB_TOPIC}"
Replace ${…}
with your Project ID, and Pub/Sub Topic you created ealier. At this stage, Pub/Sub Topic will receive the information when there is GCP resource change. Let’s create the validation service.
Resource Validation service
The purpose of this service is to evaluate your assets against Policy Library. Reuse your Policy Library if you already implemented it for Preventive Control.
This service is designed to validate asset change detected by CAI Feed, and you can expand it for bulk assets validation but we won’t cover that in this article.
If you just want to test the service without going through the nitty gritty, skip to Deploying to Cloud Run or Testing on Local Machine or simply checking out the code here.
Design overview
Service initialization is started with a typical HTTP server and configured to listen to port 8080 as default Cloud Run serving port. Policy Library which consists of policy constraint, template, and rego files is loaded to memory during this initialization.
When the service is ready, it routes HTTP POST requests receving Pub/Sub Message format from the Push Subscription. Data payload in Pub/Sub Message is base64 encoded, and the structure is CAI Temporal Asset. It contains Asset and PriorAsset, we only need Asset which has the latest resource attributes to be evaluated.
Config Validator is Go library to evaluates the GCP resources against the Policy Library, it accepts Asset as input and Asset Violations as output. Unfortunately the Asset required for Config Validator input, is not the same as CAI Asset, therefore we need to map necessary attributes before we can execute the evaluation.
At the of the process, you can choose to stream asset violations to console or persist it somewhere else eg. database.
Coding — step by step
Let’s deep dive into the code, starting with go.mod
in your main directory.
go.mod
module gcp-resource-validator
go 1.20
// Prevent otel dependencies from getting out of sync.
// Cannot be upgraded until k8s.io/component-base uses a more recent version of
// opentelemetry.
replace (
cloud.google.com/go/asset => cloud.google.com/go/asset v1.11.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp => go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.20.0
go.opentelemetry.io/otel => go.opentelemetry.io/otel v0.20.0
go.opentelemetry.io/otel/metric => go.opentelemetry.io/otel/metric v0.20.0
go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v0.20.0
go.opentelemetry.io/otel/trace => go.opentelemetry.io/otel/trace v0.20.0
go.opentelemetry.io/proto/otlp => go.opentelemetry.io/proto/otlp v0.7.0
)
replace (
cloud.google.com/go/asset => cloud.google.com/go/asset v1.11.0
)
require (
cloud.google.com/go/asset v1.12.0
cloud.google.com/go/pubsub v1.30.0
github.com/GoogleCloudPlatform/config-validator v0.0.0-20230328162739-ff3a6b2846d9
github.com/gin-gonic/gin v1.9.0
google.golang.org/protobuf v1.30.0
)
Note: as of April 2023 Config Validator must use some older opentelemetry and cloud asset dependencies. To overcome this, use ‘replace’ in your go.mod
We have go.mod
ready, to download dependencies:
go mod download
I use gin, you can use your choice of http server. First initialize the server to listen to port 8080 and a POST route /validator
. Create a new Validator
to load Policy Library files. It is hardcoded and will be statically built to the container image, you should load the files from Cloud Storage buckets for production implementation.
main.go
package main
import (
"log"
"github.com/gin-gonic/gin"
)
func main() {
router := gin.Default()
validator, err := NewValidator()
if err != nil {
log.Fatal(err)
}
router.POST("/validator") // No handler yet
router.Run(":8080")
}
validator.go
package main
import (
"log"
"github.com/GoogleCloudPlatform/config-validator/pkg/gcv"
)
type Validator struct {
gcv *gcv.Validator
}
func NewValidator() (*Validator, error) {
cv, err := gcv.NewValidator([]string{"./policy-library/policies"}, "./policy-library/lib")
if err != nil {
return nil, err
}
return &Validator{
gcv: cv,
}, nil
}
If you have existing Policy Library files, copy the constraints and templates to ./policy-library/policies
and the library files to ./policy-library/lib
.
If you start new Policy Library, you can clone Policy Library from github:
git clone https://github.com/GoogleCloudPlatform/policy-library
Policy Library from github has no constraint added, we can create ./policy-library/constraints/gcp_storage_location.yaml
to test the service later. This constraint will report violation with high severity if you have Cloud Storage bucket created not in allowlist
, in this case asia-southeast1
. This is optional step.
gcp_storage_location.yaml
apiVersion: constraints.gatekeeper.sh/v1alpha1
kind: GCPStorageLocationConstraintV1
metadata:
name: allow_some_storage_location
annotations:
description: Checks Cloud Storage bucket locations against allowed or disallowed
locations.
bundles.validator.forsetisecurity.org/healthcare-baseline-v1: security
spec:
severity: high
match:
ancestries:
- "organizations/**"
parameters:
mode: "allowlist"
locations:
- asia-southeast1
exemptions: []
The directory structure will look like this:
go.mod
go.sum
main.go
validator.go
policy-library
+- policies
| +- constraints
| | |- gcp_storage_location.yaml
| | |- ...
| +- templates
| |- ***.yaml
| |- ...
+- lib
|- ***.rego
|- ...
Now we have the service initialization part ready. You should be able to test running the server:
go run gcp-resource-validator
If everything is going well you will notice this at the console output last line:
...
[GIN-debug] Listening and serving HTTP on :8080
Ctrl-C to terminate the server.
Note: as of April 2023 if you use sample Policy Library here, there will be some warning message pertaining deprecated v1alpha1 constraint templates. You can safely ignore this.
Next let’s add the route handler.
main.go
…
router.POST("/validator", validator.Handler)
…
validator.go
...
import (
"context"
...
...
"cloud.google.com/go/asset/apiv1/assetpb"
"cloud.google.com/go/pubsub"
cvassetpb "github.com/GoogleCloudPlatform/config-validator/pkg/api/validator"
libCvAsset "github.com/GoogleCloudPlatform/config-validator/pkg/asset"
"github.com/gin-gonic/gin"
"google.golang.org/protobuf/encoding/protojson"
)
type pubsubMsg struct {
Message pubsub.Message
}
...
...
func (v *Validator) Handler(c *gin.Context) {
var msg pubsubMsg
if err := c.ShouldBindJSON(&msg); err != nil {
log.Println(err)
return
}
var tprAsset assetpb.TemporalAsset
protoUm := protojson.UnmarshalOptions{
AllowPartial: true,
DiscardUnknown: true,
}
if err := protoUm.Unmarshal([]byte(msg.Message.Data), &tprAsset); err != nil {
log.Println(err)
return
}
cvAsset := &cvassetpb.Asset{
Name: tprAsset.Asset.Name,
AssetType: tprAsset.Asset.AssetType,
AncestryPath: libCvAsset.AncestryPath(tprAsset.Asset.Ancestors),
Resource: tprAsset.Asset.Resource,
IamPolicy: tprAsset.Asset.IamPolicy,
OrgPolicy: tprAsset.Asset.OrgPolicy,
}
log.Printf("Validating asset: %s\n", cvAsset.Name)
violations, err := v.gcv.ReviewAsset(context.Background(), cvAsset)
if err != nil {
log.Println(err)
}
// process the output result here
...
...
}
pubsubMsg
wraps Google pubsub.Message
, we use this struct to bind to HTTP Post request data, and extract the payload. Unmarshal the decoded payload to assetpb.TemporalAsset
, and use its Asset
(not PriorAsset
) to map to ConfigValidator Asset
. We are ready to execute ReviewAsset
at this stage, and it will return list of violations as the output.
You can add final step to print the output to console, or insert to database. Complete code is available for your reference on the next section.
Deploying to Cloud Run
Prerequisite:
- Docker installed on your local machine
- A repository on Artifact Registry or Container Registry on your GCP project
- You have necessary access to your GCP project
Clone the code
The code is intended for demonstration only, not for production use.
git clone https://github.com/edyw/gcp-resource-validator.git
Build and deploy
Build the image locally, tag and push to container repository. Replace ${…}
with your setup.
docker build --tag gcp-resource-validator .
docker tag gcp-resource-validator ${REGION}-docker.pkg.dev/${PROJECT}/${REPO_NAME}/gcp-resource-validator:latest
docker push ${REGION}-docker.pkg.dev/${PROJECT}/${REPO_NAME}/gcp-resource-validator:latest
Deploy Cloud Run using the image from container repository.
gcloud run deploy gcp-resource-validator \
--image ${REGION}-docker.pkg.dev/${PROJECT}/${REPO_NAME}/gcp-resource-validator \
--region=${REGION} \
--allow-unauthenticated \
--service-account=${CLOUD_RUN_SERVICE_ACCOUNT}
Note: You should enable Cloud Run authentication and restrict ingress in production setup
At the end of the deployment, you will have a Service URL: https://gcp-resource-validator-xxxxxxxxxx-as.a.run.app
Pub/Sub Push Subscription
Go to the Pub/Sub Topic created earlier, and use it to create a Push Subscription with the Cloud Run Service URL + /validator
as the Endpoint URL.
End to end test
Trigger a change on your GCP resource to test the end to end flow. If you use the sample constraint gcp_storage_location.yaml, create a Cloud Storage bucket in location other than asia-southeast1 to test the violation output. For simplicity, sample code streams the violations output to console, use Cloud Logging to check the result.
Testing on Local Machine
To test the service on your machine, follow these steps.
Clone the code and download dependencies:
git clone https://github.com/edyw/gcp-resource-validator.git
go mod download
Run the server:
go run gcp-resource-validator
You should see this from the server last line:
...
[GIN-debug] Listening and serving HTTP on :8080
The sample code use this constraint gcp_storage_location.yaml located in directory ./policy-library/policies/constraints/
to evaluate if the location of Cloud Storage bucket is in allowlist (asia-southeast1).
To test this, use TemporalAsset data wrapped as Pub/Sub Message payload. In ./test-asset
directory, storage_location_us_msg.json and storage_location_sg_msg.json are json formatted Pub/Sub Message, the TemporalAsset payload is base64 encoded, you can refer to storage_location_us.txt and storage_location_sg.txt for decoded version.
Let’s test storage_location_us_msg.json, open another terminal:
curl -X POST localhost:8080/validator -d @./test-asset/storage_location_us_msg.json
You should this output from 1st terminal:
Validating asset: //storage.googleapis.com/bucket-test-1
Violation 1 (high-GCPStorageLocationConstraintV1.allow_some_storage_location): //storage.googleapis.com/bucket-test-1 is in a disallowed location.
Next test storage_location_sg_msg.json:
curl -X POST localhost:8080/validator -d @./test-asset/storage_location_sg_msg.json
No violation detected, the bucket is created in allowlist region.
Validating asset: //storage.googleapis.com/bucket-test-1
No violation detected