Extending OpenPolicyAgent for CICD Gates
I am a Software Engineer at IBM and this is another article on my, and my team’s, journey to a Cloud Native landscape.
Recently, we have been enhancing the CICD system that is a part of the IBM Automation Platform and the next phase was tackling the gate system.
Currently, we use the environment gates that are a part of IBM UrbanCode Deploy, however as we moved into a more cloud native and self-service implementation we wanted to have a more useable policy-based system. Think Kubernetes or IBM Multicloud Manager.
Enter the CNCF project OpenPolicyAgent (OPA). This works perfectly for the architecture, has great performance, and can be leveraged for other areas of the platform such as API authorization in the future.
Getting Started
To really understand whether OPA was a good fit, we needed to start a proof of concept. Initially I was hesitant with the Rego language — having no knowledge of it or its history. There is the REPL playground and some good language documentation, however, it’s always great to see real world scenarios and sample code. As ours was not a straight-up use case, we needed to find custom use cases and make a start, and Kevin Hoffman laid it out well in his article; Corrupting the Open Policy Agent to Run My Games.
The playground allows you to write the Rego and provide JSON input. I would highly recommend starting here and following the examples and the articles mentioned above.
We had already done some designs for the data models and tried to understand what policies and rules were in relation to CICD and our current architecture.
I decided it was as good a place as any and moved onto using these in further playground tests.
The first use case was to create a policy validating packages, and optionally the package version, referenced in the code. Using built-in object iteration to ensure it was the correct type of rule that we wanted to check, we compared the rule version and artifact values to the data and validated through regular expression matching.
For the case of playground, both the rules and data were inputs. (This is our first policy — there may be other ways to achieve this.)
package citadel.policydefault valid = falsevalid = output {
count(rules) >= 0
output := count(rules - rules_matched) == 0
}rules[output] {
artifact := input.policy.rules[i].artifact
version := input.policy.rules[i].version
output := concat("", ["^.*:", artifact, ":", version, "$"])
}library[output] {
output := input.data.components[_].components[_].component_id
}rules_matched[output] {
output := rules[_]
re_match(output, library[_])
}rules_not_matched[output] {
output := rules - rules_matched
}
The input we have split into two sections. We have the rules conditions, and the data to validate. The data is directly from JFrog Xray and is a list of all the packages found in the artifact.
"policy": {
"id": "12345",
"key": "package_safelist",
"name": "Package Safe List",
"description": "",
"order": "1",
"rules": [
{
"type": "maven",
"artifact": "commons-.*",
"version": ".*"
},
{
"type": "maven",
"artifact": "jackson-annotations",
"version": "2.[0-9].*"
}
]
},
"data": {
"component_name": "service.jar",
"package_type": "Maven",
"created": "2019-05-04T12:43:27Z",
"components": [
{
"component_name": "spring-security-core-4.2.3.RELEASE.jar",
"component_id": "org.springframework.security:spring-security-samples-preauth:4.2.3.RELEASE",
"package_type": "Maven",
"created": "2017-06-07T23:20:14Z",
"components": []
},
{
"component_name": "spring-boot-starter-aop-1.5.7.RELEASE.jar",
"component_id": "org.springframework.boot:spring-boot-starter-aop:1.5.7.RELEASE",
"package_type": "Maven",
"created": "2017-09-12T05:19:16Z",
"components": []
},
{
"component_name": "log4j-web-2.7.jar",
"component_id": "org.apache.logging.log4j:log4j-web:2.7",
"package_type": "Maven",
"created": "2016-10-02T06:31:10Z",
"components": []
},
{
"component_name": "jackson-annotations-2.8.0.jar",
"component_id": "com.fasterxml.jackson.core:jackson-annotations:2.8.0",
"package_type": "Maven",
"created": "2016-07-03T17:20:36Z",
"components": []
},
{
"component_name": "json-20090211_1.jar",
"component_id": "org.apache.geronimo.bundles:${pkgArtifactId}:20090211_1",
"package_type": "Maven",
"created": "2010-12-17T04:29:30Z",
"components": []
},
{
"component_name": "jackson-datatype-json-org-2.8.10.jar",
"component_id": "com.fasterxml.jackson.datatype:jackson-datatype-json-org:2.8.10",
"package_type": "Maven",
"created": "2017-08-24T00:19:22Z",
"components": []
}
]
}Moving to an implementation
Next, it was time to really see if it works outside of a public playground. I stood up the local Docker implementation and used Postman.
docker run -p 8181:8181 -v $PWD/data:/citadel openpolicyagent/opa run --server --log-level debug “citadel”:/citadelWhen mounting a folder with the policy and data in it locally, it appears the ID ends up having the folder name in it.
[GET] http://localhost:8181/v1/policies
{
"result": [
{
"id": "citadel/policy.rego",
...Lesson learned: The ID seen above is not the same as when you query the /v1/data endpoint, which uses the ID of the package name.
Lesson learned: Read the definition of what the objects become once loaded here. The biggest point is that once everything is loaded, it is all data of different types. This will help in understanding the data API.
This has simplified our time to implementation dramatically with the alpha release of our upgraded policies and gates going out in just two sprints.
It was also very easy to move this into our own applications Helm chart following the kubernetes examples in the deployment documentation.
The Implementation
From proof of concept to alpha, we created:
- A ReactJS front-end, allowing users to create and maintain policies
- A Java micro-service, performing the CRUD operations for the UI, integrations between systems holding the data and OPA for validation
- OpenPolicyAgent as the policy engine performing validation.
With this app, we can create five different types of policies:
1. Static Code Analysis: any metric that can be retrieved from SonarQube can be validated
2. Unit Tests: any usual metric for unit tests such as total tests, passed or failed tests, and coverage
3. Security Issue Analysis: validate numbers of security issues in Critical, High, Medium, or Low categories
4. CVE Safe List: validate the Common Vulnerabilities & Exposures (CVE)
5. Package Safe List: validate the packages/libraries and their versions
The following screenshot depicts the page where you enter the same rules I mentioned in the article, previously submitted via Postman.
See the whole user experience in the animated GIF:
Thanks go to my entire team, in particular (Marcus Roy, Tim Bula, Glen Hickman)!
