DevSecOps: Delivery
Secure delivery is all about how an application or product progresses down the release pipeline and mitigates the risk of exposure to vulnerabilities, detection of defects, and establishing traceability of significant events.
Do you have a *documented* release pipeline? Have you defined a governance model for the release pipeline — how should software get from User story to Production? These are the kinds of questions you must be able to answer in order to have confidence and trust with the business that what you are delivering is delivered in a secure way:
- What are the qualities we hope to achieve with the release pipeline? (i.e. traceability from ideation to production deployment, confidence that code has been vetted enough to mitigate the appropriate amount of risk, consistency in delivery among squads, a standard in branching/merging, immutable build output, and so on)
- Who deployed what, where, and when?
- What is the promotion path of how code progresses from local, development, test and through to higher level environments like, production? (in the cloud native context, the promotion path might be defined in modern constructs like Kubernetes namespaces and clusters)
- What people are involved and what do they do? (i.e. Squad leader performs code review then merges code, Operations provides secure/vetted base images for the squad to consume)
- What are the gates to get from one environment to another? (i.e. only workable code in the Dev environment, production deployments must involve a human/manual approval)
- What risks were mitigated as a part of a deployment? This is typically a story of what type of testing and/or scanning was performed, at what point in the promotion path and a catalog of the results. Immutability is also an important characteristic that must be proven. In effect this comes down to proving that what was deployed into production is what was actually built and tested, mitigating the risk of the “bait and switch” of executables.
Next, can you prove all of the above?
Even better, can you prove it, easily?
In my work with clients, some of these fundamental aspects often go undocumented or are “scattered” in various internal systems. Here is an example (client details obfuscated) of an MVP build out of a re-engineered release pipeline that describes the beginning stages of how applications should be delivered.
This example shows the initial elements of what I have described above:
- Definition of a promotion path (Dev, Test, UAT, Prod, DR environments)
- Traceability of code to a source code repo, which logs who committed code, who merged into the branch, and when it happened
- Pipeline tools being the only mechanism to progress releases through the promotion path
- Using testing and workflow approvals (for higher level environments) to mitigate the risk of defects being introduced and enforcement of Separation of Duties
- Immutability through the use of static “snapshots” being the carrier of release content in the promotion path
- Traceability of significant events through work item/ticketing tool integrations
- Definition of metrics to measure “success” to the client and expose visibility to meaningful delivery insights
Secure delivery is all about building in the observability, traceability, risk mitigation and compliance, for a release pipeline which is always, in motion. Consummability is key here. In a follow on post, I will elaborate more on Continuous Improvement, the use of telemetry, data collection, metrics and analytics to better understand patterns in your delivery.
The observability aspect is critical for auditors and ensuring compliance. Firms have been fined hundreds of millions of dollars, in the United States, for violating regulations and for being unable to prove their compliance. How valuable is observability to the business? Forbes has a compelling article that brings clarity to this issue. How much is your enterprise willing to risk on engineering observability and traceability?
After an application release makes it through the promotion path and lands in it’s production runtime environment, the “Day 2” activities of managing the application take top priority. The “Ops” side of DevOps is no place to relax when it comes to security. Observability continues it’s value as an application reaches it’s “runtime” glory. The Build to Manage instrumentation described above culminates in having the right monitors defined, the proper alert mechanisms in place, and meaningful reporting and dashboards to support audits need to be put in place. An extremely valuable and progressive approach in “Day 2” operations is using ChatOps to bring the *right* people together, with access to the *right* information, at the *right* time, when an incident occurs. I highly recommend the IBM Garage Method for Cloud and the Operate practices as a comprehensive set of practices to implement for modern operations.
If we were to view application delivery through the lens of security we would want to make sure that we understand the risk to the business of exposing the application’s functions, it’s data, and it’s integrations to it’s end users. There are compliance and regulatory issues that need to be engineered in the delivery of the application. I have covered some security-oriented activities that you might consider in your release pipeline, some might be new to your enterprise. As a “head nod” to Lean, we should always be asking “Why?” when it comes to examining how current state of the art is in design, delivery, and operations. I have justified these suggested activities in the name of observability, traceability, risk mitigation and compliance…the trade-off? VELOCITY. The answer to “Why?” should never be “this is just something that we have always been doing.” BEWARE. This is the mentality that often corners heritage thinkers into repeating processes from the past, just because “they were there”.
Question with boldness…everything.
Understand your governance process and everything that happens as a part of application delivery. Justify everything. Match it to a risk that is being mitigated. So, when asked “Why does integration testing occur today?” The answer is “Integration testing helps mitigate the risk of outage due to a dependent system or service consumed being unreachable or down and breaking my app.” Conversely, an adequate understanding of testing and scanning practices can unveil vulnerabilities that can be discovered. A survey of current practices might reveal that unit testing might not be a common practice, revealing a vulnerability and risk that non-workable code is making it’s way into the promotion path.
Question: When is it ok to slow down the release pipeline?
Answer: When mitigating the risk of unacceptable exposure to the business warrants the slow down.
Increasing quality is always a balance between: mitigating the appropriate amount of risk with velocity. I have a picture of some of the activities that are often done in a release pipeline that mitigate risk.
Question: What would happen if we didn’t do any of the things surrounding the pipeline in this picture?
Answer: With 0% focus on velocity and 100% focus on quality the result of the pipeline above would be stagnation/paralysis . With no tolerance for change…even a new feature (because that’s change!)…the business is unable to respond to markets and consumers.
With 100% focus on velocity and 0% focus on quality the result of the pipeline above would be a “junk factory.” No priority on quality with anything that enters the release pipeline and gets deployed exposes digital reputation of the business.
Success is having velocity AND quality as equal considerations in delivery. The extremes are the enemy of success, and, in fact, velocity and quality are proven to go hand in hand when it comes to successful agile delivery.
Be sure to check out my prior blogs on DevSecOps and Secure Design & Development. And if you haven’t done so already, check out 2019 DevOps Trends by Chris Lazzaro,
Bring your plan to the IBM Garage.
Are you ready to learn more about DevSecOps and delivering value with the IBM Garage? We’re here to help. Contact us today to schedule time to speak with a Garage expert about your next big idea. Learn about our IBM Garage Method, the design, development and startup communities we work in, and the deep expertise and capabilities we bring to the table.
Schedule a no-charge visit with the IBM Garage.
Do these ideas and concepts resonate with you and your digital transformations? Let me know with your comments below.
