Secure DevOps #3 — Secure SDLC
This is the fourth in a series of posts that will outline a framework I’ve developed and successfully deployed for applying effective security to a DevOps environment.
Part 0 — Setting The Scene
Part 1 — Identity
Part 2 — Environment
Part 3 — Secure SDLC
Part 4 — Embedded Expertise
This part is crucial to your new secure DevOps capability, after all it is about the actual development bit. Most of the discussions I’ve had around this element have been very polarised — some people subscribe to the view that the development effort inherently can’t be controlled and that we should accept this and move the controls to the right in the deployment process. The other half subscribe to the more traditional security approach which attempts to impose strong controls on the process.
I fundamentally disagree with both positions— moving the control to the right might look like it achieves your objective, but in reality all it does is delay your involvement to the point that you risk it being too little too late. And attempting to lock down the development environment like it’s a production environment achieves little for security (they’re going to find a way around your controls) whilst simultaneously weighing heavily on productivity.
This framework proposes a slightly different approach.
As already outlined in the previous post, the first point is that we’re “hands off” with development — so long as they accept our deal.
But hands off doesn’t mean “no control”. We’re going to do it differently by leveraging what your development teams are already doing.
Secondly, we’re not moving the control to the right. Instead we’re embedding the control throughout the whole process such that when the project moves to the right, it’s already been living and breathing our controls and there are no late-day surprises built in by default.
Integrated Testing
Testing is part of any development lifecycle, agile or not. The difference is understanding how a modern developer actually tests as opposed to how a traditional testing approach might plan for that to happen.
Historically, using waterfall methodologies and RAD approaches, testing was envisaged as some kind of entirely separate phase to development; as if it were completely divorced from the actual development itself and as though it was only ever done in discreet chunks of activity.
What this failed to grasp was that as developers work they are constantly testing and iterating their code. Testing is innate to the process.
Source code control is basically standard now. Back when I was “developing” back in the day, there was no real source code control to speak of. We once dabbled with some kind of Microsoft solution and the net result was that it wouldn’t let us get our code back out and we have to manually copy and paste it back into project files so that we could use it again.
But not so any more.
Since we are presented with a natural bottleneck, and natural focal point, it makes absolute sense for us to exploit this for our benefit. Just as we did with identity and with our environment controls.
The client I worked with was managing their source code using Git. We made us of Git hooks to trigger post-commit scripts. These scripts depended on what was checked in.
For general source code files, a series of checks were made against the source code for common security problems — insecure function calls for example. We also checked for hard-coded IP addresses, and for certain obvious words like “PASSWORD”, “admin”, etc.
This gave us a very basic first pass of every single piece of source code that was committed to the repo. And it gave us that pass every single time there was an update. Since it was automatically triggered, the developer didn’t need to operate the control themselves, they simply followed their existing worklow. Allowing your developers to follow their existing workflow whilst simultaneously improving the security posture has to be the number one aim in implementing a successful DevOps framework.
Where Docker containers were being worked on, scans were triggered with the opensource Clair tool. This provided some basic level of checking for Docker containers.
We looked into many open source and commercial source code scanning tools. There were a handful of really good sounding options out there but at the time our testing and proofs of concept led us to the realisation that these technologies weren’t mature enough yet. They all essentially came with the overhead of needing a specialist to operate and tune the scanning tool and, more crucially, couldn’t be tightly and transparently integrated into the development work flow.
They did have some nice features though — like scanning-on-demand, which could be initiated by developers from within several popular IDEs — and I’ve been involved in some assessments more recently with a different client which appears to show that these tools have come a long way over the past few months so it may be worth revisiting whether you can integrate with a commercial tool if you believe it can bring some additional value to your secure SDLC.
But crucially in your assessments, you must hold firm to the requirement that these tools cannot impact the workflow of the developer. If the control relies on the operator operating it effectively then the control will fail.
Automated Scanning
Your security function probably already has a vulnerability scanner. If it’s a fairly modern one, then it probably has the ability to be invoked and managed via an API.
The principle that we should scan everything that is in our production environment has been around for a long time. Standards such as PCI-DSS mandate that your cardholder data environment must be scanned on a regular basis, for example.
There’s nothing inherently wrong with this idea — the scans are going to raise to us a bunch of security vulnerabilities and present us with some patching options.
The issue with it though is that it’s too late.
If it’s in your production environment then if there is an unpatched and exploitable vulnerability there then it is already too late.
Instead of waiting and rolling incident response, we can move this testing left in the process and improve our development process.
We introduced the idea of Known Good Builds in the previous article, and showed how this can reduce the effort required for security — everything that we deploy comes from this build, and if the build changes then we start again.
Through API integration, we can build a vulnerability scan into the deployment process for all server builds. When a deployment triggers and the build scripts or manifests are run, if the build is not a known build then the scan triggers and the results are made available to the developer.
Since this is happening at each build automatically, the first technically viable build of the environment and the application furnishes the developer with useful information about their project. And we’re still a long way from a production environment.
This automated vulnerability scanning provides a constant vigil against vulnerabilities in new builds.
To compliment this at-build scanning, we also maintained a regularly scan of the production environment (since not all of our infrastructure was being deployed in this way at the time). This proved to be a smart accident — because whilst we are assured that everything entering our environment has no unacceptable vulnerabilities, even with our 30 rebuild cycle ensuring evergreening of everything, one day our dashboard lit up with High vulnerabilities in production. The latest signatures in the scanning tool had identified a new vulnerability in software we were using. After researching it (as part of the normal vulnerability response cycle) it was identified that vulnerability already had a patch — and all we needing to do was to cycle the deployment to pick it up. It was a thing of beauty — watching each of the many production stacks seamlessly cycle, with no user impact, pick up the latest patching, and rejoin the live service in real time.
It was also quite special to see that initial vulnerability report come in with every single server hit by the exact same finding at the exact same time. This truly was commodity infrastructure.
Leverage Existing Testing Cycles
Whist earlier we spoke of testing being on a much more micro-level than pretty much all project frameworks understand, the existence of the testing phase isn’t actually a bad thing and is certainly something that we can and should leverage.
Due to its heritage in project life cycles, testing is one of those things that people seem to just understand. I’ve never experienced a scenario in a project where the testing team had to argue for testing to be included against a cacophony of naysayers insisting that the cost of such testing needs to be justified to the penny.
Compare that with security teams trying to get engaged and the contrast is stark.
So, as we’ve done many times before in this framework, we’re going to implement another control where it basically already exists.
A lot of the security testing that we would ideally like to do is very specialised and needs dedicated, skilled resource. I’m going to talk more about that in a slightly wider context in the final part of this framework in the next post.
But a lot of the testing is actually fairly straightforward, and is about the application logic and common faults.
There are now some pretty excellent tools out there which not only commoditise the testing, they do it in a really useful, natural language. I’m a particular fan of BDD Security which we had a very successful proof of concept implementation with at one client. This take security outcomes and states them in a simple series of assertions which can be automatically tested.
The reason I’ve listed this control towards the end of a deployment cycle instead of with the other testing at early builds is that this testing requires the application to be in a broadly finished state. We’re not testing the coding here, we’re testing the functionality and whether abusing that functionality is possible.
Summary
Now we have implemented a strong identity-based environment, utilising existing environmental controls to allow us to separate out or development and production assets, we’ve moved on to integrating testing and assurance capabilities into the natural lifecycle. Again we’ve taken advantage of several existing control points and bolstered them to meet our purposes — and all without having a detrimental impact on our development teams.
The next article will outline the final piece in this framework — skilled security professionals.
After that I’ll summarise the whole framework into something more digestible.
