Change Risk Management: A Brief, Secure Addendum

Back in April of this year, we shared a little bit about our newly-minted Change Risk Management (CRM) process that we’d just rolled out. After a quarter’s worth of risk assessments, avoided outages (H1 2020 had 30% fewer disruptions than H1 2019), and more automated tooling, we realized that there was one aspect of change that CRM was agnostic towards: security.

Now in fairness, this was somewhat intentional. While we were crafting CRM, myself and a few other engineers were working on improving our security processes separately, and the separation was conscious so that we wouldn’t get in each other’s ways. However, we came to a point last month where both processes were mature, and it was time that they be wed. In this post, I’ll explain how we folded security into CRM, as well as what we learned along the way.

Championing Security

This picture was simply too good not to share. Borrowed from the good folks over at Secure Development.

In an organization of our size, scaling up our security efforts would’ve been quite the challenge. And without wishing to get too soap-boxy, writing secure code is as important as writing functioning code; a vulnerability should be treated exactly the same way we treat bugs. So security is everyone’s responsibility.

Bearing in mind how we democratized CRM, we applied a similar approach to security, building a volunteer-based security team that featured engineers from virtually every team in the organization. These “Security Champions” would be responsible (on a rotating basis) to review changes that involved security risks, identified by the teams making the changes.

This approach worked quite well, and as a member of this team, I’ve found that my nose for security concerns has strengthened quite a bit since we implemented this process. There are far more discussions on a regular basis surrounding security concerns, even when the Security Champion on-call isn’t notified. But now the question should be fairly obvious: how do security concerns affect the overall risk of a change?

What’s Another Rubric Among Friends?

Okay, so we didn’t actually implement another rubric. Instead, we simplified down all of the security questions present in our Software Development Lifecycle (SDLC) into these four questions, meant to be asked both before work begins and before a deployment:

1. Are you accepting new data from the customer or a third party?
2. Are you modifying any access control or authentication mechanisms?
3. Are you handling any credentials, secrets, or personally identifiable information (tokens, IDs, location data) as part of this change?
4. Are there new dependencies or infrastructure as part of this change?

If an engineer answers “yes” to any of the above questions, they have to notify the on-call Security Champion and request a code review of the change in question.

This fork of the process provided us with two main benefits. The first is that security is now entangled with risk management (as it should be), and the second is that the original CRM process could basically remain unchanged. Answering “yes” to any of the above questions doesn’t change the “rules of releasing” (again, check the last post for details on what these are if you’re interested), which means we didn’t need to retrain engineers or rewrite any of the automated tooling we’d already created (the release-ledger had a very fancy checkbox added to it to capture this data). And similarly, the preexisting security processes could continue to hum along as they were.

Pulling it All Together

Now that questions surrounding security are melded with CRM, we’re getting dangerously close to SDLC nirvana: a place where the path from ideation to deployment is given to you for free by automated tooling and simple processes. Undoubtedly, there will be additional forks we’ll have to add to CRM, and there will be evermore security and compliance questions we’ll want to fold in, too. But CRM has turned into quite the backbone for our deployment processes here at JW Player, and I’m excited to see what else we can do with it.

As an aside, I’d like to thank Matt Brennan for his help and guidance throughout both of these projects. He’s been a great mentor and we at JW have a lot to thank him for!