Securing your Workload by Managed Dependencies
Running a micro service workload offers a lot of benefits that makes us move faster and to make changes with limited risks. If a release of one service introduced an error, the entire site won’t come crashing to a halt.
But it also creates a few challenges and in this article I will focus on one of them, how to keep your workload secure and up to date with the latest patches.
Zero Day Attacks
A challenge in today's software world is that the time between the information about a potential exploit is released and when it has to be fixed in production has shrunk. A lot.
Good examples of this are Log4Shell and Spring4Shell which where both massively abused by different actors on the Internet; investigations shows that the abuse even started before the security vulnerability was publicly disclosed.
So if we assume that we have a day to patch our most critical systems, and preferably all; how do we design our applications to support this?
Managed dependencies
In this example, I will go through a hypothetical scenario with a workload of Spring Boot applications. This workload of course would rely on the public releases of Spring and Spring Boot from Pivotal and we will also assume that our company has a few internal libraries that take care of common tasks that we reuse between applications. We might end up with something that looks like this
A bit lengthy but to summarize
- We rely on Spring Boot 2.2.6.RELEASE
- We have two internal, fixed version, dependencies, predictly-analytics-framework and predictly-security-framework
- We have two external, also fixed version, dependencies, springdoc-openapi-ui and jib-maven-plugin
All of these can contain vulnerabilities and we have to be able to respond quickly. So what prevents us from keeping this application secure?
Keep up with the official releases
This is probably the most pressing issue in this instance. The application is years behind the official release schedule and there almost certainly would be breaking changes in the upgrade path. The reason for this could be many, lack of time, no governing process, etc.
Internal frameworks rely on official releases
Since we have built some common libraries, these also rely on the official releases since they would extend Spring and Spring Boot with our internal best practices and patterns.
Even if I wanted to upgrade this service, I would still be stuck waiting for a release of our internal frameworks that are guaranteed to work with the new version of Spring Boot.
Locally managed dependencies
The service also has locally managed dependencies, e.g. springdoc-openapi-ui. Since this is a great development tool to have in our services, we could expect this to be present in a lot of other service as well and potentially in many different versions.
Scope of change
Finding and patching all our micro services will take a lot of time since we might have more than 50 different services in production specifying the current dependency.
This final point is worth considering one more time. The amount of work required per service per release will in the end be a major driver prohibiting us from keeping up with the official release train.
Design for maintainability
What we need is a design that keeps the amount of work per application to a minimum and a way of working that guarantees quick and easy adoption of new releases once available.
Centrally managed dependencies
To successfully keep track of what versions are being used in production and to quickly upgrade, we need a centrally managed list of releases that other systems can build from. Since we are using Spring Boot, we can extend their existing solution for this.
In this project, you can also include other dependency management projects such as spring-cloud-dependencies and spring-cloud-gcp-dependencies etc. By making this available, applications can drastically reduce their local configuration to just what they need to run.
As can be seen here, the local application now only specifies one version and that is that of the parent. The team that owns and maintains the parent project is tasked with the responsibility to guarantee that the versions of Spring and Spring Boot and our internal libraries are in sync with each other and that they work. They are also tasked with providing upgrade instructions between minor and major releases.
Release procedure
A release train would look like this
- Pivotal releases new versions of Spring and/or Spring Boot
- The internal libraries are upgraded to match and tests are run
- The parent project is released with new versions of both Spring Boot parent and internal libraries
- Each micro service is upgraded to the latest version of the parent project
As you can see, a release and an upgrade is reduced to just upgrading to the latest parent project version and this should guarantee that you are good to go. This should then be streamlined further by using a tool such as Dependabot to automatically check for upgrades and create pull requests.
Central governance v.s. local autonomy
What we have created is a list of curated frameworks including versions that we should expect our teams to rely on. The rationale for this is security and being able to easily and quickly respond to both regular and critical patch releases.
But does this not kill developer creativity and productivity? The answer is that it might, but I haven’t found that to be the case. If a developer want’s to introduce a new framework, they can still do it in their local application by just specifying that dependency. They can even skip entire releases of the parent project or override dependency versions locally should they so desire.
However, this is most likely not what you want. Instead, you should promote a behavior where new libraries and utilities are pushed to the parent project so that it is easy to see what is actually running in our micro service workload. This will of course lead to a very large pom.xml in the parent project over time but since all it does is specify and keep up with versions, it doesn’t actually impact the actual micro services.
What becomes critical now is to find a way of working that creates new releases of the parent project with a regular cadence that the teams can keep up with and a routine to enforce upgrades when critical exploits become known and patches available.
