Continuous delivery for data protection by design and by default — Automating for Traceability, Visibility, and approval
In a series of articles, I will present a way of continuously delivering software while protecting the privacy of natural persons by design and by default.
Ensuring traceability and visibility are the two most important pillars for maintaining an application through its’ entire lifecycle and for doing data protection by design and by default.
The importance of automating for visibility and traceability
10 years ago Forrester Research introduced Traceability, Process automation, and Visibility as the three pillars of Application Lifecycle Management 2.0+. 10 years is a long time, but I believe the statement still holds true today due to the increased focus on privacy and security. In this article, I will highlight the importance that traceability, visibility, and automation has in regards to data protection by design and by default.
Visibility; is the degree to which something can attract general attention and prominence. The project's goals need to stay visible to the project’s team member in order to stay relevant. The project manager’s ability to reroute a project that is traveling a disastrous course relies on the continued visibility into a project’s current route, impediments, and status. In order to steer a project for compliance, visibility is needed into the projects ongoing challenges and activities for protecting your project’s system portfolio according to data protection by design and by default.
The term Traceability (or Requirements Traceability) refers to the ability to link product requirements back to stakeholders’ rationales and forward to corresponding design artifacts, code, test cases, etc. Traceability supports numerous software engineering activities such as change impact analysis, compliance verification or traceback of code and so on. Traceability is also important for auditing purposes to keep track of who, what, when and where. Without traceability, you may never be able to recreate or explain to others what you did to get your project to where it currently is as you do not know how you got here. Having traceability is critical in order to avoid repeating mistakes and to understand your current situation. It is, therefore, an integral part for the ongoing integrity of your project’s system portfolio in relation to data protection by design and by default and to ensure compliance with the rights of the data subject in accordance with article 13 and recital 61.
Relying on manual processes for ensuring visibility and traceability is a sure way of building your project’s grave. It’s highly recognized as an impossible task. (Kannenberg, Andrew; Saiedian, Hossein, 2009) The more systems you have, the more difficult it gets, which is why microservices come with their own set of tools for ensuring visibility and traceability. If you are building up a microservice architecture, I advise you to adopt the slogan “automate or die”. Not as a threat, but to communicate where the project is headed if the words are not heeded. It’s not a joke. I am dead serious. That doesn’t mean that you shouldn’t create manual processes. You need to model your processes before you can automate them. The manual route will come first. You might end up living with the manual process for the first one/two iterations before you automate them, but that shouldn’t be any problem. Just be sure to communicate that the process in question still hasn’t reached its end state.
I still haven’t been able to find an application life-cycle tool that does everything from requirement gathering, issue tracking, CI/CD and version control. Some vendors like Atlassian has come a long way, but you still need to create a list of plugins that you will be used to cover the rest of your security, traceability and visibility requirements. You also need to create a backlog of tasks that needs to be executed in order to stitch everything together. Your team will need quite a bit of time to set everything up, and they will have to continuously maintain your application life-cycle suite through the lifetime of your project. Doing microservice development will make the procurement or implementation of an application life-cycle management suite mandatory. Your microservice project will naturally fail without it. This is especially true if you additionally are using multiple programming languages across your services.
The application life-cycle management suite will be a prerequisite for being able to maintain a change management process for making changes to privileged environments because of the complexity involved in ensuring visibility and traceability. It doesn’t need to be there from day 1. In my current project, we have maintained traceability and visibility in our change management process for a long time by clever use of version control and issue tracking, Every commit has an issue number and every config-, code-, release- or -deployment is maintained by version control and can be traced to a specific scrum/kanban card and from there to the design document and eventually the initial requirements documented in Confluence. The change management process is held in place by having at least two senior team members vouch for any config-, code-, release-, or deployment change. It works, but it’s a lot to maintain and there is a lot of work left to be able to secure and automate reports to ensure proper visibility.
It’s important to stop to consider the number of microservices and programming languages. Think ahead! Try to analyze what your biggest obstacles for ensuring visibility and traceability across your teams are. Listen to testers, support operators, sysadmins and project managers. They might offer new and better insight into where you might lack visibility in your development processes.
Creating a change management process for privileged environments
To make changes, you need traceability from the next production release back to the original person that wrote the requirements. You also need to be able to trace the requirements and acceptance tests down to the specific program code, configuration change or deployment. Every development and test activity, deployment, config change, deployment, and release should be visible on a card and placed on a board. Changes could be applied through pull-requests approved by at least two senior team members to ensure that the development/test/deployment and release process is correctly followed and applied. Direct push to master should be disabled by default in accordance with doing data protection by design and by default.
Automate everything and automate everything well. Automate moving the kanban cards. As Soon as the cards are forgotten, the board won’t make any sense which makes team members question the current state of the system. When doing microservices, asking questions is a luxury you can’t afford. When writing requirements and design document, keep them version controlled in a wiki system like confluence in the same way as you version control code. Then, write automated tests to verify that the generated API documentation is correct and hasn’t diverged from the original design document. If the API documentation changes and diverges from the original API design document, go back and seek approval for changing the design document or correcting the API documentation with tests. Only after approval by the data controller, should changes be merged to trunk. Having code merged to the trunk does not necessarily mean that you are ready for a production release. Security and privacy reviews should be in order, delivered and approved by the data controller before the final push to production. Cloud version control systems like Bitbucket has buttons for merging and revoking pull-requests. Those can be used for deployment and rollback if needed and doesn’t cost a single minute to implement.
Approval of releases to privileged environments
According to article 32.4; “The controller and processor shall take steps to ensure that any natural person acting under the authority of the controller or the processor who has access to personal data does not process them except on instructions from the controller…”
The data controller, therefore, has to approve the processing before the release. Getting approval from the data controller can be done quite smoothly if the data controller is comfortable using a kanban system. Approval can simply be done by moving the cards on the kanban board. It’s important to be aware that approval is more than just approving a release and setting a maintenance window. The data controller is responsible for making sure that data protection is being done by design and by default. recital 78
An incremental test and approval process for your system is easier to execute for both you and your data controller compared to a big-bang release. A continuous approach will, however, require you to do security and privacy reviews on a sprint by sprint basis and continuously update your incident response plan. (I will come back to how that is done in one of my next articles.) If these processes allow for traceability and visibility into your implementation for doing data protection by design and by default, then giving approval should be easy. Feature toggles will ensure that you can toggle on only the features for processing of personal information that is ready and approved for production by the data controller. Feature toggles are also a good way of ensuring continuous integration and development during a production release, thereby avoiding code freeze and making it possible to stabilize the system quickly while addressing security- and availability issues. Feature toggling does, however, require central management, automation, visibility and traceability and your usual open source solutions used for feature toggling might not support this.
To conclude
As previously mentioned, you should make sure that two senior team members are required for approving any given production change whether it is a code merge, config change or deployment. Version control systems like Bitbucket can be set up with pull-request and branching restrictions to enforce the above-mentioned practice. Traceability, visibility, and automation are prerequisites in order to continuously deliver software while protecting the privacy of natural persons by design and by default. Remember that approving a release is more than just accepting the latest feature and agreeing on a maintenance window. Choose a continuous approach over a big-bang approach when security and privacy reviewing your software. If in doubt, seek advice from your peers.
References:
Kannenberg, Andrew; Saiedian, Hossein (2009). “Why Software Requirements Traceability Remains a Challenge”
