Quality and dev-ops at the bank

In 2016 I became a member of the team engineering the Collector Checkout product here at Collector Bank in Gothenburg, Sweden. I’ve spent most of my time here as a team lead, which means that I’m a backend engineer but it’s also my job to make sure we are a healthy team and that we feel accountable for the stuff we build.

This story is about the things we do to ensure we create value to our customers and at the same time taking full responsibility for the things we build. You build it — you own it!

Our board

We do a Kanban-ish process. Bottom up so the things at the bottom are the most important ones to close first.

We never pile bugs in a backlog. They end up directly on the board and are either fixed first thing or closed if they are not worth fixing.
We have deliberately used the word build over dev and QA over test because quality is a team responsibility and testers can help when we build stuff.
We are constantly experimenting with the board, this is why we have a white board and not only a digital one.
We try to visualize as much as possible to create a shared understanding of our work.
We have a specific lane for awaiting the outcome we are seeking, and a process to learn from our mistakes.

Our kanban board — always work in progress

Planning

When we plan our work, also called refinements, we try to embed traceability from the start. We ask ourselves questions like:

What is the customer/user value of this?
How can we measure the value?
How do we know we have succeeded?

Code reviews and pair programming

We encourage pair programming and always perform code reviews when we don’t.

This is a must have to create a quality culture and pair programming often makes our solutions better from the beginning since more brains on the problem tend to create better solutions.

Demos for the team

We often conduct small demos for the whole team in order to spread knowledge, but also to get early and valuable feedback. This is also keeping everyone involved so they feel that they actually own the product they are creating together.

Enablers

Automated Infrastructure

We run the entire Bank in Microsoft Azure. Now that was a bit of an achivement being one of the first banks ever doing that with all regulatory requirements that we had to be compliant with. But we did it!

We own our infrastructure enabling us to scale up/down and build new things without interaction with other teams.
We run code to create that infrastructure for us.
Everyone in the team can do this.

Continuous Deployments

To be able to move quickly we are using Octopus to deploy our product into production. This happens several times per day and is no big deal.

Everyone in the team can deploy.
We always have releaseable code, and we use feature flags to be able to ship to production even if we are not fully through QA.
A simple way to roll back helps us if we do bad things.

Traceability

For a system that handles monetary transactions it is extremely important to keep a good audit trail of everything that has happened. We are using EventStore as one of our databases, but we also log extensively.

We have one place where all logs go and a tool to easily search them.
We constantly tweak log levels to keep our attention on the important problems.
We log a lot! And you should log more than you think, and when you have done that — put some more logs in.

We use Elastic Search and Kibana/Grafana for our logs

Monitoring and alarms

Data and monitoring is a first class citizen in our team. It means that we measure almost everything we build in order to see if we achieve the outcome we are seeking.

The dashboards are central and is the first thing each team member goes through when arriving in the morning.
To keep an eye on the system while we are not at work we use PagerDuty and Slack.
Services such as Runscope and Cloudmonix ensures we are live from the outside world as well.