Visibility in a blameless culture leads to process improvement

Setting the scene

Blameless culture is a culture where we avoid placing blame on an individual but rather seek reasons for failures in other factors like poor design, environmental factors etc. In such cultures, individual contributors feel safe, eagerly report failures and seek solutions rather than scape goats. The table below shows four levels of blamelessness in that may describe your culture:

The opposite of blameless culture is what is sometimes called pointing fingers. In this model every failure is attributed to an individual that is considered an owner, or even a cause of it. Assigning one’s name to a failure may cause them to become defensive and less eager to cooperate on finding solution and covering for their actions. The morale hit is an obvious thing and it’s not a culture we believe in at GSK.

So how does one create accountability in a blameless culture? Visibility.

I joined a project where the product was a multilingual, multimarket portal for healthcare professionals. My job was to design, implement and maintain continuous integration and continuous delivery. What could go wrong? Where was the catch?

Problem found and named

I setup a Jenkins automation server and scanned source code repository. This took a long time, upwards of 1.5 hrs. — unacceptable in continuous integration terms. I investigated the repository and discovered the issue — there were over one thousand open branches and over two dozen open pull requests. Most of them abandoned and hadn’t changed for months. A common practice in computer science is to reduce the problem before actually solving it. I counted the branches, found that over 90% of them are useless and kindly asked developers to remove these unused branches. As you may expect this never happened. Every programmer had more important things to do than cleaning up the repository.

It works. Why change?

Some would say, “I am not affected by cluttered repository”. As I was the one that complained about cluttered repository, I decided to clean it up myself. With a handful of scripts, I identified branches where no changes had occurred for months. I presented that list to the team and asked permission to remove them. Permission was granted — happy days. The number dropped drastically from 1000 to around 100, which was acceptable for me. I could get a list of all the branches with just two API calls with pagelen=50 rather than twenty.

But had I solved the problem? The short answer was no. The repository was cleaned but the root cause remained and in a few weeks time I faced the same problem with over 300 open branches. Where did I go wrong? I targeted the result rather than the cause, which was a custom of opening feature branches, developing, merging code and never removing the merged branch. This was the thing to be combated and I pondered how this could be done without violating omnipresent blameless culture.

Blameless Culture vs Responsibility Matrices (RACI)

My targeted thinking brought poor results in the long run, so I decided to try it another way. I recalled one of the Project Management trainings on the concept of responsibility matrix known to many as RACI (Responsible, Accountable, Consulted, Informed). In this concept it is absolutely crucial that you have an accountable person for everything. Just one person. This is the ultimate killer for dissolved responsibility phenomena.

I thought that the problem can be decomposed into many smaller ones where every developer had some contribution, and this could be measured. I decided to give every single branch ownership — from now on the developer that commits last to a branch is considered its owner. Then I counted how many open branches each developer owned and put it in a Grafana dashboard and sent it to them:

At this point, I could not say we were in blameless mode. Blame was placed, but no one was punished. What was most important was that everyone knew who was going to take action.

A little miracle

After placing the ‘Wheel of Accountability’ on the dashboard, developers managed to remove over 100 branches without any automation in under an hour.

The urge not to be considered ‘the great repository polluter” was that strong.

The problem is solved…

For now. Developers have developed a habit of removing merged branches. Branch count is measured and displayed continuously on the dashboard and is kept at levels I could previously only dream of. I now keep the ‘Wheel of Accountability’ on our dashboard for the team to see. Creating a culture of visibility resulted in accountability, awareness of the implication of dead branches and ultimately process improvement.

References

Why companies can benefit from blameless culture, by Rui Su
Postmortem Culture, by Google
Blameless Culture, by Jason Smale
Fundamental attribution error on Wikipedia
Bystander effect on Wikipedia
Diffusion of responsibility on Wikipedia