Panoptikum

A review on how we changed our work to Kanban to gain back control over work tasks two years ago. But not only that, with Kanban shifting from on-premises to home work during COVID-19 was easy for us!

Our Kanban board with natural WIP limit 😊😉

Some words about Kanban

We can find a lot of comparisons of Kanban and Scrum on the internet. If you a search for “Kanban vs Scrum” in Google, you can find many essays about this topic. There are pros and cons for both methods, but in my opinion, it does not matter what you use, as long as it work for you and/or your team. One of the most important points, at least for me, is, that you can use a Kanban board as a single person, as a team or both. And it is highly flexible in the way you can use it. There are no rules or crazy guidelines on how you should use it. Sure, there are some pinpoints you can consider, like the number of columns you use, Kanban pull signals or a work in progress limit. Alternatively, you can read a bunch of books about Kanban.

But at the end, that’s all theory, and the sole thing that you absolutely can do is to set up a Kanban board and keep moving! That's where you can learn most because true is what works! [watch this video!]

Two years of Kanban

In the following sections, I will try to explain why we launched our Kanban board and how we are operating it today. Maybe you can pickup some of these insights with you or, at least, you have some fun reading this story!😏

The starting point

Before 2016, my team only comprises two people, including myself. In 2017, another colleague joined my team. During the decade between 2008 and 2018, my team was always in one office. Communication was easy to handle because everyone could hear what was going on in which project. But, with the beginning of 2018, one of my colleagues moved to Vienna and my team split up. Before this point in time, we turned to use Kanban to get more transparency into our work about what we are doing all day long and to mange work tasks over distance.

Thus we started a new group called kanban inside our GitLab and opened a project called Hosting Continuous Integration which we use only for our Kanban boards and to handle our issues.

Image for post
Image for post
First ten ever created issues for our Kanban

The screenshot above shows the first ten issues we’ve ever created for our Kanban board. As you can see, we started in January 2018 nearly two years ago! Currently, it’s not easily possible to get this data from the GitLab UI directly. I will show you how you can get this data by some simple SQL statements later in this story. For now, the SQL statement to retrieve this list is:

select distinct on (created_at) created_at, title, iid from issues where project_id = 685 order by created_at asc limit 10;

Don’t worry, I am definitely no SQL expert! In the above statement, 685 is the project id of our Hosting Continuous Integration Kanban board. We will jump into the details of SQL for GitLab later. For now, let's have a closer look at our Kanban boards, issues and data.

The Kanban boards

Currently, we use five Kanban boards, called Operations, UC-Co-Responsible, UC-Product, UC-Responsible UC-Update. We will start with the Operations Kanban board, as it is by far the oldest one and the one which we use for the direct work control.

Image for post
Image for post
Our Kanban boards

We use the Kanban boards prefixed with UC- for our Update Cycle iterations, but more on them later.

The operations Kanban board

For the operations Kanban board, we use the three major labels kanban::accepted, kanban::todo and kanban::doing besides the two default one's open and closed.

Our major issue labels for our Kanban board

In GitLab, every time you move an issue from one column to another column, the issue will labeled automatically according to the column name. These major labels are so called scoped labels in GitLab and scoped labels are mutually exclusive! Scoped labels can help you tidy up your types of work even more. There’s one book which I recommend you to read, because it covers the idea of a Kanban board. It explains the types of work thinking, and it has a cool story to read. The book name is The Phoenix Project (A Novel About IT, DevOps, and Helping Your Business Win) and if you need something to read during Christmas holidays, pick up a copy!

Our Kanban board labels

As expressed above, we use a couple of labels to structure our issues and to structure our work. The following picture shows all of our current labels.

Our Kanban board labels

Why we are using this labels and how are we using them? Well, lets start with the simple ones.

Doing and To Do are the default labels of a GitLab Kanban board. We do not use them anymore, because they are not mutually exclusive. We would like to avoid that multiple labels of the same type can be putted on an issue. Therefore, we now use the labels kanban::accepted, kanban::doing and kanban:todo for the Kanban board columns.

Next, we use one of the work::outage, work::planned and work:unplanned labels to categorize the type of work for the issue. An outage is pretty clear — something stopped working. 😉 Normally, we have planned work. This means that the issues which we create came from a brainstorming, a common task, or something like that. If we create an issue, because someone calls us on the phone or by chat, we need to label the issue as unplanned work.

Then, we apply one of the priority labels, prio::1, prio::2 or prio:3. Simple and effective.

In addition, we have various other labels which we apply if needed. The k8s::bosnd-operator and k8s::general labels are for specific Kubernetes development tasks. nl::auto, nl::news and nl::open are used by our newsletter functionality. All issues labeled with one of these labels will automatically be part of our monthly newsletter. For some other colleagues, we have established additional labels like dev::ci, dev::se and dev::win.

At last, there’s a larger group with labels, which define our products, the responsible and co-responsible person for the product and the update cycle for this product. In our operations working world, a product is a software which runs in production to provide some kind of infrastructure service (e.g. Apache httpd load balancer, Puppet,…) or which we need to do our daily work (Ubuntu Linux workstation, various binaries,…).

(co-)responsible::bernhard, (co-)responsible::stefan, (co-)responsible::martin and (co-)responsible::mario (that’s me 😁) are the labels for the products and the products Kanban board. The labels update::unknown, update::annual, update::semi-annual, update::quaterly and update::monthly are setting the update cycle. To see how this works, read on in the next section!

The UC Kanban boards

As already explained, we use a bunch of labels for our Kanban board. Not only to steer our daily work business but to organize and control our periodical work. This is very important because you must update your software products (libraries, binaries, etc.) to stay on a supported version level. It does not matter if you are using Open Source Software products, bought software products or software which is covered by a maintenance contract. In either case, you must stay up-to-date with your software versions for security reasons! Sure, every company and every business has different requirements. However, you need a plan. Without a plan, you can’t handle all the work periodically and you can’t show it to others, managers or whoever is in the need of this information!

Kanban boards with different views based on product labels

The Kanban boards in the picture above illustrate our current views on our products. There are many of them (scroll bars) but with the use of labels, everyone can have a quick overview.

The screenshot in the top left corner (1) shows the kanban board with columns based on the product type, and in the top right corner (2) the screenshot shows the update cycles of the products. The co-responsible (3) and the responsible (3) screenshots of our Kanban UC-boards are at the bottom.

With this four views and the corresponding labels, the management of our products which we use in the operations team is straightforward and, at any point in time, we can explain everyone who is interested in what is going on and which software gets updated next. Each planned update is an issue, and the issue links to the related product via a “related to” link. Many thanks to my team colleagues who are always taking care about this perfectly! Here is how it looks like, by an example from Bernhard Rausch.

The CoreDNS product issue with the related update issues

This shows the genuine power of Kanban boards. Some labels in combination with a cool idea enable you to handle a lot of products with ease! But we didn’t stop here! In addition, we use the relates to and the is blocked by feature to visualize issue dependencies a lot.

How do we work with the boards?

To open up a common issue for our operations Kanban board, we use issue templates heavily. First, we use a default template for our Kanban board. If we click the “+” icon to open a new image, the template configured in the general section of the GitLab project is used.

Image for post
Image for post
Default template used by clicking on “+”

Our template looks like this. The /label, /weight and /epic short-codes will label the issue as we need it currently. But, we think about a change for this part for 2021 (read the lookout at the bottom of this story).

The default template within the GitLab project settings

Sadly, in this view, you cannot choose one of your issue templates which are part of your GitLab project. Currently, you must maintain this section separately. We think about opening an issuse in the official GitLab project for this, but this will need some time too.

Second, we’ve created plenty of issue templates to support the colleagues who need a service from us. The template directory also contains the update-templates which we use for the updates of the products.

Excerpt from our template repository

At the moment, we have 74 templates, and we use a simple naming scheme for them. Like with Git commit messages, we use the imperative form for the naming. But, in the future, if the amount of services still grows, we might switch to another naming scheme, where the product will be the prefix for the template name.

The templates are even more important because we can use them to bypass this odd guest user restriction in GitLab-EE. Currently, as of December 2020, it is not possible to have guest users in your GitLab installation, if you bought commercial licences for it, like premium or start licences. For this reason we implemented an operations dashboard, where all users of our enterprise can open an issue for my team if they need some service, regardless if they have a GitLab licence or not. 😁

The same issues as in the previous screenshot displayed in our operations dashboard

As you can see, we are using the same issue templates also for our operations dashboard. To do this, we’ve simply adopted the GitLab idea to build a webpage out of a static Git repository. My colleague Stefan Ringhofer created this dashboard and did the programming — many thanks! 💕 The dashboard is not 100% perfect, but aiming for perfection is not our goal anyhow — what we would like to achieve is, to provide a simple low-threshold way for our users to open up issues for our Kanban board. That’s our 💯!

A natural WIP limit

Working with a Kanban board means managing multiple tasks at the same time by using a time slicing mechanism. There is always some amount of work in progress and if you read about Kanban (or Scrum) you will stumble over the Work In Progress (WIP) limit, eventually. Pros and cons of WIP limits are depending on how you work and what work you have to manage. You can do really deep scientific calculations about a proper WIP limit or you can just ignore this topic. Both ways or any nuance in between are acceptable. 😊 For us, there is a natural WIP limit threshold, at least for me. And its simple.

If the issues in the doing column are reaching the bottom of my screen, I’ve reached my WIP limit.

If you are working with the Kanban board WIP limit this way, you’ve to think about the questions, if it should be possible to move issues backwards on the Kanban board. Some say that moving issues backward on a Kanban board is a no go. My opinion on this is, as with the WIP limit, that it depends of the type of work you manage with your Kanban board. If you are a developer, then the Kanban board (or Scrum board) will probably contain only issues (or tasks) already cleared by the management of the project. If you are mostly managing operation issues and are using the Kanban board also as a think-tank, then you might have to move issues backwards to the backlog. Why? Sometimes issues or task become redundant after some time, because the underlying system has changed, the environment changed, or there is simply no more time for that topic at the moment. So, yes, we are moving issues also backwards on our Kanban board. 😊

Numbers! Show me the numbers!

Ok, now let’s get into the numbers and some handy SQL statements to pull them out of the GitLab database. We use the GitLab omnibus installation on-premies. This allows us to connect directly to the database on the GitLab server. GitLab currently works on multiple enhancements for statistics gathering. If you are interested in some of these enhancements, you can upvote the issues on the GitLab project, for example this one.

You can connect to the database with the following few commands. First connect to your GitLab host. Second, impersonate yourself as the gitlab-psql user.

sudo -u gitlab-psql -i bash

Third, connect to the local running PostgreSQL Database.

/opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production

That’s it. Now you can issue SQL statements as usual. For the first chart, we need the opened and closes issues per month. We can do this by running the following SQL statements:

select count(distinct created_at) as count, to_char(created_at, 'Month') as month,extract(year from created_at) as year from issues where project_id = 685 group by 2,3 order by year, to_date(to_char(created_at,'Month'), 'Month');select count(distinct closed_at) as count, to_char(closed_at, 'Month') as month,extract(year from closed_at) as year from issues where project_id = 685 group by 2,3 order by year, to_date(to_char(closed_at,'Month'), 'Month');

In these statements, the project_id with the number 685 represents the GitLab project number of our Kanban board with all our issues inside. After the import of the output data into Excel, we are able to create a neat chart. 😋

Opened issues, closed issues, and the difference 😄

This chart shows, that from the 22nd January 2018 until the end of November, we’ve created 2964 issues and closed 2743. The reason for the difference between these two numbers is visible in September 2020. In September we’ve created issues for every of our products, as explained above and currently we’ve 98 product issues. If we subtract them from the open issue count, we see, that we currently have 123 open issues. You can find out the number of issues labeled with a certain label, by running the following SQL statement as an example.

select count(*) from label_links as t1 inner join (select id from labels where title like '%product%') t2 on (t1.label_id = t2.id);

The blue bars in the chart above shows if we are closing more issues (negative numbers) than we open (positive numbers). In November closed many open issues, because we finished a long planned revision work. The next statistic which is truly interesting is the burndown chart and the burnup chart.

Burndown chart and burnup chart

These two charts are very efficient, to get a quick overview about the current situation a milestone. In sum, you have many ways how you can get a lot of numbers out of GitLab. Some of them are already part of the GitLab statistics, other ones must be retrieved via database queries, but that’s not that difficult as you could see in this story. 😎

Lookout and takeaways

It’s now the end of the year 2020 and as always, my team and I shared some thoughts about our GitLab usage. We think that we will delete the ops label because today it’s simply redundant. During 2020 another team moved from our Kanban board where they were a part of to a separate issue and Kanban board and therefore we do not need this label anymore. Also, we will remove the dev label. Hey, we are DEVOPS 😂!

In addition, we will remove the yearly epic. This epic makes no sense, because we plan quarterly — one of the lessons we’ve learned from this year! The quarterly milestone will be deleted as well. We’ve learned that we ran our Milestones on a monthly basis.

Next, we plan to add two additional labels, probably named internal and external. As you have read above, we know have different opinions to open issues. Then we will clean up our open issues. Some of them are really old and they must be closed because no one really cares about it anymore. And we will also change our default issue template to remove the automatic epic labeling. Not everything is an epic 😉

Hopefully, you had fun reading this story and maybe you could take some benefits for using a Kanban board with you!🤗 Finally, to sum up this story and our last team meeting of 2020…

This year, Kanban rules the world for us!

Thanks Bernhard Rausch, Stefan Ringhofer and Martin Steinwender for being such wonderful and inspiring team mates and persons! 🤗

And to all other ones out there — Happy New Year! 🎉

Last edited on 26th December 2020

Speaker, GitLab Hero, Docker Community Leader, Author, Cloud Architect — 𝗜𝗺𝗮𝗴𝗶𝗻𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗺𝗼𝗿𝗲 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝘁𝗵𝗮𝗻 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲.

Get the Medium app