Optimising DevOps Team Efficiency: the Evolution of BestSecret Packs

Making a dynamic and fast-growing DevOps team efficient is no easy feat. Nevertheless, the journey of our DevOps team transformation into a fluid organisation has been underway for two years now, and it shows no signs of slowing down. My colleagues, Andres Hojman and Rhedil Gomez, and I, have been at the forefront of this transformation. And we have a story to tell. In this article, we will share our insights and experiences on how we achieved this remarkable feat.

The Input Parameters

Back in mid-2021 Cloud Services Platform team (in short, CSP) was a ‘traditional’ DevOps/Cloud Engineering team, recently formed to develop a reliable internal environment for the company to go Cloud. The team aimed to develop and maintain their own Cloud environments, facilitating the company’s shift from on-prem monolith to microservices architecture.

While the number of team members grew steadily, an external consulting company supported the team. The team used scrum framework to plan their backlog, set time-boxed goals for the sprint, performed bi-weekly reviews, plannings, and other traditional agile ceremonies. Naturally, we also had our daily standups. Even though the team was 100% remote, spread across several countries in Europe, the team efficiently used pair programming approach.

Our team had grown to the size of about 20 engineers from different backgrounds and cultures by early 2022. But as the size of the group increased, so did the length of our daily meetings. The team was working hard and following the guidance of external architects who set their goals and directions. It was a common challenge of coordination and communication: we were making progress, but we didn’t have a clear
vision. It was clear that the team could benefit from a reorganization.

The First Brainstorming

It was then that our Cloud VP, Siegi Polysius, urged us to explore different ways of organising our team. We examined various models of team structure, but we were especially drawn to the Spotify Tribes method: a system of autonomous units within the team, each focused on a specific domain of business. However, we realised that this method was too complex and impractical for our relatively small team.

We realised that we could not rely on the usual techniques. We had to come up with our own plan, suited to our team and its remarkable skills. That was the origin of the Packs idea.

The Rules of the Game

Our team transformation required us to consider the boundaries and limits of our new approach: did we want to adopt a rigid and conventional style, or to embrace a chaotic and anarchic one?

Neither option seemed appealing. We decided to find a balance between the two extremes: to match our team structure with the unique talents
of our team members, while ensuring efficiency and quality in our work.
It was clear, that whatever we come up with in our first iteration would be
adjusted by our team members and would go through numerous further iterations. We’d be iterating until we find the most efficient and comfortable solution, fitting the needs of our team and their super-powers.

Hence, it was only natural, that the first rule, implied by our organisational culture stated:

Nothing should be written in stone and we should not stick to any solid or unchangeable concept. Instead, we have to keep evolving and improving our routines. The fluid organisation, in other words.

We agreed to seek feedback on any changes we might have made and to prioritise the group’s consensus over our individual preferences. This was the only way, we believed, to find a balance that suited our team.

Areas of Responsibility

We are a classic DevOps/Cloud Engineering team, building and maintaining a Cloud platform for our company’s microservices running on AKS-clusters.

So we divided our work into six areas, based on logical separation:

1. Containerisation: managing AKS clusters, containers, deployments, etc.
2. Monitoring: ensuring a reliable and efficient cloud environment.
3. Security: protecting our systems from threats and vulnerabilities.
4. Data: offering various database and messaging solutions to our microservices developers.
5. Infrastructure as Code: deploying and maintaining the infrastructure
   level, following the IaaS model.
6. Reliability: setting up and maintaining the pipelines to keep everything
   updated, taking care of disaster recovery and high availability options.

Each Pack is responsible for one topic from the list above. We also analysed the tools we used and distributed them among the areas we had chosen for the Packs.

The Terminology

Let me briefly describe some terms, which we decided to use in our Packs team organisation.

a Pack — we adopted the term “Pack” for a group of engineers who shared an interest in a specific area of the cloud.

At BestSecret Tech, each team has a playful nickname. Ours is The CloudCats. We prefer the term pack over stray, even though cats usually form the latter.

an Anchor — each Pack has an elected Anchor — the representative of the Pack, attending the weekly Zookeeper Reviews (see the definition below). The Anchors facilitate the Plannings, Reviews and Dailies of their Packs. Anchors do also determine the strategy and priorities of their Packs.

As proxies, Anchors relay information between the Packs and the upper management, translating the strategy and ideas from the top to the pack members and adjusting the Pack’s plans, backlog, and strategy accordingly. Anchors are somewhat similar to pack leaders in real wolf packs, but the term is avoided to promote a horizontal team structure.

The Pack Shufflers — the three of us, who developed and implemented the
Packs structure. The Pack Consultants. Facilitate the overall team ceremonies like the Zookeeper Reviews, Plannings, Cycle Reviews, Retrospectives etc.

The Zookeepers — a special pack, including managers, architects, and the Pack Shufflers. The Zookeepers define the global strategy for the team, which is further propagated by the Anchors to their packs.

Zookeepers Weekly Review — the assembly of Pack Anchors, where they unite to share the latest updates from their Packs and discuss their strides towards meeting cycle commitments. This gathering serves as a hub for exchanging valuable information and insights, fostering a collaborative atmosphere where Anchors not only report on their own progress but also engage in lively discussions to enhance the collective wisdom of the group.

a Cycle —we organise our work in 6-weeks Cycles. On the first day of the Cycle, each Pack plans their overall strategy for the upcoming 6 weeks and presents it in a Zookeepers Cycle Planning. On the last day of the Cycle, the Packs present their achievements and progress in a Zookeepers Cycle Review, and the team has a chance to celebrate the success together. We make Cycle commitments, which define the scope of our effort for the upcoming 6 weeks. This helps us plan our workloads consciously.

Zookeepers Cycle Planning — the meeting happening on the first day of the Cycle, gathering all team members, where they share their cycle commitments and delve into strategic planning for the upcoming cycle. Led by the Pack Shufflers, this session serves as a forum for collaborative decision-making, blending each person’s input to chart a path toward shared goals.

Zookeepers Cycle Review — the meeting happening on the last day of the Cycle, when the Packs come together to showcase their achievements, conduct demos, and engage in reflective discussions on the challenges and situations encountered throughout the cycle. This meeting serves as a platform for collective introspection and celebration, encapsulating the journey traversed and the lessons learned along the way.

The Collaboration

The Packs Methodology was not meant to undermine the team spirit that we had cultivated, but rather to enhance it. Our goal was to foster that culture of innovation, teamwork, and mutual support.

The team members had the freedom to choose, collaborate, participate, and contribute to any pack (or several packs) that suited their interests and skills.

Initially we were thinking of some guardrails, such as the maximum number of packs one can participate simultaneously, the “notice period” before leaving the pack, the minimum and maximum numbers of pack members etc. But these “rules” were never really implied, as from the very beginning we could see that our team’s Pack system would be well-off self-regulating. Each Pack, again by trial and error, would define their optimal setup. The Packs Methodology was a learning process, where we experimented and discovered new ways of working in the Cloud.

In the outset we observed that some individuals joined multiple packs, but this was not a problem. They adjusted their membership according to their own comfort and efficiency. They knew how to balance their workload and interests, and how to contribute to different packs without compromising their quality.

We trusted our team and it proved to be a wise choice.

The Cloud is not a place where one can work alone. One needs to collaborate and be engaged with other areas and domains.

The Packs Methodology was not meant to separate us from each other, but rather to help us work more effectively. The Packs also needed to work together to ensure the successful delivery and productive work.

We also had to figure out how to apply the pair-programming approach, which our team liked very much, within the new Packs Methodology. This turned out to be the key to the Packs collaboration solution: whenever a task was marked as “shared”, we expected the members of the packs to pair up on it.

We can now very often see people from several Packs working together on a task in a pair-programming style: each bringing their own domain experience and skills.

The Packs are not isolated from each other, but rather connected by constant communication and collaboration. The information flows freely among the packs, as they share their insights and experiences in personal conversations and during the daily meetings.

With this in mind, we had to decide how to organise our Jira boards in the
best possible way, encouraging collaboration. We avoided creating separate boards for each Pack, but instead decided to tag each task with one or many pack names. We created the filters so that each pack could quickly access their “tailored” collection of tasks on the board.

We have never seen our Packs transformation as just dividing our team into several smaller ones. Instead, our main goal was to optimise our team organisation and make our daily work more efficient, comfortable, and productive. So that the team members have the freedom to take the lead in their areas of expertise.

The Constant Evolution

We started our team evolution journey from the very first Packs Methodology description, which was presented to the team and adopted without further edu. Our team members had the freedom to choose which packs they wanted to join, and which roles they wanted to play there: active or rather passive. The packs came up with the Anchors and Anchors Backups.

From then on, the packs had their own meetings, such as dailies, plannings, and reviews. This way we’ve significantly increased the efficiency of the team and reduced the amount of time previously occupied by attending the common, general, long-lasting, and not-always-exciting meetings. The Packs have defined their “dailies” schedule depending on the pack size, and how dynamic their workload would be.

We also asked Packs to come up with the Pack Manifesto, defining whatever guidelines they’d like to add — whether the Anchor role should be elected or rotated or whether a “notice period” to leave the pack is needed, etc.

Over time, the Packs have undergone continuous evolution in their routines and regulations, embracing preferred approaches while discarding those less favored.

Throughout these 2 years, we’ve been constantly adjusting our Packs
Methodology to the needs of our team and company in general, aiming at making our environment as conductive to team effectiveness and well-being as possible.

We’ve been through a lot of iterations, retrospective brainstorming sessions and discussions with the team, trying to address the emerging challenges along the way. Time is our best ally: we always assess the changes we make after a period of time and see if they are worth keeping or if we want to try something else.

For instance, opting to place the Reliability Pack in “hibernation mode” proved strategic — we didn’t entirely abandon it, just pressed pause. Over time, we recognised this decision as optimal for CSP’s needs.

Summary

We could delve into numerous examples of how CSP iterated and fine-tuned its setup while aligning with our North Stars. However, it’s crucial to remember that this process is highly tailored to each team, its unique business domain, and organizational structure.

As previously emphasized, a cornerstone of our philosophy is the refusal to etch anything in stone. Since the inception of our Packs Methodology, countless revisions have occurred, reflecting the fluidity inherent in our approach. Central to this evolution are the insights gleaned from our Retrospectives, serving as the wellspring for new ideas and enhancements.

However, it’s imperative to recognise that the Packs Methodology isn’t a universal panacea. While it seamlessly integrates into progressive and forward-thinking environments, offering ample room for innovation and experimentation, its applicability may vary.

For instance, in more traditional and regulated sectors like banking or finance, adaptation and customisation are essential to align the methodology with the industry’s unique nuances and constraints.

In the modern world it is vital to be open to new ideas, new mindsets, as things change over time. New times require new solutions, especially in such fast-moving industry, as IT.

So, to sum up everything once again:

The Pack Methodology is one of the ways to optimise your team routine to work smarter, not harder.

It’s all about finding your own groove and making it work for you. Experiment with new ideas, test them out, and get feedback. Don’t be afraid to fail and learn from it.

Keep what works and ditch what doesn’t.

This article wouldn’t be complete without expressing gratitude to the people, who supported and keep supporting us in our constant evolution journey:

• first of all, our wonderful CSP Team. Go CloudCats!
• our Director of the Cloud Platforms, Milos Radovanovic,
• our Cloud Vice President, Siegi Polysius,
• Aaron Bell, who supported us as a Project Manager at the very beginning of the Packs journey!
• everyone working for BestSecret!