Technical chapters at Mindsay

As the back-end chapter lead at Mindsay for several years now, I often find myself explaining the role of the chapters to developers at Mindsay as well as colleagues outside of our tech team, or just generally to other developers or managers working in tech. The notion of chapters within tech teams is pretty common in organizations. They’re also sometimes known as guilds. The goal of this article is to share more broadly the culture of the technical chapters at Mindsay, from how they were created to their current organization and the strengths and weaknesses they face.

A few years ago, before the introduction of chapters, the Mindsay tech team was organized into feature teams. These teams were composed of 3 to 6 people who were fully autonomous and in charge of a certain part of the product (for example, one team was in charge of the bot building experience, another was in charge of the Analytics features, etc.). We had 4 such teams, each one with front-end developers, back-end developers, and a product manager.

As these teams were mostly autonomous on their domains, they organized themselves internally: the back-end and front-end developers of a feature team decided on the guidelines they wanted to follow, like what libraries they would use for HTTP requests or how tests would be written. But as this happened organically and independently in each team, we were not always choosing the same technical solutions. This led to a disparity in the way of doing things that slowed down collaboration between teams. It also made it harder for developers to move from one team to another as they had to adapt to new guidelines. Some teams became more experienced than others in some domains (testing, writing modular code, using services like RabbitMQ or Redis), and this experience was not naturally shared across teams. Finally, we had no overview of the technical robustness of our stack, as each team was making its own choices on its own codebase. Some were inadvertently privileging faster short-term developments over long-term robustness and maintainability.

After a few months, it was clear these issues would only amplify and that we needed a better organization to allow us to share experiences and advise teams on their technical choices. Two chapters were thus created, a front-end and a back-end chapter, that would exist alongside the feature teams to tackle those issues.

The chapters’ objectives

The chapters’ most important mission is to share experiences between teams and make sure every developer is aware of the best way of doing things. This may be done by writing guidelines or pair-programming with developers of various chapters, for example.

With time, we also recognized that there were some important technical tasks that were not in the feature teams’ responsibilities, so these responsibilities were assigned to the chapters. One example is monitoring the global robustness and ease of maintenance of our code and of our stack as a whole (some points also being shared with our SRE team). The chapters are also advocating for technical refactoring projects to be implemented when our situation is deemed too risky.

How are chapters organized?

Chapters are composed of one developer from each feature team, and one person is appointed as chapter leader.

Chapter members meet every week to discuss technical subjects that regularly come up in their feature teams. It may be technical pain points or uncertainties about the code itself or our process of writing code. The result of the discussion is often a guideline that is then shared with the whole tech team.

Other subjects may be technical risks that are known about the team’s services (for example, it may be a memory leak that could bring the stack down someday). Those risks are estimated. Solutions to solve or mitigate them are proposed. Finally, all the risks are sorted in order of priority so we easily know which ones should be tackled first.

If addressing these risks takes only a few hours, chapter members can take some time to do it themselves. About 20% of their time at Mindsay is dedicated to performing chapter work. For risks that require more work, the matter is discussed among the whole tech team. If the risk is high enough or related to certain business priorities, a feature team will dedicate specific time to address the issue.

What impact have chapters had at Mindsay so far?

Here are some examples of the impact of the back-end chapter at Mindsay.

In terms of guidelines, we now have enough guidelines for a team to start a new project and know which tools to use and how to use them (for example, we recommend Flask + Gunicorn in python, with several packages to handle API monitoring, validation, testing). These are just guidelines and can be ignored by the team when it makes sense (for example if the python service needs to be asynchronous then Flask may not be the right choice) and the chapter members always remain available to provide advice or clarification.

The back-end chapter also identifies and handles technical issues we have on our stack. We add missing monitoring on some parts of our stack. We look for security issues in the packages we are using and upgrade them. And we often investigate and fix exceptions from unplanned edge cases that are occurring in production code that was written a long time ago.

As a final example, some chapter tasks consist of refactoring parts of our codebase that became legacy and do not follow the current guidelines, but which feature teams developers are still working on. When such refactoring occurs, the chapter members take care to explain to the members of the feature team how the refactoring was done and why it was needed, to make sure that developers grow and are able to perform such refactorings in the future. This also pushes developers to take care about the maintainability of the code they write or stumble upon, thus encouraging them to clean every part of the codebase they touch a bit more.

Thus far the chapters have had an important positive impact on the technical team at Mindsay, but there are still some issues with the current organization that we are trying to solve.

What are the weaknesses of the chapters?

In theory, the chapter prioritizes points that should be tackled by the teams, but in practice, most of them are directly tackled by the members of the chapter. Only a few times were some tasks handed off to a feature team. In these cases, the tickets involved important risks that needed to be solved and the solutions were not quick to implement (too much time to give to chapter members). But most tickets either have a small risk associated or are deemed “quick enough” that the chapter can handle them. This leads to the chapter having quite a large backlog of risks to tackle, none of which is very urgent, but that are still important and take quite some time to perform.

We also saw that the time the chapter members dedicate to the chapter itself is a fixed 20% of their time. This is not optimal as sometimes their feature team is working on an important feature and it would be better to release it sooner and skip the chapter’s work in the short term. At other times, the problem is reversed. It would be better that the chapter members spend some more time fixing important risks (for example, if some risks are identified as the cause of a recent downtime and are suspected to be at risk of occurring again in the short term). The 20% time rule is sometimes shifted in practice to account for these cases, but this decision is often made implicitly by the chapter members and not really talked about with the other members.

Finally, as the chapters have a broad scope, it is often hard to identify what might be a risk on our stack, how important the risks are, how to solve them, and how to prioritize them. These are complex questions, especially when the risks are of different nature. For example, a security risk and a performance risk are difficult to compare, as their impact and the circumstances in which they occur are very different. Writing high-quality guidelines is also hard to do as if a guideline is poorly written then it may just not be followed at best, or it may lead other developers of the whole company to start writing code in a bad way at worst.

Conclusion

Nonetheless, the chapters have really had a substantial beneficial impact at Mindsay: training developers, ensuring they all have access to the knowledge they need as well as identifying and tackling the technical risks on our stack in a holistic way.

If you do not have a chapter or an equivalent system in your organization, then I’d advise you to consider it. If you already do, then I’d be glad to hear about your experience or how you may have solved the issues we currently face. Please leave your thoughts in the comments!