In the Loop: How a Release Team Centralizes and Aligns Processes
Most people have a turning point in their careers. For me, it was when my manager challenged me to set up a new team to improve all release-related processes. Once that happened, for weeks, all I could hear was this question echoing in my mind:
Since everyone advocates DevOps with autonomous teams that control all processes, why should a company have a team that centralizes processes like release management or even care about feedback?
Soon enough, I realized that the journey of the development teams to achieve self-sufficiency would be much harder without someone to help them. And that some level of supervision is always required to make sure that things are done in the right way.
I am sharing my story in hopes of inspiring others who might end up in a similar situation to try something different and challenge the status quo, and I invite you to tailor our processes to write your own story.
So, let’s start at the beginning.
Once Upon a Time in a Nearby Department
Our engineering department (R&D) had one maintenance team that handled all the feedback reports, such as suggestions, crashes, and defects, coming from the field. The maintenance team would handle everything.
Back in the day, we had monolithic releases every few years, and that team structure made sense. But now that we implement ideas and roll out updates much more frequently, things have changed, and we reorganized the engineering department as a result. The goal was to create self-sufficient teams, each with a well-defined product area. Reports that were then centrally handled by the maintenance team would now be directed to the appropriate product team.
How could we do this? How could we ensure a consistent process for everyone? That’s where the release team came in. We needed a way to coordinate team dependencies and common processes. The objective of our new team was to help other teams with product releases and bring feedback straight to them. Of course, everything goes through a screening process so that it gets to the correct development team. This effort between all development teams is made possible with our “be helpful” culture. After the teams address the comments, questions, and concerns, we then support them with the necessary releases.
At the same time, the new team inherited the release process, which was scripted by an extra-long checklist with a lot of manual steps. We had to fill in Excel files and insert the new version number in multiple legacy tools. Can you imagine the hassle? With this process, releases would take weeks!
It clearly wasn’t the way to go.
Ch-ch-ch-ch-changes: Turn and Face the Strange
We needed to change a few things. And following the mindset of fail fast, learn fast, we were not afraid to challenge the status quo and promote changes that at that time didn’t seem feasible. Many of those changes were not necessarily related to processes; they were structural changes in the product architecture and improvements in the test infrastructure.
But, there were four areas that were either in the scope of the release team or that the team could contribute directly. These areas were reworked to bring more agility to processes, remove the workload from teams, and provide them with mechanisms to deliver value and receive feedback faster.
1. Release Content and Release Notes
Since we were working with multiple teams, we needed a way to keep track of what was being delivered to our customers. How could we make sure that relevant information was included in the release notes of each version? How could a support engineer know which version fixed or implemented something?
We use Jira and Portfolio to plan and manage backlogs and sprint plans, so our first thought was to use them to produce release notes. But since we had a couple of special use cases, these tools were not an option.
This meant we had to roll up our sleeves and build an application that met our needs:
- Keep everyone aligned about every release milestone and its contents.
- Be able to trigger release processes.
The former ensures that everyone in R&D knows what will be delivered and has the right visibility to take actions and be self-sufficient. The latter streamlines the release process by automating processes that were once manual and prone to error.
This was a big step in our journey because now everyone concerned could see what was being done and our process of gathering information to produce outputs like release notes happened at the click of a button. Release notes became part of an automated flow.
2. Release Criteria and Dogfooding
One thing down, three to go. Now we had to answer these questions:
- When do we start the release process?
- How do we know that everything is done and with quality?
Although the continuous integration pipeline gave us visibility of more than 6000 tests, it wasn’t enough to understand if we could move forward because we didn’t know if all planned issues were done and ready to be shipped.
Furthermore, this decision should be simple and clear for everyone like a traffic light (red for stop, green for go). So we defined release criteria, a set of rules that indicate if an issue is Release Ready and if the branch is in good shape to start the release process. This would help us determine how far we were from releasing a new version.
One of those release criteria and perhaps the most important one was to formalize dogfooding as part of the process. At our company, we have several teams working with the product, so we immediately thought, “Why don’t we bring them to the release process and use them to ‘eat our own dog food’ first? These teams can test the product before customers, provide faster feedback to DevTeams, and work as a safety net to catch problems that weren’t caught in our QA validations.” And that’s what we did.
In addition, to manage the risk and be more in control, we created several rings with different teams of dogfooders and made sure that every version passed through each ring before reaching the public. Transitions between rings are decided by release criteria and each new ring expands the number of people using that version.
3. Telemetry and Feedback
Now that we had the release process working, along came some new questions:
- Were users really using the new version?
- How can we compare the quality of the current version with the previous?
- Are there any showstopper issues?
- What are customers reporting?
We only had an old application to store crash reports, which didn’t fill all our needs but no one dared change it. Having to deal with this legacy tool was a nightmare, so we could either buy a new one or build our own. Well, you know when people say “there’s an app for that?” In our case, there wasn’t, so we built it ourselves. This new application is now a crucial tool, giving us more visibility on the actions of our customers. As a consequence, we’re able to address 10 times more reports than before!
Additionally, crashes were only a subset of events that we could be collecting. We wanted more. We wanted to know the number of times a feature is used or the number of clicks. As we were deciding what to do, we noticed that a team had already started sending telemetry events to a Data Lake. So why not take the opportunity to also implement telemetry instead of relying only on crash reports? We designed and implemented a common and scalable telemetry system, where crash reports and customer feedback are just one event type among all other telemetry received.
Today almost all teams use telemetry to measure the success of a project or to make decisions. It allows OutSystems engineering as a whole to make informed decisions based on real data.
4. Visibility of… Everything!
Just when we thought we were smoothly handling several releases at the same time with teams receiving frequent feedback, we realized there was still something to be addressed: full visibility. And what does this mean? Well, we realized people still had questions about the release milestones. You see, we worked with different personas from different departments who, depending on their context, would need specific information about product releases. So we talked with stakeholders to address their different needs and identified three use cases:
Plan work: This impacts several teams like support, marketing, or documentation that don’t contribute directly to releases but still need to be aware of when things will happen to sync their tasks. For this, we created the Release Plan:
Be informed about ongoing releases: More focused on engineering members, this ensures that everyone can check a release status: the ring, number of crashes, and received feedback, for example. This led to the release dashboard:
Be informed about a specific release: This allows anyone in the company to dig into a release and explore its details: for example, content (delivered features and issues), adoption velocity, users, crashes, and stability comparison with previous versions. And this became a detail screen:
We even took things one step further and recently implemented a subscription mechanism that allows people to learn whenever a date changes for a specific version. Now people don’t need to check with us so frequently because they know that date changes will automatically appear in their inboxes!
Released Happily Ever After
Can you still say it doesn’t make sense to have a single team to centralize all these processes? I’m convinced that it was the best choice because this allowed us to keep the car moving while we were changing the tires. Nothing had to stop.
We could feel the pain of our teams who were dealing with problems every day but not having time to address them because they couldn’t lose focus of the main goal: to build our amazing product. As for us, we never aimed to do their tasks. All we wanted was to create the best conditions that allowed them to be self-sufficient.
Today, we’ve eased the pain of our teams. We can release some parts of the product independently, and it only takes three hours from a commit until the first users install it. Everyone is on the same page and we’re almost like those guys that are in Mission Control when SpaceX launches a new rocket, checking if everything is up and running until it reaches the target. At least, that’s how we feel, one mission at a time.
What’s your mission? How will you achieve it?
