Part 3: Our Architecture Decision Process

This is the third post of a four-part blog series about how the One Medical technology team uses Architecture Decision Records.

By Amber Houle and Jeff Ammons

In Part 1 of this series, we looked at how One Medical started using Architecture Decision Records (ADRs) as a tool to help us make better decisions and to document the what and why behind those decisions. In Part 2, we started to look into some of the challenges we saw in implementing ADRs and how we approached solving them. Part 2 also covered the roles of decision leads and our architecture councils. In Part 3, we’ll look in-depth at how each step of our architecture decision process works (and some pitfalls to avoid).

As a quick refresher, the overall decision-making process follows a double-diamond of creating a proposal, finalizing the decision, then finally, communicating that decision.

Part 1: Creating the Proposal

The first section of the decision process is this:

Define the Problem

1. Begin with defining the problem you are aiming to solve.
2. Spend time here gathering feedback from stakeholders and other engineers as appropriate. More time spent on problem definition will pay itself back in reducing unnecessary scope and discussion time later on!

Document the Options

1. Document the problem, use-cases, requirements, and decision context as well as potential solutions.
2. This could be a slide deck or a Google Doc. Do some research into options. Consider prototyping some options.
3. Write up pros, cons, and considerations for the different proposed options.
4. How well do these options solve your problem and use-cases?

Be sure to include security, infrastructure, quality, and compliance considerations in your research and documentation. If you’re not sure what these might be, please ask in [the respective team’s Slack channel].

Collect Feedback and Iterate

1. Send this write-up out to [our all engineers email list] and post it in the [all engineers slack] channel for comment.
2. Use discussion and feedback on this document to improve the document and your understanding of the problem and solutions. Anyone interested is encouraged to participate at this stage.

These instructions are fairly simple, but there are some intentional decisions built into them that are worth looking more closely at.

Starting with the problem — This step is one that we sometimes skip in problem solving, often to our detriment. Often, as someone leading a group to make a decision, we have thought much more about the problem than others. As a result, there are often a lot of assumptions and mental models we have built up in our head which aren’t obvious to others. Not defining the problem clearly often means those assumptions and differing models show up much later with circular discussions and miscommunications as people attempt to solve for their own variation of problems.

Using a slide deck or document to start — To start, we used Github for the entire process. Github allows for some discussion and debate, but often feels more final than a Google Doc, given its higher friction to change (it also doesn’t handle comments as well as Google Docs). It’s helpful to start this process with a tool that feels easy to change and comment on.

Sharing broadly — We wanted to make it possible for engineers to decide which decision they wanted to follow or participate in. Our system is complex enough that we wanted to make sure that the decision lead didn’t have to be an expert in every part of it in order to know who should be included. This also allows for transparency early in the process, before any final decisions have been made, so anyone who feels strongly about a given decision can have their perspective heard before the decision is finalized. As an added bonus, this allows engineers with less experience the same access to the decision-making process as more senior engineers, giving them a chance to learn.

Part 2: Finalizing the Decision

Schedule an In-Person Decision Meeting

1. The goal of this meeting is to make the best possible final decision and move forward towards implementation.
2. Selecting the right participants is key. Select five to six engineers for this meeting, starting with the appropriate architecture council (if it’s not clear which council to involve, ask your engineering director or VP for help selecting who to invite).
3. Once you’ve accounted for the Architecture Council members, fill any remaining seats with engineers who you believe will help make the best possible decisions — aim to add diversity of perspective and experience solving similar problems in the past.

Prepare for the Decision Meeting

Preparation for the meeting should consist of finalizing an agenda for discussion, a clear description of the problem, and the list of options with pros and cons. Consider including a matrix of possible solutions vs. evaluation criteria.

Whenever possible, consider sending any materials out to the engineers involved for their review prior to the meeting.

Meet and Decide

There are two possible outcomes of this meeting:

1) The group decides on an option and agrees to move forward to implementation.

2) The group decides that more information is required and agrees to decide after this is obtained.

If deciding on option 2, the group should also decide how that later decision should take place: should it be another meeting like this, or can it be made by a smaller group once more information is gathered, or some third option?

At the end of this meeting, the council should have pushed “approved” on the ADR Pull Request or put in specific change requests.

This part can be challenging as it involves managing a group of engineers to agree on a final decision. Sometimes this is simple, other times it’s not.

Again, a few points worth noting:

Schedule the meeting ahead of time — This is simply due to logistics. Finding overlapping time from five to six people can be challenging. It’s frustrating to have finished all the steps and be ready for a decision, only to have to wait another week to find time on people’s calendars.

Send out materials ahead of time — Often, the architecture council members will have seen the previous decision-making documents, but if they haven’t, having time to process and think through the proposal ahead of time saves time in the meeting. Ideally it helps improve alignment and collect missing information ahead of time also.

Meeting outcomes — Not every decision meeting will lead to a decision, but it should make meaningful progress towards making one. If the group can’t reach a decision, it’s fine to decide what additional information would be required.

Aim for, but don’t require, consensus — Frequently, if the decision lead has done good work, the decision is clear before the final meeting occurs. If it’s not, the group should debate and seek a common agreement on the best direction to proceed. Don’t let perfection be the enemy of progress though. If the group needs to cast a vote to make a final decision, do so.

Part 3: Communicating the Outcome

Write up the Decision as an Architecture Decision Record (ADR)

1. Most of the ADR can likely be copy/pasted from previous documentation. Make sure to include any pertinent discussion and context from the decision meeting also.
2. The ADR should still pass through PR acceptance and the owner should address any comments that arise. There is a Github group (adr-reviewers) for approving ADRs which includes all architecture council members and staff engineers.

Communicate the Decision

1. Once an ADR has been finalized, let everyone know about it via email: [our all engineers email list] and Slack: [all engineers slack channel]. Include enough context so the engineers affected know enough to read the ADR.
2. Ideally, your email should include the concrete next steps and how this decision will impact things in the next few months. (If it won’t impact anything in the short-term, maybe consider not doing this process at all until it will?)

If the decision is suited for it, consider presenting it in our weekly engineering forum as well.

Note that this process still contains the final ADR documentation, ensuring that we maintain all of the benefits of clearly-documented historical decision records.

The aim of this final stage of the process is to ensure that other engineers (present and future) are able to understand why we made a given decision. It also serves as a useful artifact to finalize the process. Any decision is only effective if people understand and follow it. By having this final document and communication push, we attempt to ensure that people have the context and communication necessary to make uptake successful.

Up Next:

Now that we have an understanding of what Architecture Decision Records are, as well as an understanding of the process, check out Part 4 of the series, where we look at a concrete example within One Medical.