Architectural Significance Test
Business value/risk, stakeholder concern, quality level, external dependencies, cross-cutting, first-of-a-kind, past troublemaker?
What is important when analyzing and designing software architectures, and what is not? This post proposes 5+2 criteria for the architectural significance of requirements (and other artifacts) and applies them to several examples.
Motivation
Let’s assume it is the first day for you on a project that has been running for a while (for instance, one that modernizes some legacy system); you might be an external consultant that has just been hired, or have switched role and project internally. What to work on first?
As a software engineer with architecture responsibilities, I would like to prioritize technical issues quickly so that architecturally significant issues are addressed at their most responsible moment and we do not have to revise decisions and designs unnecessarily later. When doing so, I would like to be as objective as possible to avoid that loud voices receive more attention than they deserve so that the truly urgent and important things are tackled first.
The time budget per issue is one to two minutes at most, as it is not unusual to be confronted with 10 to 100 issues per day — thanks to email and auto-notifications in tools such as Kanban boards, issue trackers, and source code management systems. Modern world!
Seven Criteria for Architectural Significance
So much for motivation. Starting from insights from peers and then reflecting on my own heuristics, I ended up with seven criteria to assess whether issues qualify as Architecturally Significant Requirements (ASRs):
1. The requirement is directly associated with high business value (benefit vs. cost) or business risk.¹
2. The requirement is a concern of a particularly important stakeholder such as the project sponsor or an external compliance auditor.
3. The requirement includes runtime Quality-of-Service (QoS) characteristics (such as performance needs) that deviate from those already satisfied by the evolving architecture substantially.
4. The requirement causes new or deals with one or more existing external dependencies that might have unpredictable, unreliable and/or uncontrollable behavior.²
5. The requirement has a cross-cutting nature and therefore affects multiple parts of the system and their interactions; it may even have *system-wide impact* short term and/or long term (examples: security, monitoring).³
6. The requirement has a First-of-a-Kind (FOAK) character: for instance, this team has never built a component or subsystem that satisfies this particular requirement.
7. The requirement has been troublesome and caused critical situations, budget overruns or client dissatisfaction on a previous project in a similar context.
The list is ordered logically, from project goals and key players to analysis and design elements to organizational matters; this outside-in order does not imply that criterion 2 has less weight than criterion 1 or 3 (and so on). Criteria 1 to 5 are somewhat more objective and easier to agree upon and generalize than criteria 6 and 7, which are highly context-specific.
I have been using and teaching these criteria for several years now; I do not claim that the list is complete, but picked the criteria that I deem most useful.⁴
A Scoring/Assessment Template (ASR Test)
A table is an obvious format for the resulting seven-criteria ASR Test:
One option to fill the cells in such a table are basic values such as Y = yes, N = no, ? = unknown; if you prefer, you can also enter a bit more elaborate ones such as H, M, L and n/a.
If you cannot agree on the significance of a particular requirement immediately, do not hesitate to mark it as open and track it as a new backlog item (or maintain a separate parking lot of open issues for periodical reconsideration).
Instances of one criterion may appear several times when assessing a single requirement. For example, criterion 4 (external dependencies) is met twice if the splitting of user story unveils that two additional public Web APIs must be consumed to realize the story.
I do not fill out the table explicitly every time I screen and filter technical issues, but run the test rather (sub-)consciously while reading any issue that grabs my attention (in a chat message, an email, a Trello board and so on). If you do want to create an explicit representation, a project wiki, team space or other collaboration tool may host it.
The ASR Test is not a quantitative tool, a weight calculator; relevance assessments are always qualitative, subjective and context-depending; the test makes formerly tacit knowledge somewhat explicit (“worst first”). Do not spend more time on assessing relevance than on tackling and solving issues!
Examples
The following table scores some requirements and constraints according to their architectural significance:
Light Tool Support: miro Board
For your convenience, I created an ASR board in the online whiteboard platform miro for this table format (first five ASR criteria only); feel free to use and/or extend it (under a Creative Commons license).
Here is an example of a filled-out miro template:
Based on the test result, I’d probably look at the payment change request in the online shop first and the design of the security credential cache next. I’ll not get involved with the class renaming (unless I have to review the change w.r.t. project-wide coding guidelines).
Concluding Thoughts
Here are the take-away messages from this post and some additional advice/remarks:
- When deciding which technical issues to focus on, consider value, cost and risk implications. Follow an outside-in approach, starting from project goals and external interfaces that you will have to consume (and rely on).
- The “five plus two” ASR criteria that work best for me are: 1) business value and risk, 2) key stakeholder concern 3) unusual quality-of-service requirement (order of magnitude more advanced than previous ones) 4) external dependencies that are uncontrollable, unpredictable or unreliable, 5) cross-cutting, system wide impact; 6) first-of-a-kind character (novelty for team), 7) bad experience and trouble in the past.
- I use the criteria not only when joining a project (as the motivating user story at the start of this post assumes), but also when returning to a project after an external event and checking my mailbox, when planning my week/day and so on.
- Balance tactic progress (for instance, indicated by iteration velocity and long-term thinking and development of the system; some non-functional properties only become relevant (and might start to hurt) in the long term.
- The ASR test presented in this post works for requirements (mostly non-functional but also functional), design elements (components and connectors, viewpoint depending) and architectural decisions.
ASR scoring is not an exact science, but qualifies as a “mighty” method element imho; Michael Keeling’s ECSA 2020 keynote has a definition of this term and success criteria (method superpowers, that is).
An extended version of this post(that also reports on some architectural decisions with a rather early “most responsible moment” for making them) can be found on my other blog.
Do the above ASR criteria work for you? Did I miss a criterion (test question)? Contact me!
Notes
¹ Do you wonder whether technical risk is missing? Please have a look at the next six criteria…
² Note that dependencies may be compile time ones (library, package manager such as maven) or runtime ones (remote API, message queue) or even have a logical or organizational nature only (examples: approvals, deliveries). Also note that we might now know all dependencies and their properties when assessing architectural significance; hence, it might be appropriate (or even imperative) to ask questions and/or prototype.
³ You might want to differentiate between (1) deliberately independent and (2) heavily intertwined parts of the system here: (1) The new requirement might be in conflict with another one considering the desire to avoid dependencies; a tradeoff analysis or prioritization decision might be required. (2) This might be inherent to the problem or an architectural smell indicating a need to refactor.
⁴ The issues can be requirements (incl. change requests) as called out in the introduction of the seven criteria above, but also pending architectural design decisions or even design activities on structural elements such as components and connectors.
© Olaf Zimmermann, 2020. All rights reserved.
