Autonomy versus Enterprise
autonomy: the state of existing or acting separately from others (See also INDEPENDENCE)
alignment: an arrangement of groups or forces in relation to one another
The mantra of a product team is that of autonomy. The word is interpreted in dozens of ways but likely you will hear words like “freedom,” “choice,” and “independence” feature heavily in those definitions.
Some Product Teams can interpret it as “we can do what we want” and in a larger scale enterprise that will naturally cause a lot of conflict with attempts at governance, standardisation and alignment between product teams and organisational goals.
Similarly the reaction from those responsible for governance and bigger-picture goals such as enterprise architecture or compliance can be an over-reaction. An even bigger push on trying to clamp down on teams with enforced tooling, reverting to platform mentality and trying to push controls via commandments that impinge heavily on a product team’s workflow and technical choices.
So how do we solve this conflict between a product team’s need to have freedom in technical and practical decisions, with an enterprise’s need for control and safeguarding of a coherent and consistent delivery of business critical systems that may span dozens of product team deliveries?
Why is Autonomy so important to a Product Team?
Three driving forces of a successful Product Team to be able to provide the best and most agile delivery possible are:
· Ownership of the entire value stream
· Minimum dependence on others
· Focus on (and measurement by) outcome
Owning the value stream is more of an organisational issue. Conway’s Law highlights that much of the scope of a team’s autonomy will be created by a solid foundational work before a product team is even born. If done correctly, the result should be a lot of natural autonomy because there are no / few stakeholders making decisions outside of the team.
For the rest of this article we will focus on the other two points and how the conflict between Product Team Autonomy and Enterprise Requirements can be easily settled assuming that the product team has been given ownership of the value stream.
Minimum dependence on others
There are two elements of dependence for product teams, the surrounding architecture and the tool-chain. Remember that for every dependency added, of either type, there is an exponential increase in effort to achieve any specific task.
Architectural Dependence
Architectural dependence can be difficult to avoid. If you are a product team working with the checkout experience for example, then it’s possible to find that your team cannot even release without testing by, approval of and collaboration with the frontend team, the stock check team, the pricing team and other similarly related services.
Likewise, there are likely to be compliance and regulatory constraints that mean you are also needing to collaborate and get support from the security, privacy and even legal teams.
So how can the checkout team maintain autonomy when there are so many external inputs to how you can design and deliver your product?
Toolchain Dependence
One of the premises of small product teams they can focus on building a specific product in the absolute best way without needing to crowbar a product into pre-chosen, generic platforms, languages and tools. They can decide on how it is built, how it runs and how it will give the absolute optimal result for that specific product. This leads to increased innovation, optimisation, cost-savings and much more agility within the ever-changing technology landscape and business requirements.
On the other hand, enterprise wants things to fit into nice boxes and categories — there is usually a call for more “standardisation” and that standardisation is then described via specific tools. For the enterprise, EVERYTHING would be easier if all applications were written in Java, Run on Tomcat, Using Apache as a web server and Postgres as a database. Then by magic all that is needed for any new service to be created is to join some boxes together into another identikit architecture!
That’s not standardisation, that’s vendor-selection and bureaucracy.
So, in this situation, if the business requirement is “make it faster” and we as a product team know it would be 1000% faster if we could run it on a serverless platform and rewritten in Go… then neither the business nor the team is happy — yet the enterprise is!
The same is true not just of the final application, but every step in the SDLC from build, to test, to run, to working methods — the product will be better and deliveries faster the more in control the team is of the toolchain to deliver it.
If the product team has it’s source control, issue management, test tools, security tools, IDE’s, package management, documentation, CI & CD practices etc already pre-determined by outside forces — there is no room to innovate or improve any of it. Indeed even getting access to all the tools for a new team member can end up being a weeks-long nightmare of dozens of tickets to external teams.
“Aligned Autonomy” == “Standardisation”, “Standardisation” != “Tools”
Standardisation is of course crucial in one area — interaction between components and services!
Service A should not need to be transforming JSON to XML in order to talk with Service B who will respond with a CSV file over FTP.
Standards should simplify interaction meaning that they likely should be based on protocols and/or something like IEEE standards that will be interoperable with the outside world too!
HTTPS, GraphQL, FTP, git, JSON, IEEE P1363, etc are standards already written for us and all but guaranteed to be supported by any vendor in the space.
Often in describing standards or trying to achieve “alignment” we overshoot dramatically and end up prescribing HOW via tools instead of OUTCOME via protocols.
The outcome desired is that (for example) “all API’s must be provided according to REST Level 3 with optional GraphQL and be semantically versioned” — That is a standard. It achieves the enterprise requirements for consistency and interoperability without ever needing to be held hostage by a specific vendor.
It leaves complete autonomy for the team to build those API’s using Go, Java, Node, Rust, Kotlin or flavour-of-the-week. No user in the world cares what IDE you used, which cloud platform you run on, what you use for CI & CD, or whether your issues were managed with Kanban. Only that the Outcome is what was agreed upon.
At the end of the day, that too is what the enterprise wants — agreed outcomes. API’s that everyone can use, engineers who work using git for source control, secure products, performant products and functional services.
So both the Product Team and the Enterprise want the same things — but the alignment comes from protocols NOT the toolchain used to get there.
Take an easy example, such as Security :)
Let’s take just one topic and see how we can solve both the Architectural and Toolchain dependence issues for product team that both achieves the enterprise goal of an OUTCOME with the need for AUTONOMY for the product team.
This is a good topic as it’s an area that typically focuses on Tooling to “solve” the problem but also it includes both the technical and the softer aspects of working with IT in general.
Desired Outcome
The desired outcome (enterprise) is “all products should be secure, protect user privacy and meet relevant compliance regulations for the markets and contexts”
One solution may be to bring all that experience and expertise into the product team. You would add a security engineer, a data engineer, UX, a lawyer, privacy expert and many more bodies into a team that perhaps already has numerous backend and frontend engineers, product owner, engineering manager, agile coach, etc and suddenly you have grown from a 2-pizza team to requiring your own Dominos Pizza store for any kind of team gathering!
But the likelihood is that you do not need all that expertise 24 hours a day / 7 days a week / forever. If you do, then it likely means that you are not engineering those problems away and are instead firefighting issues in those areas in a reactive and ad-hoc way as they arise.
What your team REALLY requires is competence, WHEN it is needed, to assist in helping you understand the topics and the challenges. It means you need education for the team and support from a security engineer to help your team understand how to perform a threat modelling exercise, how vulnerabilities can be detected, tested, triaged and fixed. What tools are available, further courses and guidelines.
The approach
As a simple example — your team might not understand a great deal about simple security practices such as dependency management, license management, image scanning, SAST and DAST testing etc.
But together with the borrowed expert(s) you can engineer maintainable and automated testing into the product and now those borrowed resources can disappear again… Teach a man to fish as the saying goes!
So after relatively short time — you no longer need a full-time security engineer in your team. The security quality gate and enforced security tooling disappears, as does the requirement for reports and pretty charts for management to feel secure as everything is automated and any result less than acceptable stops the product from being released.
The competence of the entire product team has been raised to a point that security is not just a quality gate they need to get through or Green tick-box (that may as well be a black hole for all the understanding of what that that tick-box means).
Security has become a natural part of every-day work to the point an engineer is thinking about it from the first line of code they write for new feature X. They understand WHAT is being tested, HOW it is being tested, they can ADAPT to changing conditions and can RESOLVE findings with confidence and speed.
Meanwhile, the security engineer can dissolve into the background and help other teams. They are, of course, available for support and assistance in future, but the security engineer has enriched their own experience, learnt about the product, language and the context which is valuable in helping other teams and they can share their learnings with other security engineers too.
But in summary, the lesson learnt here is that coaching and support in a key area of expertise rather than coercion to use specific tools has resulted in a better outcome for all involved. The team has been enriched, the outcome achieved and no one has to make their CI pipelines connect to a VPN to some internal Security testing server or get approvals for every release.
Focus on (and measurement by) outcome
We used a technical example of “Security” above to demonstrate how the dependence on tools or architecture can be removed whilst protecting team autonomy and enterprise alignment. But it can really be applied anywhere as long as a team is eventually measured on the outcome rather than how they get there.
Taking the earlier example again of the checkout team — if clear outcomes are defined between the teams, then everyone knows what they should deliver and what standard will be used.
For example:
· Pricing team provide pricing updates 1 time per hour via Pub/Sub according to protobuf specification X
· Stock Check team provide a per-store stock-level via REST API endpoint /stock/{market}/{store} according JSON Schema Y at a sustainable rate of 800/second
· We all use semantic versioning for our services so that we are not hit unexpectedly by breaking changes from other services
· All services provide a health endpoint at /health with key metrics for saturation, rate and response time
Notice we introduced two other standards here which are general and yet still make no demands on teams to do things in any special way. Semantic versioning is common and good practice across the software industry and there is no mandate of any specific monitoring framework or tool — yet any one can consume the data.
The result of this outcome driven goal setting is:
1. All products (not only checkout) who want that data will have no need to “align”.
2. All teams can be measured on these super measurable outcomes that tie nicely into SLO’s/SLI’s.
3. All teams can contribute requirements and requests to change service levels to each other
4. The outcomes are supported by any modern language or framework or platform
5. No-one cares that you are using Cloud Provider A or Monitoring tool B or if going to try some hot new tool for managing your backlog
6. NO-ONE is doing un-necessary work that adds zero value such as working around enforced testing tools or trying to eke performance improvements out of the totally unsuited component they are forced to use.
7. Everyone can devise and adapt their own optimised toolchains and pipelines to achieve the required outcomes.
Conclusion
Autonomy doesn’t mean “wild-west” but nor does it mean “everyone work within these pre-selected prisons”.
Autonomy doesn’t conflict with enterprise goals, it makes achieving them easier and faster if outcomes and standards are defined correctly!
Alignment should be natural if Conway’s Law has been considered in the organisation of teams and standards are really standards instead of vendors and rules
The enterprise no longer needs to artificially satisfy itself via a misleading illusion of alignment based on “if everyone is doing the same then we have control” and instead has measurable and predictable returns for it’s investments via clarity where it matters.
