Automated architecture diagrams

Who doesn’t love architecture diagrams? But who loves keeping them up date?

Often an organisation’s technical systems have a complex dependency graph, which is not always easy to discover and reason about.

There are usually lots of diagrams floating around but at TheFork (although we are not Netflix!) we encountered some challenges:

• Lack of consistency in styles and terminology (use of colours to infer meaning, what exactly is a ‘domain’ anyway, no ubiquitous language etc.)
• Mixture of tooling and no central storage (not easy to find and navigate, diagrams on LucidChart, Miro, Notion, GitHub).
• Reaching the right level of detail for varying audiences (e.g. senior managers, architects, developers, product, security and operations)
• Showing the same diagram ‘objects’ on multiple views (have to redraw each time rather than re-use, again leading to lack of consistency)

And the biggest issue of all -> Are these diagrams still up to date????

Last updated!

The Goals

As a new member of a team how can I quickly understand whats going on with the systems I will be responsible for, without asking hundreds of people (or my head exploding)?

When I’m building a new feature, how do I know what other teams and systems could be affected (and avoid nasty surprises)?

We were already adopting a new Enterprise Architecture framework to organise our teams and systems by capability. We had hand drawn diagrams for this but wouldn’t it be great to automate those and map that to the technical reality?

The model

We extended our meta model to map technical components to our enterprise architecture framework, with the following definitions:

Domain: the highest level of abstraction for a group of functional capabilities

Sub-domain: a 2nd level of grouped capabilities

Capabilities: a logical group of functionality (owned by a single team) inside a single domain

Component: an individually deployable technical artefact

With inspiration from the C4 model, we proposed the following diagram types, from which users could interactively drill down between the layers:

The landscape view allows senior leaders and product teams see the high level organisation and ownership of capabilities.

Landscape view

These capabilities are then clickable to the capability view, which shows dependencies on other parts of the system (blast radius) to aid the design of new features:

Capability view

From here you can navigate back to the top level, or drill down to the component views, which allow development teams to see the impact and dependencies of new and exiting features as well as visualising complexity:

Component view

Model vs flat static diagrams

Having a model vs static diagrams allows us to re-use the same ‘objects’ on multiple views (with different levels of detail and metadata), as well as making them discoverable via search. This makes them familiar, consistent, easier to read and saves time all round.

The technical solution

We researched some existing 3rd party tooling (like IcePanel) but decided to initially expand our existing in-house tool TheFork Map.

This tool already visualised the dependencies between our software components by parsing their configuration. This gave us the start of the bottom layer of our pyramid. But we extended this to also show:

• Message broker flows
• Datastores
• Package/library usage
• 3rd party system dependencies (still a work in progress)
• Technical debt measurements (via our in-house Beacon tool)

TheFork Map — Data sources

To add the capability views we extended this to query other metadata sources and produce a PlantUML markup file for each diagram. This is then converted to SVG and visualised using our existing NextJs website.

We generate all the visualisations automatically every night to keep them in sync.

The tooling also:

• Acts as an application catalog: a single centralised place to view team ownership and other metrics and provide an entry point to other tools (such as Datadog).
• Links and visualises our debt management initiatives (see my previous article on automatically enforcing decisions and this amazing deep dive from my colleague Nicholas Suter).

The future

We see many exciting possibilities to enhance the feature set for this tooling, including:

• 3rd party tooling integration/migration: Backstage, Cortex, DataDog etc. Some crossover with what we built, but not easily all within a single tool. The metadata preparation work we’ve done would still be relevant even if we migrated.
• New views/abstractions based on the datasets (infrastructure, GDPR etc.)
• Visualising business processes with BPMN
• Visualising data flows

Conclusion

Having this kind of tooling brings us many benefits:

• Trust in the diagrams (freshness)
• Easier technical analysis and solution design
• Visualisation of our current debt (and architectural issues)
• Smoother on-boarding for new team members
• Central access point (application catalog)

Overall we are pleased with what we’ve built so far, enabling us to easily see how our enterprise model maps to technical reality, and keeping that up to date with minimal effort.

However there are also a great many improvements we can make on our journey to level 5 of Simon Browns Architecture Diagramming Maturity Model!