Architecture Best Practices for Epic Based Product Development

Organization

Product development of large scale system requires good organization and processes so that we can develop sustainable product that continuously evolves but its main architecture becomes manageable.

On most organizations service based teams are too small to manage tens of microservices. One typical engineering organization is to form service groups that typically manages 3 -5 microservices. Each service group has service group lead with one or more service leads. Typically service lead and other senior engineers would decide on their service architecture. There are also number of Enterprise/Technical architects to align overall product architecture by working with each other and service groups closely.

For large scale software systems architects needs to solve extensive set of problems. Large scale products typically contains multiple core domains. It is impossible for one architect to know all domains in depth. Similarly there are multiple core architecture concern;

• Concurrency architecture,
• Inter-service communications,
• High availability,
• Rolling upgrades,
• Observability features (alarms, logging, system status, telemetry)
• Configuration management,
• Load, stability and performance testing,
• Toolset and third party library management,
• Common Data Management,
• Third party system integrations,
• Costumer dependent customizations,
• Security and access control
• and so on.

As you may imagine it is impossible for unique architect to know and work on all these extensive areas. Beside the company cannot hire architect for each area. The most effective approach is being flexible in terms of architects responsible areas. Architects would focus and provide solutions in areas that he/she has experience in. Generally 2-3 architects work in coordination to solve specific problems. These problem solving activity always should include service group leads, service leads and software engineers from related service groups.

Planning

Product development is planned as epics and stories using Jira tool. Epics are generally new capability additions to product. Epics are decomposed into multiple user stories. Most of the epics are created by product manager based on customer requests and product vision. Software architects also define new epics which are related to software quality attributes. New observability capability or architecture debt item to improve existing communication would be typical epics created by architects.

Epics are evaluated quarterly and high priority epics are taken to development plan. The epics are assigned business value, time criticality and risk factor (with scale from 1 to 10) and using these values it is prioritized based on Weighted Shortest Job First method.

Epics are reviewed and complex epics are selected for architecture design while more straightforward epics are moved directly to development phase. Epics requiring design are assigned to an architect for architecture design.

Design and Documentation

Arc42 template is already being used for documenting overall product architecture. However for each epic update to this document would not be enough a main product architecture is high level and cannot go into detail of each feature.

Separate document Arc42 based template can be used to evaluate and document architecture impacts of new epic to a product and its quality attributes. This template can be Confluence page, markdown document in code repository or plain word document. This document can include following sections:

• New requirements/ capabilities can be described with its impact to user experience.
• Solution approach can be discussed whether new microservice is being introduced or new capabilities to be added to existing microservices.
• Design details explains what is changed in software architecture. C4 Container diagrams can be used to show how new service will interact with other services. C4 Component diagrams can shows new modules to be added to existing micro services or which modules will be updated to handle changes. It is also very useful to define flow between services/modules using C4 dynamic diagrams or UML sequence diagrams.
• API changes introduced by this epic should also be documented. New interface together with data structures/messages can be documented in same place.
• Cross cutting concepts can also be evaluated within same document. High availability, rolling upgrades, testability, capacity impact and security are some parameters needs to be evaluated in the scope of new epic related changes.

Design Gate Meetings

Responsible architect coordinates multiple meetings with related stakeholders (other architects, service leads, senior engineers) to discuss the design approach for epic and and its impact to architecture. Once the design becomes mature and it is documented, a design gate meeting is organized with wider audience including upper management, product, Devops, test and development stakeholders.

In this design gate meetings architecture is presented to stakeholders with its impact to deployment, user experience and security etc. This meeting is good opportunity for getting stakeholder feedbacks and lettings them to plan their work for related epic.

If there are major concerns related to architecture follow up meeting can be organized to address stakeholder concerns. Otherwise design gate meeting is finalized as changes proposed to be implemented.

Implementation/Test

Once design gate passes, epic can go into development stage where related user stories are created and planned in sprints.

Architects continue to work closely with development team for the development of new epic. Intermediate design meetings, design/code reviews with development team is performed to make sure that implementation goes as designed. Closer the architects to development team, the better the architects can design their product architecture. Architects improve them self by observing impact of changes they propose to system.

Architects should always design and be part of profiling, stability, load tests, stress tests and functional tests. These final tests on the product are final verification of the architectural changes.