From Challenges to Success: A Consultant’s Short Guide to Effective Solution Architecture
Introduction
As an experienced consultant architect, I’ve had the privilege of working with numerous organizations, addressing their complex technological challenges, and creating effective solutions. Along the way, I’ve noticed various themes that architects delivering solutions need to recognise and consider, within broader organisational landscapes that we don’t control.
If these are not managed, they can potentially divert focus and impede your architectural endeavours. I’ve no doubt that seasoned practitioners will find these familiar, but they may serve as useful insights for those in the earlier stages of their architecture careers.
I highlight some of these themes in this post, maintaining brevity over completeness over a much longer list!
The Need to Find Lost Knowledge
During client delivery, a common challenge arises when existing systems knowledge falls short. Poorly maintained documentation, unreadable code, and lost knowledge hinder effectiveness. This challenge is further complicated when the knowledge loss extends to dependent systems you need to integrate with, making alignment with new solutions difficult.
Various organizational teams may share knowledge with you from governance and operational perspectives, but this does not necessarily indicate a comprehensive understanding of the existing solution for replacement, re-architecting, or modernization. Additionally, knowledge gaps can exist when integrating with existing systems, making interface changes for new solutions challenging. This is especially true with legacy technologies that are often resistant to change. Furthermore, organizations may lack a mature enterprise integration capability, which can compound things by forcing you to explore more intimate details of those existing systems.
To address knowledge shortcomings, it is critical to recognize the importance of seeking answers from the right people at the right time, including third-party organizations and your client’s partners where necessary. If you find knowledge levels are insufficient, consider implementing schemes to retrieve it using technical means and adjust planning accordingly. Some of these can involve code reviews, behavioural testing, reverse engineering, etc. Above all, ensure close communication with key organizational stakeholders and make sensible trade-offs as needed to protect the integrity of your solution and that of the overall project.
Who’s the Master Anyway?
Organisations vary in their enterprise data management maturity and that can have consequences in successfully architecting new solutions that are subject to them. Organisations’ enterprise master (and reference) data management approaches can constrain how your solution is architected and designed and needs to be carefully considered. Also, within the scope of your delivery, there may not be the opportunity to change or improve an organisation’s current implementation and it thereby becomes a solution constraint to work with.
You may encounter basic master data management schemes used to align data manually across isolated systems or ones that performs daily batch updates that keeps what can become a delicate state of equilibrium. Worse still, master data may be managed out-of-band for some systems, with the existing solution providing a failsafe mechanism such as human intervention. This can occur when manually uploading transactional data thus encountering new master or reference data failures as they happen and at that point affording the opportunity to remediate appropriately.
But when change occurs through the introduction of new systems or solutions, those existing schemes can be impacted. Typically, your solution architecture will facilitate systems integration and automation that will change the rules of the game. Supplanting manual uploads via UI, or replacing batch loads with near real-time feeds can break existing master data schemes if not factored appropriately.
The bottom line is your solution design needs to analyse all enterprise master data dependencies, and their exchange mechanisms, and design and work with any constraints.
Health Warnings
A key activity is establishing the health of the existing solution, including that of its upstream and downstream dependencies. This provides valuable insights about the overall stability which helps manage risk. While new solutions may provide opportunities to ‘clear the deck’ of known issues, their introduction will typically disrupt existing systems exchanges. This is especially true if there’s no enterprise integration solution in place and direct integration is your client’s de facto option.
Also, when new solutions are introduced in such scenarios, the existing systems into which they need to integrate with maybe change-resistant due to their limited capabilities or lack of in-house skills to perform more invasive changes due to their legacy or proprietary nature. In these cases, the obligation will likely fall upon new solutions to compensate. This can lead to unplanned scope increases and place a greater workload on your solution design and implementation.
Operational behaviours can also tell a lot about the health of existing systems. This is especially important if modernising. Incident and problem management sources can also help identify issues that are being managed around rather than directly addressed. From an operations perspective, it’s often with a good reason why things are the way they are. But what’s key is pursuing this to establish the extent to which any known issues will be impacted, exasperated, or resolved when a new or modernised solution is introduced.
End-to-End Failures — Who Cares?
Complex business transactions can traverse multiple systems, forwarding, or sharing data, and in the process, transforming, and enriching datasets in line with each system’s needs. When bad business data gets exchanged, resulting failures need to be recognised and escalated appropriately.
Bad data can occur due to poor enterprise data management and a host of other things that cause it not to be recognised within the target system. Integration solutions if they exist, will likely not have access to the target system’s internal business logic that recognises the data as bad, so they can’t detect these types of errors. So, when the target system of these data exchanges is your new solution, you need to design how failures are handled. This may transpire to be a source or upstream system which passed you bad data and therefore becomes the owner of exception escalations.
For these types of scenarios, not differentiating business data errors will likely trigger unnecessary work by support teams to conclude that the new solution is working fine and that the issue is with an upstream system. So, solution design needs to differentiate, bad business data vs. system issues and prompt their routing appropriately to those best positioned to speedily resolve. Priming requires providing sufficient levels of error metadata so that the owning system can efficiently diagnose and help resolve issues and generate corrected updates or re-feeds.
Managing Technical Debt
An important step in delivering architectural outcomes involves examining an organization’s existing technical debt and evaluating its implications on your delivery. Technical debt is the result of trade-offs made by organisations and that debt can influence your architecture, design, delivery, and phasing and therefore needs to be carefully assessed.
New solutions stand to remove existing technical debt, but the benefits may always not be fully realised until subsequent stages. Debt can exist beyond the boundaries of your solution, impacting integration design and forcing you to manage it as a set of constraints, considering its impact and finding ways to mitigate risks associated with it.
Even when introducing new solutions, architecture and design processes can uncover additional inter-dependencies with existing technical debt. These unexpected side effects can slow down delivery and impede progress during system integration testing, highlighting the importance of proactively managing technical debt throughout the project lifecycle.
In certain cases, technical debt, in some shape or form, may be temporarily incurred between solution delivery phases. Phased delivery is often required to avoid the associated risks with a big-bang release approach. Phasing may bring with it temporary ‘scaffolding’ bridging old and new, that continues to exist until future phases are successfully delivered. So, debt is not always a ‘bad’ thing, managing it is what’s important.
About the Author:
Gerry Whelan is an Enterprise Architect here at Version 1.
