Technical Pitfall Series #1 — Lack of foundation
As a long-time application architect, I’ve had the opportunity to witness and be a part of many technical endeavors in all stages of the development life-cycle. Sometimes I have been fortunate enough to be involved at the project’s inception; other times I have joined the team when the development has already begun or in some cases to help wrangle a mature product. These varied situations have nonetheless exposed common patterns (dare I say flaws) in the product development process that I strive to counteract or avoid altogether in the projects I lead. The Technical Pitfall series of articles will discuss some of these common challenges and how best to defeat them in your own projects. The first pitfall I will address is not starting with a solid foundation in a new software platform, which I will introduce via an analogy:
Imagine you are building a house for your giant family. You want a five bedroom house with four bathrooms, a wine cellar and an indoor pool, so you start framing those out first. You put up the beams and the drywall, apply bricks to the siding and you start building the roof. The house looks great from the street. When you walk inside though, there are all sorts of problems. All these rooms have a dirt floor — because you forgot to build the foundation. The four bathrooms don’t actually function — because there is no plumbing. Now you have to tear everything down because you can’t build the foundation after the fact — money, time, and sweat all wasted, and now your family has to wait even longer to get out of the tiny apartment they’ve been crammed into while the house is being built. No one is happy.
Your application landscape is the house, and all the features that the product owner wants are those rooms and amenities — but it does no one any good to stuff the landscape full of the features if the infrastructure and architecture aren’t in place to make them work. And like in the house analogy, it’s a lot harder to put in a robust architecture when the features are already in place without massive amounts of rework. This seems like common sense, but many systems are built features-first, and then when issues of scalability or extensibility crop up, the technical team faces huge difficulties in adding the “plumbing” needed to allow the system to function as it should and to grow with the business.
If building the foundation first is common sense, why do so many fall into the pit of not doing so? One answer has to do with the dynamics between the product owners (“the business”) and the technical team. When it comes down to it, the technical team exists to support the business, and an application that is all foundation and no features lacks market value. For this reason, the business is always going to be pressing to get more and more features into a release, understandably so. However, it is the technical team’s responsibility to serve the business’ needs without leading them down a treacherous road. Everyone’s goal is to have the first production release of a platform be useful to the business, and in the case of a customer-facing application, trouble-free. The key here is making sure the business understands that this goal is best served by building a robust foundation and then building features on top of it.
One approach I have used to further this end is to have the first “release” not be a public release but an internal “proof of concept” milestone. The majority of the work in that first release is foundational; ensure that communication between all the involved systems is correct and has robust error-handling, verify that the security model is working as intended, and include the simplest possible version of the “feature functionality” in this initial proof of concept release. Then, engage your internal stakeholders to help the technical team test this basic functional set end-to-end. Since what you are testing is effectively the backbone of the future system, your team will be able to isolate, troubleshoot, and fix problems in these crucial core systems without having to weed through feature functionality that would make it far more difficult to determine where the problems actually lie.
Another roadblock is the misconception that taking time to build a strong foundation is goes against the Agile development practices en vogue today. This is far from the case! This foundation can be built using Agile principles amongst the team even before there is anything to ‘show’ to an end user. Feature development iterations involving stakeholders likewise can still be done in an Agile manner once the basic plumbing for core integrations and security is in place. The benefits of Agile development come from being able to develop iteratively and revise as you go to end up with a quality, business-meaningful product. You’ll lose a lot of that if you create a Frankenstein of mismatched, bolted on functions that don’t work well together and cause a maintenance and troubleshooting nightmare. Remember, generally the stakeholders aren’t the ones dictating the how the underlying framework is built — they’re concerned with the functionality atop the framework. To test this premise, feel free to engage a stakeholder on the ins and outs of the low-level API communications between systems and watch their eyes glaze over until they interrupt and say, “I just want it to do X, I don’t care how you make it work.”
Once the foundation has been vetted and is performing as intended, then it will be time to build in more useful and complicated feature sets into the first actual release candidate. What you will find is that even though you waited a bit longer to implement them, they will be easier to build and far easier to troubleshoot than they would have been if you built them without the solid foundation, and it is very likely that your actual time to market *with a stable product* across both releases will be the same or even shorter than it would have been if you skipped the proof of concept release phase. The more disparate systems that are involved in your technical landscape, the more crucial it becomes to start with a foundational release and the more gains in terms of product stability and overall time to market you will achieve by doing so.
I applied this approach to a recent project where I was fortunate to be leading the design from the start. The architecture involved integrations with several third party platforms in addition to custom services and portal UI, so it was crucial to ensure that the flow of information between these systems was solid and correct, and that error tracking and recovery were robust. The proof of concept milestone built out the end-to-end flow with only very simple business rules in the way; this was not a market-ready release, but were able to complete beta testing several weeks ahead of schedule and to then shift our focus into feature development. The first market-ready release was also completed ahead of schedule because bugs found in the beta period were mainly limited to the features themselves and not to communication between the systems, so they were much easier to isolate and resolve.
When pitching the foundation-first approach to your stakeholders, consider this: what the project stakeholders care about is when a product will actually be ready to use and what features will be available when it is. They should not be concerned with internal tech team deliverables. Given that a solid foundation doesn’t need to lead to a longer time to market, the key to stakeholder buy-in with this approach is to focus on the actual delivery dates of the market-ready product and the goal of a smooth, pain-free release for the system’s users. If you do discuss the interim build with stakeholders, make sure to set expectations that the proof of concept build is an internal milestone, so that no one expects a market-ready product until the feature release is ready to go. Avoid using verbiage like “release” to refer to that milestone to avoid this confusion. After a successful market release, your team will have all the proof they need that foundation-first works.
