Why CEOs should care about their product architecture?
Companies who develop products make hundreds of technical decisions every day, from the background color of their web app to the cloud provider it is hosted on. Only a few of these decisions affect the product architecture while most of them do not.
From my experience, nearly all people who are involved in building products understand that choosing the wrong product architecture may lead to scalability/performance issues, longer development time, potential security risks, and higher costs. They also understand that it affects HR staffing, the skills the company needs, the culture and eventually the success of the company. Often, the technical details are a bit overwhelming, but nevertheless, CEOs and other stakeholders understand that they must be part of the decision-making process, make sure that all the needs and risks were mapped, and maybe even have the final say on major architectural decisions.
But how can CEOs, who are in most cases non-technical people, figure out which of the thousands of decisions their company makes affect the product architecture and which do not? Their time is limited and they need to manage it carefully. Normally if they hear something in a conference, read about it in Medium, experience a heated debate between their tech team members, or being asked to approve an expensive procurement, they manage to grasp the importance of a decision. Obviously, the correct process would be for the tech team to raise the awareness of major decisions to the management, but sometimes, the tech team is either not experienced enough, not aware that a decision is critical, or doesn't want their decisions to be questioned. CEOs do their best, but they struggle, and eventually, they miss critical decisions.
Perhaps, the first step in helping CEOs is to define clearly what product architecture actually means.
Some people define architecture as the way in which functionality of the product is assigned to its modules and the way in which modules interact. The architecture is usually described as a block diagram with boxes inside boxes and arrows showing the relationship between them.
What I like about these diagrams is that they are a great communication tool. They provide a clear definition of the boundaries between the various teams. They help in aligning the team on the plan, in delegating tasks and in assigning roles to the team members. However, having a clear block diagram is not enough to ensure the success of the product. Some important technical decisions lie hidden inside the boxes, between the arrows, and in other dimensions. Also, for software products that constantly change, after a few weeks, the diagram will either be not relevant or become a spaghetti of lines, arrows, and various shapes.
Other people define architecture as the “biggest” and important decisions that you usually make early on in the project lifecycle. For example, deciding on the platforms, technology stack or frameworks that they are going to use, or deciding what to build, what to outsource and what to buy. The problem with this definition is that it still doesn't give a clear definition of what is considered a “big” technical decision and what is not. Additionally, companies might find themselves making big technical decisions during the project, and not necessarily at the beginning of it.
Lastly, a product architecture definition which I, personally, like is simply the set of technical decisions that are hard to change. Frequently, these are irreversible decisions or decisions that will stick with the product for a long period of time. Consider as an example, the background color of a web app, it’s fairly easy to change, so it’s probably not part of the architecture. However, using python as the main software language is fairly hard to change, so following this definition, it’s part of the product architecture. You now understand why I believe CEOs should care about their product architecture. If it includes all the technical decisions that are hard to change, you want to be sure you are making the right choices.
Some people think incorrectly that all hardware decisions are by definition part of the architecture. We all know that buying hardware components is expensive and replacing hardware is hard, for example buying storage for your data center or choosing the CPU chipset for your IoT device. However, what people don’t realize is that eventually, software components last many more years than hardware ones. Take as an example the decision to use Windows as the underlying OS of your server. You can use Dell servers, replace them with HP servers, and eventually migrate them to the cloud, that’s easy! However, Migrating your OS from Windows to Linux is much harder.
Generally, we can group technical decisions into three categories
- Technical decisions that are easy to change: Values of configuration variables, minor versions of software libraries, etc…
- Low-risk technical decisions which we probably won't need to change: Using Git as a version control system, Using Linux as OS, etc
- High-risk technical decisions: Buying storage for a local data center, deciding on a software language, a wireless protocol or a framework/platforms, etc
As a leader, you must understand which technical decisions are irreversible or going to be really hard to change and see that you approve of the analysis and final recommendations before your team implements them! Make sure that the team document the Shadow Architecture: all the alternatives that they decided against, but keep in a drawer just in case something goes wrong and you need an exit strategy. Document the decision-making process as well so that you can always go back to it and confirm that the arguments are still valid.
Why are some technical decisions harder to change than others?
Consider, as an example, a small product team that is about to build a minimal viable product. The team includes a developer or two, a data scientist and potentially a product manager. They decide to use technologies that they are familiar with to build something quickly to show to their users. they might even use this opportunity to evaluate a trendy new technology that they are excited about.
The MVP attracts a lot of attention, and the team wants to add additional functionality to it. Again, time to market is key, and the team decides to build on top of their MVP rather than refactor their product and rebuild it. They recruit more developers who are skilled with the initial tech stack and scale up the product. After a few months their environment looks like this:
- Your tech stack decides on who to recruit not you!
Yuval Noah Harari in his book “The history of mankind” asks us to look at the agriculture evolution from the viewpoint of the wheat. He claims that the wheat “manipulated” Homo sapiens to work for it, to feed it, fight for it, bring it water and clear more land for it to spread into.
Similarly to the wheat, you can think of a software language as a living entity and look at the role it is playing in your company. Who decides which developers to hire? The software language, or you? If your code is written in Python, you will hire Python developers to continue and “feed” the codebase and support it. From all the developers in the market, the software language filters more than 90% of the candidates, you are left to choose from the 10% of the candidates who remain.
To conclude this observation, there is a “job security” kind of relationship between the developers who are proficient with a certain technology and the technology itself. You can’t replace the technology without replacing your team. They keep each other safe.
Developers tend to use familiar technology, hyped technology, and not necesserily the right technology. They might follow a trend blindly, or not follow a trend arrogantly. They would get excited about some brand new, fresh-off-the-press framework and push to use it in your product. Some of them will treat technology as a faith, or will fall in love with it and act in irrational way. As a leader it’s your role to control the fire in the team: add oil, pour water… Avoid group thinking, but also too aggresive debates or strong opinions. Taking technical risks is important, but only if the promise and rewards justifie it. Always be on the guard and educate your team to inform you of any new technology they use and make sure you build a process around these kind of decisions!
2. Deploying too soon
One developer throws a stone to a well but even 100 experienced developers cannot remove it. Such is the case of deploying too soon. Once users start to use your product and engage with, once data is populated into your database, changing your product's underlying design and architecture becomes an almost impossible task. New constraints and relationships that hard to change are now formed. Instead of improving and shaping your MPV, you now need to handle change management, migration, backward compatibility, scheduled downtime, etc.. You might become a victim of your own success.
According to Google, a decision is: “a conclusion or resolution reached after consideration.” However, in our case, the architecture was evolved and not steered. Who made the initial decisions and why is not clear, but to change it now, would mean to spend resources on existing functionality instead of new ones.
Risk analysis → Graph not Table
When experienced managers analyze risks, they don’t just map the risk factors themselves, but also how they are related to each other. The risk severity is multiplied or diminished according to the number and type of relationship to other risk factors.
When mapping technical risks, each entity has its own risk attached to it: If you use a technical component that is obsolete, in a beta version, from a single or small supplier, it’s riskier than mature components from big and established suppliers. Some employees are more likely to quit than others. When you recruit a junior inexperienced developer you take more risks than when you recruit a skilled one. Some clients would be open, willing to take risks and see the relationship as strategic. They would define their needs accurately and communicate them clearly. Other clients would be unforgiving, change their minds every meeting and expect your entire team to work 24/7 just for them. But these risks, as much as they seem high or low, can be multiplied or diminished if they are linked to other risk factors:
- When you have only one employee who knows the low-level details of your product - The risk is multiplied.
- When you decide on a centralized component (Data lake, Data warehouse, Event bus) that is connected to many other components, the cost and complexity of replacing them are huge.
- When a strategic client uses an outdated version of your product the risk is multiplied.
Conclusion
We make technical decisions almost every day, not just technical people do, all of us, CEOs, project managers, designers, sales and even HR. What we are sometimes not aware of, is that some of these decisions will have a huge impact on our agility as a company in the future. As much as we try to move to agile processes, unfortunately, parts of our product are still going to be rigid and maybe that’s fine and important, but at least let’s make sure that we make the right choices at the right time.
