Winging it

Or how to survive the flood of technological options

In Owning it up: How business owners decide if it’s not their business, I tried to explain how business owners should respond to technical questions. (Spoiler alert: they shouldn’t.) It did raise a new question though: how should technical people make decisions, in the face of the myriad of options to choose from?

Once upon a time, life was simple. I still remember the time when HTML was defined by less than 10 tags. When there was just a single browser to deal with. And just a single web server, that only could be programmed to do something dynamically in a single way.

Scroll 25 years forward, and I think it’s fair to say that it expanded into something that became impossible to fully comprehend by anyone. Just the HTML standard itself has around 150 tags now. But styling was pulled almost entirely from HTML into its own thing, CSS, which itself has now grown beyond anyone's full comprehension. And CSS itself forked into SASS, and SCSS and LESS and Stylus and others.

And there is not just one way to generate HTML dynamically: it literally exploded, and it becomes even impossible to count the number of ways in which you can have dynamic content on the web. And if you limit it to server-side rendering only, then there is a myriad of options now to get servers to render HTML or even to have servers doing anything at all.

Now, even if you would disagree with me and think that it is possible to fully grasp one of these things (let’s just say, CSS), then I’m sure you would agree with me that it is impossible to fully grasp the combinatoric explosion of each of these options.

The Dramatic Question

Which brings me to the dramatic question that has been haunting me over the last couple of years:

Is it even conceivable to know for sure that you’re picking the optimal technology?

I’ve come to believe that it isn’t:

• The parameters that factor into ‘optimal’ are changing every day.
• The combinatoric explosion of all options has become impossible to navigate, even if you would be able to stop the time and study the current state for months in a row.
• In reality, you can’t stop the time, and new options are getting added every day. In fact, the number of new ideas getting introduced is growing exponentially, so the chance of ever catching up is getting smaller every day.

So now what?

If it turns out that it is impossible to make any sensible assertions on the validity of the combination of options we can choose from, then it makes you wonder if we’re doing something wrong in our industry.

I want to answer that question with both ‘yes’ and ‘no’. I want to say ‘no’, because we cannot escape the evolutionary mechanism that causes this impulse for diversification and specialization. It’s baked into the way the world works. In fact, we probably even wouldn’t have it another way, because it’s the thing that we love about innovation.

However, I also want to say ‘yes, we are doing it wrong’, because we really suck at hiding complexity and handling diversity.

Let’s take a look at cars. There is an infinite number of cars in this world. I think I’m able to drive a lot of them without too much effort. Still, I have no clue what’s going on under the hood. A car is doing a really good job of hiding the essential complexity of getting the car in motion. It offers me a simplified interaction mechanism that allows me to get moving: press the pedal to go faster, turn the wheel in order to change your direction and press that other pedal to stop faster.

With the things that we made up so far in the software industry, that is hardly ever the case. React created a new language to produce HTML, but you really cannot use it unless you understand HTML itself. It also creates alternative ways to produce CSS, but you still need to understand everything CSS has to offer in order to make use of it.

It’s as if, before you start driving in a car of a particular type, you first have to dive under the hood and not only inspect how everything is wired, but also understand the inner workings of your engine.

Needless to say that the same also goes for the server-side. Kubernetes is the dominant way of getting things on the server, but there is no single way to do it, and if you want to understand the consequences of doing it in a specific way, you really need to understand how your Kubernetes configuration translates into the lower-level fabric of your server configuration.

Where to look for improvement

From that point of view, there is a lot to say in favor of things like Zeit, Netlify and perhaps Nuxt. At least there is the ambition to introduce a clean abstraction layer. You don’t have to be able to grasp the entire stack. You can just assume that abstraction layer to behave according to its claims without having to understand how it’s achieving it.

I just hope there will be a time where — if you want to get something out there in front of customers — you no longer have to worry about EC2 instances or Kubernetes clusters, or gateways or any of that. Instead, you’re able to program to an abstraction that you can just expect to behave in the most sensible way. We need better decoupling. We need to be able to get into a driver's seat and use code to drive the application and leave it to do the manufacturers of the thing we drive around to have the best way of interpreting our code. It should not require us to understand the entire stack, from the higher-level user interaction patterns we’re offering to our end-users down to the way bytes are floating through registers and the network.

Having proper abstractions will require a firm information hiding regime and a keen eye for what we — the programmers — really need. We don’t need a myriad of ways to get things done. We need just a single and simple way to express what we want to get done, and if we can rely on that, then we don’t really care how it’s getting done.

I’m sure many people when reading this will start to feel really uncomfortable. They do want to control how it is done. But ask yourself if that really is what you need. Perhaps, it is just yet another consequence of not getting the right thing from the lower-level abstractions and therefore being forced to go in at a deeper level to be able to control that layer too.

I would be in favor of getting treated more like a customer. I would be in favor of having the people behind competing lower-level abstractions to get together to see if they can come to some sort of alignment, and to agree on an abstraction that truly starts hiding more, so that I can spend more time working on understanding the needs of my customers, instead of spending all my time thinking about the lower-level plumbing.

Having proper abstractions might also require something else. It might also require people getting together to investigate if the layers that were introduced in the past still make sense today. Perhaps there is a way to ax out an entire set of abstractions, and see the entire stack reduce in size as if it’s a game of Tetris.

On that, I think Dark, the programming language is onto something. They are trying the Tetris approach by offering a simplified programming model to API developers, and then replace a whole slew of complexity by rethinking what it means to get that to land on production infrastructure, not shying away of ripping out some elements that we have always, perhaps incorrectly, considered essential.

I truly hope that I will still see the day when creating customer-facing software on the web has decreased in complexity. If that happens, then I’m sure it will have been a consequence of introducing abstractions that are better aligned with the intent of what we want to achieve.

In the absence of a real answer

We are clearly not there yet. But that means the complexity of it all has reached a state in which we cannot sensibly reason about it any longer. So what can we do?

• Pick solutions that explicitly aim to hide complexity.
• Pick solutions that allow you to achieve something quickly.
• Pick solutions that are easily composed.
• Consider the balance between early adoption (competitive advantage) and wide adoption (support guarantees).
• Assume everything to be disposable.
• Accept that there will be a time in the future where you will realize you got it wrong.
• Pick solutions that define their operation in terms of behavior using plug points that can be customized to your needs.
• Avoid solutions that force you to implement plug points of the get-go. There should be sensible defaults.
• By the time you realize you need to implement some of these plug points in another way, consider ripping out the entire thing and start over again.
• Don’t get lured into thinking that a certain solution has everything you will ever need. It won’t.
• Standards offer some level of assurance that there will be degrees of freedom in the future. But if standards are not driven by the needs of the consumers of these standards, they won’t be of much help either.
• If it looks overly complicated, that’s a tell.
• If it cannot be made to work in half a day, that’s a tell.

And perhaps, above all, if you are experienced, trust your gut feeling. Now, that last remark justifies its own blog post, and I promise to write it once I have the time to do it, but think about this way:

We are trusting our very lives to software that has been trained to help us make decisions. In the training phase, the software learns how to give a proper response to data being passed in, without really understanding why that is the proper response. It doesn’t really understand the underlying mechanism, and it certainly does not have a way to explain the underlying mechanism. It just knows how to respond because it learned to identify patterns.

If we trust computers to make decisions based on hidden patterns in data without being able to explain the underlying mechanism, then why do we hold different criteria for human decision making? Breaking down the problem into smaller concerns and mapping features of a technology to the concerns that need to be addressed is certainly not guaranteed to give you the best answer.

Perhaps relying on the gut feeling of experienced people works just as well; their gut feeling will be based on a livelong exposure to choice and consequence patterns. They might have only registered some of it subconsciously and — just like computers —they might not be able to explain the underlying mechanism, but —again, just like computers— they might be doing surprisingly well in helping you to make the right choices.

(As it happens, I think of myself as being that experienced: I had the opportunity to be involved in the decision-making process in a wide variety of situations and tended to be around long enough to experience the consequences. If you are looking for a second opinion regarding some technical decision that needs to be made, and you want to tap into that experience, feel free to get in touch at my firstname at eastpole.nl.)