The Fear of Picking a New Tech Stack
It all started with a simple question asked by Yerachmiel Feltzman: “DBT experts — do you know SQLMesh? How do they compare? Does it make sense to start a new (small) project with it, instead of DBT to take advantage of DBT lacking features included in SQLMesh as indicated in the link below? https://sqlmesh.readthedocs.io/en/stable/comparisons/”
As part of the elite programmers working at Tikal, I am faced with a question like this a lot. I do not mean the regular question comparing two mature projects such as Liquibase or Flyway. The question I am talking about is comparing a mature software product to an alternative but less-known software product.
The question beneath
There are plenty of articles on the internet, comparing anything comparable and even incomparable. So when someone is asking a question like this, what is he really asking? In my opinion, more often than not the underlying question is:
“What am I missing that everyone else is seeing?”
The Usual Dilemma
Let’s set the two software products side by side and list the pros and cons.
The Mature software, well, it’s mature. Fewer bugs, hopefully, a large community, big vendor backing, a lot of knowledge and articles on the internet, you get the picture. In one word: Confidence!
The newer software, well, it’s new! Not 100% polished, not all features yet, some bugs, but a real problem is the lack of documentation. I don’t mean the “documentation”, I mean the articles, blogs, and tweets people share problems, tricks, and workarounds with. THAT’s the problem. In one word: Early Adoption!
So why even consider it?
My answer to his question (above) was the following:
“It looks interesting, but it has long lost the race against DBT. As everyone is adopting DBT, and the tooling is growing at an explosive rate, I think SQLMesh, as interesting and/or good as it may be, will fall behind just for that reason alone. The real question is whether the “missing” features in DBT are something you need and can’t live without and whether this feature has not been solved/appended by a tool of some sort.”
Let’s dissect the answer real quick. In my point of view, there is nothing wrong with trying out new software, or not using mature software, as long as the reason behind that decision is based on merit, and not on bias.
When choosing a technology stack you are actually choosing the path which you think is best for your product over time.
It is much easier to go with the flow. You just use the mature and well-documented software, you know what you know, and there is little you don’t. Even if the chosen software does not exactly match your needs, it might even be an overkill for your use case, you still feel in the safe zone, and that’s what’s important! Or is it?
Food for thought
Besides the drawbacks I stated above, there are other issues to consider, which might shed a more positive light on new software.
Being new, developers of the new software might have
- Learned from the mistakes and problems the community had to deal with using the prior software
- Circumvented many problems existing in the prior product
The new software might surprise in other areas as well:
- it might even look like a better version of the prior software,
- it might even fit your needs exactly.
Even more, it might come with a new, fresh, although small but vibrant and enthusiastic community.
What I mean is
There was once this client who was in need of a message queue. No low latency high availability crazy traffic style message queue type o’ stuff. Something simple. They chose Kafka of course. Why? Because it is considered to be the de facto standard message queue today. And they didn’t even need the persistence or exactly once or any of that.
Why is everybody still insisting on using Apache Kafka when there are many other viable options, such as Apache Pulsar or NATS, or even RabbitMQ?
Or for Example
I am working with Apache Airflow for some time now. It is a great piece of software. Truly magnificent. I remember organizations started using it even though it was full of bugs and crashed every other day. But even today, I can’t define different code bases for Airflow, and I can’t even have a test environment on my single Airflow installation. And there are even more bothering basic issues which other, newer workflow orchestrators, have long since solved.
Why are we still using Apache Airflow when there are many other alternatives such as Prefect, Dagster, Flyte, and Mage, just to name a few?
What if you might just take the road less traveled and give a new technology stack a chance?
We don’t know what we don’t know
This is what he wrote back in answer:
“Interesting point of view. It’s more or less the same argument against Polars (over Pandas), Mage, Prefect and Dagster (over Airflow), Redpanda (over Kafka) …you get the point.
Yet, people still push those new tools for some reason and they might get the upper hand eventually. Btw, it’s worth knowing that DBT Labs also had to fire a lot of people.. so who knows what the future holds.
https://www.getdbt.com/blog/dbt-labs-update-a-message-from-ceo-tristan-handy/. If I really need a DBT-like tool, I would start with DBT, since it’s the default nowadays, but I would try comparing after-fact. (AFAIK, SQLMesh has a DBT-to-SQLMesh-converter out of the box :-))”
The main takeaway from his answer is that he too, is not taking the mature technology stack for granted, but rather discussing all available options based on merit and need. It might be daunting to choose the road less traveled. It is venturing into the unknown, dealing with problems to which you might not find a Stackoverflow answer. But, on the other hand, it might just be the right decision for you!
The fear of making a fool of yourself cannot be a factor in the decision making process
Bottom Line
I will always consider a new technology stack if it fulfills the following conditions:
- The software is actively maintained and there are timely releases
- It is FOSS with current active big brand customers/users
- All basic needed features are supported for me to even consider it
- It fits my needs exactly, and not just as a side effect.
- It solves one or multiple problems the mainstream technology stack has
Best Practices
This is what sets elite programmers apart. We are willing to try new things, and we are willing to deal with new problems as they arise and challenge us. Fear of failure does not dictate our commitment to excellence. For us, choosing the mainstream option, just for being the mainstream option is not a good enough reason to choose a technology stack.
I’ve learned quite a few things in my 25+ years doing this. No matter which technology stack you choose, you always have to pay a price. The question is only whether, after paying the price, you end up with a technology stack that kinda fits your needs, or whether you end up with a technology stack that fits your needs 100%.
