Why on earth would anyone do that?! Chances are, that was the first thought you had after reading the title. If you’ve been building software long enough you know that a full rewrite is generally a bad idea. It looks tempting because you think it’s going be quick and solve multiple structural problems at once. However, it almost always becomes a never-ending time sink that creates many more structural problems that you didn’t even know existed because the original version had them solved from day 1.
You also probably know that companies who did it often feel obliged to write a blog post justifying why their case is different and wasn’t a mistake at all. This leads to a growing number of pro-rewrite blog posts, and that tempts even more companies into doing it. What an irony; but hear me out — our case is different!
Is your tool in Golang?
This is literally the question we got asked several times by users. Not even “what is it written in” — straight up “is it in Go?”. Apparently people not only care what Digger is written in; they cared specifically whether or not it is in Go. And unfortunately, it was in Python (sorry all the Pythonistas out there! Nothing wrong with it).
It just so happens that Golang is the go-to language in DevOps tooling (pun intended). We didn’t know that before starting the latest iteration of Digger. Naturally, we used the same language that the previous 4 iterations were written in — Python. Turns out, that’s suboptimal for CLI-first tools, especially if the tool is meant for automation, like running it in a CI environment.
It probably time to tell you what Digger is: an open-source alternative to Terraform Enterprise. Digger runs terraform plan and apply in your existing CI system, unlike Terraform Cloud and equivalents that are effectively dedicated full-stack CI/CD platforms with compute and jobs and logs and everything. We thought that there’s got to be a better way to run terraform than duplicating your entire CI stack; so Digger is a thin wrapper / orchestrator that seamlessly integrates with your CI toolkit like Github Actions, Gitlab or Azure Devops.
A single binary works everywhere
A command-line tool written in Python may look small but it has an implicit dependency on the interpreter, even if it ships with all the dependencies. You could of course bundle a Python distribution with the tool; but that makes the package super heavy. And if you don’t do that, there is no guarantee that everything will work as intended on the host platform. A single binary on the other hand will always work because it has no external dependencies.
Faster means cheaper
A statically compiled binary is much smaller and fast to run. This is not about performance optimisation; most modern applications don’t need anything fancy in terms of algorithms / data structures. But even then, fast vs slow boot time is a huge differentiator. Especially if the tool is part of a CI/CD pipeline; faster means cheaper! A single binary not only uses less CPU cycles or minutes; it also uses much less RAM. Which means that you need less VMs to handle parallel runs.
We did it in 2 weeks
Yes, the whole thing in 10 working days. How is this possible?! Well, I have a dirty secret to tell: there wasn’t that much to rewrite. We started Digger as an “alternative UI for AWS” that auto-generated and ran Terraform behind the scenes for every scenario you’d expect A PaaS like Heroku to do, like deploy a container or a database. However, we have quickly learned that this is too big a problem to tackle for a startup — simply too many edge cases.
So we “zoomed in” to focus only on infrastructure. Then we also dropped the terraform generation part and focused exclusively on terraform deployment. Which led us to realising that we are essentially duplicating a CI/CD stack, so why don’t we reuse existing CI infrastructure?
This is why the Python codebase we decided to rewriting in Golang wasn’t that huge. It was essentially a wrapper on top of Terraform that was also aware of state in S3 and stored metadata in DynamoDB. Back then we didn’t even have an orchestrator backend or an API or UI of any kind. We planned to do it in a week and we kinda did; but we were new to Go so it took another week to get it into production-ready state and move early adopters over to the new version.
The switch happened on the 20th of March in this PR with an unassuming name “copy files”. Back then we didn’t have much traction; we didn’t even know for sure that was the right thing to be building. 3 months and 700 Github stars later, of course we are now following much stricter PR naming guidelines :) Contributions welcome!
