Infrastructure as Code reminds me of “make run-all”

As I was preparing to move on from my 17 years at Google and thinking about what I wanted to do next, I reflected on my career.

Back at the beginning of my career, I learned how to use make, awk, tar, RCS, and other UNIX tools (this was before Linux existed). I wasn’t building continuously running services at that time, but I did build and run programs that ran in debugging, profiled, traced, and optimized configurations, with different inputs, on different hardware, on different operating systems, on different sizes of supercomputers, using different message-passing systems or threading libraries, and so on.

To build these different variants of the programs, I remember either writing scripts to set lots of Makefile variables containing compiler flags, C preprocessor macro definitions, and so on, and/or writing rules to invoke make recursively with those variables set various ways. The builds had to be performed in different directories, or all of the build artifacts had to be deleted via make clean before building with different options. Everything was in version control in case sources were accidentally overwritten or deleted.

To run the programs, I’d write rules like (though using pattern rules):

.PHONY: run-all run-program-a run-program-b
run-all: run-program-a run-program-b
run-program-a: program-a program-a-input
  program-a $(PROG_A_EXEC_FLAGS) < program-a-input
...

This would build objects, executables, and other sub-targets in dependency order, generate inputs as needed, and run the programs. Artifacts that were already built and up to date were not rebuilt, and the same technique could be applied to execution by generating timestamped files (e.g., with output) in the case of a successful execution. awk was handy for performing substitutions on input files. If the programs and/or inputs needed to be submitted to a batch queue, supercomputer, or another machine, they could be bundled up using tar. Files were transferred with ftp.

Though the tools, languages, package formats, protocols, hardware, and execution environments have changed, today’s build, configuration, and deployment procedures don’t seem much different more than 30 years later. We make some changes, commit them to version control, run a build, perform some substitutions, push some artifacts, and execute some commands to deploy.

Near the beginning of the Kubernetes project, I proposed using a build approach to generate Kubernetes configuration files. 9+ years later, some users have adopted a similar approach, sometimes called the Rendered Manifests Pattern, to generate Kubernetes configuration files in CI. Once they are generated, they are then deployed automatically via GitOps. Some teams use a similar approach inside Google to render the (~20-year-old) Borg Configuration Language to protocol buffers.

The top reason mentioned in that explanation of the rendered manifest pattern is to eliminate obfuscation incurred by complex configuration formats. That’s a goal I can agree with.

Infrastructure as Code and configuration template files are frequently heavily parameterized. Here are a couple short example excerpts:

Helm:

{{- if .Values.master.usePodSecurityContext }}
      securityContext:
        runAsUser: {{ default 0 .Values.master.runAsUser }}
{{- if and (.Values.master.runAsUser) (.Values.master.fsGroup) }}
{{- if not (eq (int .Values.master.runAsUser) 0) }}
        fsGroup: {{ .Values.master.fsGroup }}
{{- end }}
{{- end }}
{{- end }}

Terraform:

resource "google_compute_firewall" "foo" {
 name = "my_rule"
 dynamic allow {
   for_each = var.rules
   iterator = rule
   content {
     protocol = rule.key
     ports = rule.value
   }
 }

On the other hand, fully rendered, that Terraform example would look like this:

resource "google_compute_firewall" "foo" {
 name = "my_rule"
 allow {
   protocol = "icmp"
 }
 allow {
   protocol = "tcp"
   ports    = ["80", "8080", "1000-2000"]
 }
}

Unfortunately, it isn’t as straightforward, in general, to render Terraform and other Infrastructure as Code formats as it is for Kubernetes configuration files due to dynamic substitution of runtime values. It’s possible to run terraform plan and then terraform show on its output, but that doesn’t render the configuration in the HCL format. The plan can be applied, however, and Terraform users depend on heavily on performing plan before apply to understand what changes may occur.

This build, substitute, deploy procedure is certainly time-tested, but it’s a cumbersome, error-prone, often fragile process that typically involves humans at multiple stages to author, initiate, and vet changes. Some of those steps involve a fair amount of toil. Vetting the changes is made more difficult by the complex configuration formats and irreversible transformations. If there is a problem, the substitution code usually needs to be tweaked and all of the steps need to be repeated.

It seems like a simpler, more streamlined approach should be possible. Hopefully it won’t take another 30 years for one to emerge.

If you found this interesting, you may be interested in other posts in my Infrastructure as Code and Declarative Configuration series.