Choosing the right language for DevOps, our way.

Photo by Possessed Photography on Unsplash

We believe in automation, that if a task can be automated, it shouldn’t take time and energy from us. Of course there’s a trade-off. When you automate a process, you’re paying in advance, hoping that in the long run it would worth your while.
In Sanjagh, we tend to lean toward automation in that balance. We think of automation as an investment. You invest your time and it saves your future time.
So we have lots of little and big scripts here and there, which are not part of our core product. These scripts are in charge of various tasks like validating projects, synchronizing configurations, deployment automation, etc… . Such tasks are usually categorized under DevOps umbrella.
Historically, we didn’t follow specific best-practices or even conventions for our automation scripts. We didn’t even choose a standard language for them. For each task we chose a language that we thought was most convenient at that time and for that specific task. So you can guess that we had Bash here , Python there and even Go somewhere else! Until some day we had a revelation — which must be obvious to you if you have have a DevOps background — that a DevOps script must be treated as a first-class-citizen in our code base. Our DevOps programs should have their own processes (e.g Code Review, test, etc…), We should follow specific conventions and best-practices like any other project and Choosing a language would be an obvious, basic first step. But how ? Before choosing the right language, we must think about the required characteristics of an ideal language for our typical DevOps tasks.

Disclaimer
Before I dive deeper, I should point out that I’m not a DevOps expert. I’m just a software engineer who likes to automate everything!
I should also point out that I’m not trying to sell a silver-bullet. I believe that every team is unique with it’s own characteristics, strengths and weaknesses and must find it’s unique solution. Although we must learn from other’s experiences, but it doesn’t mean that we can or should directly apply them. So, from now on , whenever I say `Us`, I mean `Us` literally. `Us`, the developers at Sanjagh, which means that our solutions might not be applicable in your environment.

Photo by Javier Allegue Barros on Unsplash

How did we decide ?
We think that DevOps tasks starts within Dev teams, not inside an independent DevOps team (I should note that we don’t maintain our own Infra-Structure, we use Kuberenetes as a service). In this part, I try to organize the different characteristics of a programming language that `we` think are important for `our` DevOps programs.
The general pattern is choosing a language that makes it easier for software engineers to write their own automation scripts. Developers might see automation as a luxury, not part of the core task. So the harder or the more expensive you make the automation, the less it would be done. Here are our six decision factors, ordered by importance:

1. Team experience
   One of the most important qualities is how much our team is comfortable with a given language. We don’t want to force developers to learn a whole new language or worse, a whole new paradigm to be able to automate their own tasks.
2. Running complexity
   Our programs should be easy to run. If your automation script requires a complicated environment to build/run, It will do more harm than good. For example, lets say you need a simple script to prepare the environment for the build process of your main project, and you choose Java or TypeScript as your DevOps language. What happens ? You need to setup a build environment along with it’s build system just to be able to compile your 3 line script!!! It’s obvious that no one approaches small automation tasks, because for each small script we need another process (or even automation script!!!!) just to get started!
   On the other hand, if you use Bash, you almost need no preparations (as long as you use Linux), You directly run your scripts. Or if you use python, you are in luck (as long as you don’t need any libraries).
3. Being high-level
   What do our DevOps programs do ? Mostly text processing (config extraction or injection, code validation, etc…) plus a little interaction with the OS (file operations, Network calls, etc…), the usual stuff that any general purpose programming language can do. We’ve never required a low-level feature, like talking to the hardware, low-level OS calls, etc… . So the more high-level our language is, the better. The more concise and readable the code is, the happier our developers would be! Verbose languages make it hard to write even small scripts, Our evil minds would trick us to think that this extra effort make the automation unworthy, specially if our core programming language is high-level and concise and our DevOps language is verbose and cryptic. For example, let’s say our core language is Haskell and we choose Go as the DevOps language. The result would be that we would never do the automation ourselves, because we might write some 200 lines of Go code to do the exact thing that we normally write with 50 or less lines of Haskell code for.
4. Typing discipline and IDE/Editor support
   We love static, strong types. We think that typing discipline makes the code more readable and refactoring easier. Using advanced type systems the right way, prevents lots and lots of errors, specially when used along-side immutable data structures. Besides, you’ll get far bettor support from your IDE when your language is statically typed.
5. OS interoperability
   Such programs usually interact a lot with the underlying system, from file-system manipulation to running other programs. So the language that you pick must make it easy to interact with the OS. Note that I wrote this item at the 5th position on purpose, Because almost all modern general purpose programming languages have good support for this feature, either in the core language or as a library.
6. Community and library support
   Our DevOps programs interact a lot with external services like Gitlab, Discord, Kubernetes, etc… . So It’s good to have access to various libraries, so that you won’t have to re-invent the wheel. Naturally, choosing a popular language means that there are tons of ready to use, open source libraries for communicating with famous tools and systems.

What is not important ?
Here are some factors that we thought were important at first, but over time we’ve concluded that they are somehow irrelevant.

1. Performance
   Our automation scripts are usually small programs that are run either as a git hook or as a step in our CI/CD platform. So using a language that is well known for it’s performance wouldn’t be a big deal here. Let’s say that we write our program in a low-level and fast language, so that it would take 300 milliseconds to run instead of just one! who cares ? Our programs won’t need to be scalable too. Such concerns are usually rooted in core product where the program can have performance/scalability issues.
2. Being cross-platform
   This is another one that totally depends on the context. Our automation scripts are either running in a CI/CD workflow or as git hooks. And we (developers) mostly use Linux and rarely mac-OS. So being cross-platform is not really an issue here. From what I’ve said about the environment, our DevOps programs need to be
   1.Very Linux and container friendly, which means that it should be able to run on any mainstream Linux distribution with no or minimum effort 2.Runnable on mac-OS, even if it requires a lot of effort.
   Also it does not require to support 32-bit environments.
3. Low-level communication with the OS
   You might have heard that many DevOps programmers choose Go-Lang because it’s more of a low-level language, since they face scenarios that require low-level interaction with the OS or hardware. Well, not in our case. We’ve never seen such a scenario. Note that I’m not saying that you won’t face such scenario in any company. Our problems are usually solved at a higher level of abstraction over OS. So, being low-level is not an advantage for us.
4. Advanced concurrency
   Although most of the steps in our DevOps workflow are linear, we need simple fork/joins or async-tasks here and there, but it’s nothing complicated.

So, how does Scala fit in ?
I’ll go through our 6-part filter and examine scala through that lens.

1: Team experience
As I’ve said before, most of us know Scala and use it on a daily basis. So in terms of the first criteria (Team experience) it’s a match made in heaven.

2: Running complexity
You might think that Scala isn’t good for scripting because you need a whole project and build system like SBT for simplest, one-liner scripts, and you’re not totally wrong ( Note that SBT is really easy and lightweight, but when you think about creating a project every time you need a small scripts, it won’t look that light!) . Although you can run single file scripts directly with Scala (without a build tool) but you’ll have to say goodbye to using libraries and IDE support. But Ammonite changed it all. With Ammonite we can have just a single file and use any dependency we want using Ammonite directives like magic imports. Now we can have have a very small script (i.e. just a couple of lines) or a big one, without the burden of common scala projects. Specially when we use Ammonite Docker Image along side Docker multi-stage build, we end up with a very clean and light-weight build process.

3 & 4: Scala: A high-level static-type language with awesome IDE support
This is actually where scala shines. Scala’s powerful abstractions, specially it’s collections library makes it very easy to write any algorithm in a readable and easy to understand fashion (Given that you know scala and a little functional programming). Also thanks to Metals, Scala has awesome LSP support which means that you can use any popular IDE/Editor and have a good IDE experience. Also note that before LSP, Scala IDEs could not support single file Ammonite scripts well, but now metals supports it well. Scala’s advanced type-system makes it easier to write high quality code.

5: OS interoperability
Scala can use both Java’s NIO and legacy IO packages and also it’s own utility modules for communicating with the OS, but it’s still a little verbose. Thanks to Li Haoyi the Ammonite OPS library makes it really easy and fun to do most of what we need in a regular DevOps program, like working with the file system and spawning/managing sub-processes.

6: Community and library support
I think that most general purpose programming languages cover this area and Scala is no exception, specially since it can easily use Java libraries. But in the end I think more popular languages like Python have better support.

What other languages did we try ?
1. Bash
Originally, most of our scripts were written in Bash. But overtime We learned that it’s too hard for us to maintain our scripts. Of our six criterion, Bash has only number 2 and number 5!!!
2. Python
In our case, it didn't take long before python was ruled out. Although it seemed to have most of our conditions (all except number 1 and number 4) but number 1 was a huge issue for us and also number 2 has a few problems. Because although you’d expect python to be installed by default on most Linux-distros (or at least easily install-able), when it comes to python libraries everything changes. If you are sure that you’ll never need a library and everything you need are provided in core python,there’s no problem, but using libraries in automation environment creates some problems.For example, the fact that python libraries are installed globally by default, using python libraries in git hook scripts creates hell over time.You can use something like VirtualEnv for python but that’s just extra steps in CI and in our experience, breaks a lot in the process, specially because some packages have system dependencies that must be installed through distro’s package manager.
3. JavaScript
JavaScript land had it’s own problems for us. The vanilla JS was not an options since it didn’t satisfy our first and fourth conditions. Each of other flavors had it’s own problems too. For example we had ReScript (ReasonML back then) experience, but it was very hard to interact with the OS via ReScript.
4. Go
Go seemed promising, until we actually solved a couple of problems with it. Since we use Scala on a daily basis, Go is painfully verbose for us. Not to mention that almost none of us had real experience with Go.

Case-study : Nginx config
So, let’s finish with a real-world example. If you checkout Sanjagh’s sitemap, it has a couple of different areas which are from the top segment in URL, like `/services` or `/categories`. But there is a little problem. We have several zones (46 at the time of this writing) which are top-level URL segments because of their importance (e.g. /tehran). So you can guess that we have something like this in our Nginx config:

which is, well, disappointing. We wanted an easy way to read the list of our active zones from a file (or potentially get from another service) and populate this segment. And 40 lines later, here is our magical script !

What does this script do ? It scans all Nginx files in given path and search for this pattern: ‘@FILE[file_name...separator]'where file_name is the name of target file which contains our data, and separator is any string that is used to join the data in target file. for example in our case, instead of that disappointing location block you’ll see something like this :

Now, all we have to do is adding a step in our multi-stage docker file which uses the Ammonite docker image and just runs our script.