How scala-cli Shines in Automation
Ever find yourself knee-deep in manual and repetitive tasks, that you would love to see automated, but you are so morbidly afraid of snakes that you can't even think of using Python? Or do you simply refuse to go over the pain of figuring out how to do an if-statement in Bash? Well, I have good news for you: say hello to scala-cli, your new automation best friend!
I want to show you how we are using it in some key places to improve the lives of our engineers at SwissBorg, freeing them to have more time to do what they do best: creating bu… **clears throat**… amazing new features!
What is scala-cli?
Scala-cli was created by the folks at VirtusLab, and it's an amazing tool! Think of it as a command line assistant, that is capable of handling whatever Scala code you throw at it. As they say on their website, it lets you compile, run, test, and package your Scala code (and more!)
So, what’s the big deal you may ask? Well, it essentially allows you to use Scala as a scripting language. It takes away the headache of setting up a Scala project and it also does the heavy lifting when it comes to dependency handling, allowing you to use all those nice libraries you are used to when working on a classic Scala project. If those are not enough reasons, scala-cli also works great in IntelliJ IDE, even when scripts are set inside an existing project.
Setting up scala-cli is as simple as running the install command, but going over all its capabilities is out of the scope of this article. However, here are some links that can help you to get to know more:
- Beginner intro to Scala that also shows how to set scala-cli (Yadu Krishnan)
- scala-cli official documentation
How do we use it?
At SwissBorg our automation journey with scala-cli started around the time we migrated our CI pipelines to GitLab. During the migration phase, where a lot of testing was happening to make sure everything worked as expected, we clearly saw some repetitive tasks our engineers had to do.
Automatic notification of a newly published version
Our released versions are generated from the git commit short SHA, they are built from. So, whenever someone released a new version of a service or library they had to go to a specific place to check what was the name of the just published version.
Now, we have a new step on the build pipeline, that uses scala-cli to send a a message on Slack to the author of the commit.
//> using file "clients/slack.scala"
//> using file "utils/extensions.scala"
//> using file "utils/ci.scala"
import clients.Slack.*
import utils.ci
import utils.extensions.*
// Simplified version ...
val slackClient = SlackClient(ci.SlackToken)
val recipientEmail = ci.getAuthorEmailOfLastCommit
val slackUser = slackClient.findUser(recipientEmail).getOrThrow
val message =
s"""Pipeline for *${ci.ProjectName}* completed.
|Branch: ${ci.CommitBranch}
|Version: `$version` ${pythonVersionString}
|<${ci.PipelineUrl}|Pipeline link>""".stripMargin
slackClient.send(message, to = slackUser)Automatic Helm charts update
We deploy on K8s clusters, and we use Helm to help with that. Therefore, every time a new version of a service was published/released we had to update our Helm charts. That is still true in some scenarios for some reason, but for others, we agreed that we could automate that update.
Piggy-backing on what we did on the previous automation, we introduced some logic to properly identify the scenarios we wanted to have the update automated and then re-used the send notification logic to notify the commit author that the update was done.
//> using file "clients/gitlab.scala"
//> using file "clients/slack.scala"
//> using file "utils/ci.scala"
//> using file "utils/extensions.scala"
import cats.syntax.either.*
import clients.Gitlab.*
import clients.Slack.*
import utils.{ci, Constants}
import utils.extensions.*
// Simplified version ...
def sendToSlack(msg: String)(using slack: SlackClient): Unit =
Try {
val recipientEmail = ci.getAuthorEmailOfLastCommit
val slackUser = slack.findUser(recipientEmail).getOrThrow
slack.send(msg, to = slackUser)
}.toEither
.leftMap(err => println(s"Notifying user on Slack failed: $err"))
.merge
def swapOldWithNewVersion(...): String = ...
val fileContent = getFileContent(filePath, mergeRequest.source_branch)
val updatedContent = swapOldWithNewVersion(...)
val newCommit = NewCommit(...)
val commit = gitlab.pushCommit(newCommit).getOrThrow
sendToSlack {
s"""Commit: <${Constants.ProjectURL}/-/commit/${commit.id}|${commit.id}>
|Source: <${ci.ProjectUrl}|${ci.ProjectName}> | <${ci.PipelineUrl}|Pipeline>""".stripMargin
}Automatic changelog update
We were manually maintaining the changelogs of our services and libraries. This was part of the process when releasing a new version, which was easily forgotten or with entries added out of place.
GitLab has a set of nice features around generating changelog entries based on git trailers that a commit contains. Based on those they can compare two commits and generate new entries with the changes according to predefined templates.
This required some setup on the repository side, but after that was done, we just had to create a new automation step on our pipeline which calls the GitLab API to generate the new entry and commit the changes.
The great part here is that we were able to leverage technologies we are super used to using on a daily basis, like sttp.
//> using file "clients/gitlab.scala"
//> using file "utils/ci.scala"
//> using file "utils/extensions.scala"
import clients.Gitlab.*
import utils.ci
import utils.extensions.*
// Simplified version ...
val gitlab: GitlabClient = ...
val tags = gitlab.getLastTwoTags.getOrThrow
val newEntries = gitlab.generateChangelog(tags._1.name, tags._2.name)
val newCommit = NewCommit(...)
val commit = gitlab.pushCommit(newCommit).getOrThrow
println(s"Changelog created: Commit ${commit.id}")Automatic Protobuf linting
SwissBorg relies heavily on Protobuf and gRPC. In order to keep compatibility guarantees alongside the evolution of our APIs, we developed tooling that uses Buf to lint our Protobuf descriptors and processes the results into a suitable format that GitLab can read. With this, we were capable of presenting to the engineers in a very nice and integrated way, the impact of their changes directly on a pull request page.
However, more often than not, a Protobuf API is new and has been in development for some time, meaning that is constantly changing through iterations, which conflicts with the linting rules we had in place. To solve this, our tool started to allow descriptors that could be excluded from the analysis. However, we didn't want those exclusions to be "forgotten" forever, so we added another scala-cli automation capable of identifying them and scheduling reminder notifications to the author.
//> using file "clients/slack.scala"
//> using file "utils/extensions.scala"
//> using file "utils/ci.scala"
//> using file "utils/loaders.scala"
import cats.syntax.either.*
import clients.Slack.*
import utils.ci
import utils.extensions.*
// Simplified version ...
val slackClient: SlackClient = ...
val today = java.time.LocalDate.now()
def makeReminder(entries: Seq[...]): Option[String] = ...
(for {
result <- ExcludedProtobufLoader.load(...)
toBeRemindedToday = result.entries.map(_.filter(_.reminderDate.compareTo(today) <= 0)).getOrElse(Seq.empty)
msg = makeReminder(toBeRemindedToday)
_ <- msg.fold(Right(println("No reminders due today")))(x => slackClient.send(x, channel = channel))
} yield ()).leftMap(err => Console.err.println(err))
Automatic RFC creation
At SwissBorg, we have a Request for Change process in place, where every engineer is invited to participate, to foster documenting big changes from the get-go while providing a framework of constructive asynchronous discussion on the topic.
Creating a new RFC, however, can be a boring and intimidating task, since one has to create the template and put it in the correct place. Therefore we used scala-cli to provide a tool that can fully automate that process.
Since we host the RFCs in their own repository, whoever wants to create a new one simply needs to clone it, make sure scala-cli is installed, and run ./newRfc script.
#!/usr/bin/env -S scala-cli shebang -q
//> using scala 3
//> using platform jvm
//> using dep com.lihaoyi::os-lib:0.9.1
//> using dep org.slf4j:slf4j-nop:2.0.7
//> using dep com.github.jknack:handlebars:4.3.1
//> using dep org.eclipse.jgit:org.eclipse.jgit:6.3.0.202209071007-r
import ...
// Simplified version ...
val pwd = os.pwd
val rfcMax = getLastRFCsNumber(pwd / "rfcs")
val rfcNumber = rfcMax + 1
val fullyFormattedRfc = formatNumberFull(rfcNumber)
val title = askQuestion(s"Title of your $fullyFormattedRfc")
val repository = gitRepository()
val gitAuthor = repository.getConfig.getString("user", null, "name")
val branchName = s"rfcs/${formatNumber(rfcNumber)}-${kebabCase(title)}"
...
execute(os.proc("git", "checkout", "-q", "-b", branchName))
val templateContent = os.read(pwd / "templates" / "basic" / "index.md")
val args = Map(
"number" -> formatNumber(rfcNumber),
"title" -> title,
"start_date" -> LocalDate.now().toString,
"author" -> gitAuthor,
"sponsor" -> "Your sponsor",
)
val readmeContent = applyTemplate(templateContent, args)
os.write.over(rfcRoot / "index.md", readmeContent)
execute(os.proc("git", "add", rfcRoot))
execute(os.proc("git", "commit", "-q", "-m", s"init: $fullyFormattedRfc"))
execute(os.proc("git", "push", "-q", "--set-upstream", "origin", branchName)). $RESET")Other automations
Our tech challenge for our recruitment process is an exercise that can be solved in a single scala file. Therefore, to emphasize this idea further, we provide a link to a scala-cli template that our candidates can use to solve the challenge.
Another use case is the automated publishing of re-generated documentation diagrams to our internal knowledge-sharing tool (Confluence) from a GitLab pipeline.
We are currently trying out Compass and we will be using scala-cli to make sure we have the correct score data to use.
Why do we think it is useful?
SwissBorg can be considered a Scala shop. Although we use other languages and technologies, the majority of what we do is based on Scala.
For us, scala-cli was love at first sight, since we were able to start scripting, solving various ad-hoc business tasks, that were hard to tackle outside of service/project code, just using the libraries and the language we are used to. In practice, it presented us with a productivity boost and opened the doors to creative thinking to solve issues we had, like the automations discussed above.
As time goes by we will for sure find new use cases, and with the foundations already set, solving them will be easy and super fun.
Are there alternatives?
We can't say it is a drop-in replacement, but Ammonite has existed for a while now and it is also very capable of providing you Scala scripting abilities.
In our experience, the usability and integration issues with IntelliJ IDE, were what always set us back from using it more.
When scala-cli came out, the integrated experience of having it on the IDE or directly inside a project was so good that the spark was lit and we never looked back.
Trivia
At the time we started experimenting more heavily with scala-cli, we were also starting to explore Scala 3. The ability to control the Scala version through the scala-cli launcher was key for us to start developing our first automations directly with Scala 3.
This created the opportunity to use the new version of the language to solve real use cases.
If you want to hear more about our journey to Scala 3, here is a great article written by Smur89.
Our processes are far from being perfect and we are constantly looking for pain points we can solve, bringing value and quality of life to our fellow engineers. To ensure that, we are certain scala-cli is a tool we will keep on our tool belt.
