BCL: Cermati’s CLI Tooling Framework and Simple Package Management for Development Tools
We started with an idea to build a standard interface for our internal development tools in January 2019. At the time, we learned about a bash application framework called Bash CLI developed by Benjamin Pannell at Sierra Softworks — the code is available on GitHub.
We explored Bash CLI a bit to see what it can do and if it’s well-suited for our tooling development needs. After reading the code and testing the behavior a bit, we found it nice to use due to the well-defined structure of the project, the autocomplete feature it provides, and the flexibility to allow us to build our internal tools with different languages and stacks while having the Bash CLI application as the front-end — acting as the command-line interface for the developers to call the utilities.
But for me, Bash CLI feels a bit lacking in terms of automation. Coming from my experience as a Ruby on Rails developer — Ruby on Rails is famous for its scaffolding functionalities to allow faster development iterations — for a few years before I joined Cermati in 2017, I was disappointed that Bash CLI didn’t have the project scaffolding functionalities to quickly initialize a project and generate boilerplate code to allow developers write their code more quickly without having to manually copy and paste the boilerplate code by themselves from the Bash CLI sample project.
We played a bit with the Bash CLI code and restructured it a bit while adding the project and boilerplate code scaffolding feature. Since it ended up being quite different from the original Bash CLI framework’s standards regarding the structure of the projects — also, we somewhat built it according to our assumed needs — we decided to simply fork the project and renamed our version of Bash CLI into BCL (see on GitHub).
Later on, we added a simple package management functionality to allow us to easily package and distribute the tools we built using BCL. As the current BCL is already quite different from the original Bash CLI, in this article we’re going to talk about what modifications we made and why we think BCL is better — for us in the Cermati engineering team, at least.
We strongly recommend taking a look at the BCL GitHub repository’s README to see how to set up BCL and a quick intro to the functionalities. Please note that some of the features and behaviors might not be very well-documented yet in the README at the moment this article is published.
In this article, we’re going to focus on what we added to Bash CLI and release the modifications as BCL instead of how to set up and use BCL.
Scaffolding Functionalities
As mentioned before, Bash CLI didn’t provide us with scaffolding functionalities. This made us a bit disappointed since it’s quite tedious to set up the project structure and boilerplate code on our own every time we want to initialize a new project or add new functionality into the CLI tool we’re building.
To implement the scaffolding functionalities, we need to provide the template files for the code related to the functionalities provided by BCL as a development framework. Bash CLI is intended to be a fully Bash framework that can work with any UNIX-like environment with Bash shell with minimum extra dependency, and BCL is intended to stay faithful to that.
We went with Jinja-like placeholder variables in our base framework code template, as available in the framework/cli boilerplate code template on GitHub.
...
APP_DIR=`dirname "$(readlink -f $0)"
PRJ_NAME="{{ project_name }}"
APP_DIR="$APP_DIR/$PRJ_NAME"
...The {{ project_name }} placeholder string in the template file is to be replaced by the project name passed as the argument to the project initialization command using sed after the template files are copied into the initialized project’s directory — the implementation of the placeholder-value replacement can be seen in the init command implementation on GitHub.
...
mkdir -p "$CUR_DIR/$PRJ_NAME/framework"
...
cp $(ls -alh $HOME/.local/bin/bcl | awk '{print $11}' | sed 's/\/cli/\/framework\//g')* "$CUR_DIR/$PRJ_NAME/framework"
...
sed -i "s/{{ project_name }}/$PRJ_NAME/g" "$CUR_DIR/$PRJ_NAME/framework/cli"
...The template files are initially copied into the project files as they are, then the new files’ placeholder values are replaced with the new values according to the project name passed into the init command using sed.
The decision to use Jinja variable-like placeholder syntax was driven by our familiarity with Jinja templates — since we’ve been using Ansible for configuration management and orchestration for a few years — and the absence of conflicting syntax in the Bash CLI implementation.
sed is used to replace the placeholder values in the template files simply because sed is readily available by default in the usual UNIX-like OS environments we’re familiar with. Therefore using sed wouldn’t require us to install an extra package for BCL to work, and it sticks to Bash CLI’s minimum dependency philosophy.
Package Management
We already have a scaffolding function for our CLI tool development framework and we can start building our internal tools. But now we need to come up with a method to easily distribute the internal tools we’ve already developed.
During the installation of BCL, we should already have Git installed since BCL should be cloned from the Git repository. So we proceeded to use Git repositories as a way to distribute our CLI tool packages as we don’t consider Git as an extra dependency that needs to be installed by the users of BCL.
A BUILD file will be generated on the initialized BCL project by the scaffolding during project initialization, which should contain the following information.
packagename-v1.0.0-release
git@github.com:gitorganizationname/repositoryname.gitThe first line of the BUILD file contains the name of the package to be distributed, with the naming convention of [package-name]-[version-number]-[build-info]. The second line of the BUILD file should refer to a Git repository where the package will be published and distributed. The person publishing the package must have write access to the distribution Git repository to successfully publish the package, and the published package will be available in the target repository as a branch — the branch name will be taken from the first line of the BUILD file, which should follow BCL’s package versioning convention.
Git doesn’t seem to have an explicit maximum number of branches it can maintain in a repository the last time we checked, so the storage capacity and policy should be the limit for how many package versions we can maintain in the distribution repository.
The released package is supposed to be installed to a project by using a package dependency list file called BCLFile which should have contents such as the following.
git@github.com:gitorganizationname/repositoryname.git
package1-v1.0.0-release
package2-v1.1.1-release
package3-v2.0.1-releaseThe first line contains a reference to the distribution repository, while the other lines contain the name of the packages to be retrieved from the repository. The person installing the packages should have read access to the distribution repository to successfully install the packages.
The packages can be installed by calling the bcl package install command from the directory where the BCLFile is located. More details on the package manager should be available in the README file on GitHub.
Global vs Local Autocomplete Configuration
Bash CLI is designed to run on Bash shell, which is the most widely-used shell for GNU/Linux operating systems. Hence the penguin reference in the above image.
The local autocomplete config is another feature we added into BCL as a modification of Bash CLI, as we noticed that Bash CLI keeps the configuration files it requires to perform the Bash autocomplete feature inside the/etc/ directory. This requires Bash CLI to have sudo access for the autocomplete feature to work, and if the host is used by multiple BCL users the autocomplete configurations might get mixed up among the users.
On BCL, we modified this behavior so that the autocomplete-related configurations are stored on each user’s local context in the OS inside the user’s home directory instead of inside the /etc/ directory. This allows non-sudoers to use BCL’s autocomplete feature while also keep the global OS environment clean and avoid the problems of having the users’ autocomplete configurations getting mixed up with one another.
Conclusion
BCL has been an important part of Cermati engineering team’s development environment since we started to use it as the development framework for our internal tools. Our public key infrastructure is one of the projects that’s powered by BCL, along with some other utilities we provided for our developers.
The BCL framework and code is somewhat hackish due to it being designed to work with minimum extra dependency on the default Linux system installation. But it helped us in defining our tool development standards — enforced by the scaffolding features we added to the framework — while providing some extra features for the tools’ users such as autocomplete and package management.
The implementation of BCL might not be very sophisticated in terms of engineering complexity or elegance of the solution. The package management feature is kind of basic and a bit rough in the implementation, and the scaffolding feature is built using the bare minimum functionalities already provided by the OS environment. But from a more pragmatic point of view, it can be considered a successful project for the value it provides in Cermati’s development workflow and the adoption of the tools built using it.
