Automating Infrastructure with Atomist
You say, “AUTOMATE ALL THE THINGS!!!”, but do you really mean it?
Note: for current information on these concepts, see our doc site.
Software developers have long sought to automate as much of the process around software development as they could. Developers write tools to track changes in software. They automate building and testing of software using continuous integration (CI). More recently, deployment of software has been automated with continuous delivery (CD), often borrowing the infrastructure from CI. DevOps culture has pushed this automation into the running of software: alerting on and responding to incidents. In contrast, efforts to automate the process of software development have been more timid. Some teams use templates to standardize and ease project creation. Integrated development environments (IDEs) augment our understanding with features like syntax highlighting & autocompletion and provide some automation, e.g., automated refactoring within a single source code repository.
At Atomist, we have been thinking about how to push the boundaries of automating software development. In recent posts we’ve talked about how our we enable moving beyond templates to Generators and Editors so we can not just create projects but evolve them as standards and dependencies change. We also introduced the notion of Executors, which expand upon the single-project capabilities of an IDE by allowing refactors and other changes to be applied across all repositories in an organization. But as the practice of DevOps has made obvious over the past few years, a software project is much more than just writing the software. If we can push the boundaries of evolving source code, can’t we do the same for CI, CD, and running software? After all, in the infrastructure-as-code world, building infrastructure, running builds, deploying, and monitoring are code too. Using AWS CloudFormation, Travis CI, Kubernetes, New Relic, Datadog, and PagerDuty APIs, you can fully automate with code the infrastructure, testing, deployment, monitoring, and alerting of your application.
The convergence of the infrastructure-as-code and SOA models enables higher-order automation of infrastructure.
The convergence of the infrastructure-as-code and service-oriented architecture (SOA) models provides a fertile ground for higher-order automation around infrastructure. For example, developers use CI systems to automatically build software, run tests, and even deploy the software. In contrast, the configuration of these CI systems is often manual, requiring copying a configuration from an existing project, error-prone editing, etc. For teams adopting microservices, where new projects can be created weekly or even more frequently, this is untenable! Most CI systems provide some sort of API, so why not use the API and automate the process of configuring and enabling the CI system for new source code repositories?
Many infrastructure systems provide an API that can be used to automate everything from configuration to reporting
Automating Travis CI
Travis CI provides free continuous integration services for open source projects. The manual process for configuring and enabling Travis CI on a GitHub repository requires several steps spread across web pages, the command line, and source code. First, you make sure Travis CI has an up-to-date list of your repositories. Next, on your Travis CI profile page, you enable the repository. Finally, you create a build configuration, add it to your repository, commit, and push to GitHub. Travis CI will notice the push and build your project using the configuration you have provided.
Unfortunately, the creation of the Travis CI build configuration is not always simple. While copying the .travis.yml configuration file from a similar project can provide a good starting point, customization is often needed. For example, the build configuration may contain encrypted secrets, e.g., authentication information for your artifact repository, whose encrypted values are repository specific. Thus creating the build configuration becomes a multistep process: copying, editing, encrypting values, etc.
As we began to release our Rug archives under the atomist-rugs GitHub organization, currently over 20 repositories, this process became tedious and error prone. Since Atomist provides tools for automating code development, we immediately thought to use our own tools to automate the solution to this infrastructure-as-code problem.
We immediately thought to use Atomist tools to automate the solution to this infrastructure-as-code problem.
Travis & Rug
Fortunately, Travis CI provides a fairly complete API for their service. Unfortunately, Rug knew nothing about it. While Rug natively understands projects, files, and even Java source code, it must be taught how to interact with Travis CI. Lucky for us, Rug has an extensible type library, allowing us to write a type for Travis CI that the Rug runtime can dynamically load and use. A detailed discussion of adding user-defined types to Rug is beyond the scope of this post. Conceptually, we teach the Travis type how to parse and understand the Travis CI build configuration file. We also write functions that query and modify this build configuration. For instance, the Travis type has functions that enable and disable the Travis CI build and encrypt environment variables. If you are interested in the details of the Travis Rug type, I encourage you to peruse the travis-rug-type repository and ask questions in the Atomist Community Slack.
Travis Editor
Now that Rug understands Travis CI, how can we use it to enable CI builds on all of our Rug archive repositories? It’s often useful to take a working configuration and then transform it into an Editor. In our case, let’s look at the build configuration for the common-editors Rug archive:
The build configuration looks pretty standard: it defines the platform, language, environment variables (some of which are encrypted), dependency installation, build script, notifications, and caches. The build does reference a script, .atomist/build/travis-build.bash, which in turn references other build files. We will have to make sure we include all those files when configuring the build on a repository. The only repository-specific information in the build configuration are the encrypted environment variables, which the Travis type knows how to deal with.
We now have all the information we need to create an Editor to configure Travis CI on Rug archive repositories. Let’s write it:
The above shows the meat of the EnableTravisForRugArchiveDSL Editor. You can find the complete source code on GitHub. First we define some variables (lines 68–72). Once we establish that the project is a Rug archive (74), that is, it has a .atomist directory, we create the .travis.yml file (76). We then remove any existing Travis build script (78), create the build directory (79), and add all the files needed by the build (80–83). Finally, we use the Travis type (85) to enable the repository on Travis CI (87) and encrypt the environment variables needed for the build (87–90).
Quick Aside: Rug DSL & TypeScript
One unique thing about the EnableTravisForRugArchive Editor is that it has been implemented using both the Rug DSL and TypeScript. Above we discuss the Rug DSL implementation. The TypeScript implementation is available in the same repository. Equivalent implementations are provided so people wishing to develop their own Rugs can compare and contrast the Rug DSL and TypeScript approaches to determine which one they prefer. In general, if your Rugs require mostly features and functions available in the Rug DSL, then the Rug DSL approach is likely more concise. If you find that your Rug requires many JavaScript blocks to express changes you wish to make, then it may be better to use TypeScript. Of course, if you are very comfortable in the TypeScript/JavaScript ecosystem, you my prefer the TypeScript approach regardless. The TypeScript implementation uses interface definitions for Rug types so developers can take advantage of existing IDE support for TypeScript syntax highlighting, autocompletion, compilation, etc. We will talk more about how we are using TypeScript with Rug in a future post.
Running the Editor
Now we are ready to run the Editor on our Rug archive repositories. As detailed in the travis-editors's README, this Editor requires several input parameters. We can also get information about the EnableTravisForRugArchiveTS Editor and its parameters using the Rug CLI:
The Rug CLI’s describe command provides information about an Editor’s tags and parameters, and ends with instructions on how to run the Editor. Using the those instructions, we can run the Editor on the common-editors repository:
$ cd ~/develop/atomist-rugs/common-editors
$ rug edit atomist-rugs:travis-editors:EnableTravisForRugArchiveTS \
repo_slug=atomist-rugs/common-editors \
github_token=$GITHUB_TOKEN \
maven_base_url=https://atomist.jfrog.io/atomist \
maven_user=$MAVEN_USER \
maven_token=$MAVEN_TOKEN \
org=.orgWe provide the values for the encrypted environment variables as shell variables to avoid showing them here or having them in our shell history. The output from the above command looks like this:
Looking at the output we can see the Rug CLI resolves all the dependencies of the travis-editors Rug archive and loads everything into the runtime. It then tells Travis CI to synchronize the repositories for this user and waits for that to complete (sometimes this can take more than 20 seconds). The Editor then enables builds for the repository. Next, the Editor uses the parameter values provided on the command line to set and encrypt the GITHUB_TOKEN, MAVEN_USER, and MAVEN_TOKEN environment variables used by the build scripts. The astute reader will notice these steps are exactly those specified in the Editor. After it completes all the editing steps, it reports the changes made. In this case, it has created the Travis configuration and build files, updated the Travis configuration file three times (once for the addition of each encrypted environment variable), and created the .atomist.yml file, a record of what it has done. Once we commit and push those changes, we have continuous integration configured on Travis CI for the common-editors repository. You can view the current build status, see the build output, and look at historical builds on the common-editors Travis CI page.
Get More Done
The infrastructure-as-code revolution is good only inasmuch as we continually push the boundaries of automation. As described in a previous post, Rug understands your code, so it can push code automation further, whether that code be Java, Javascript, Clojure, Python, Go, AWS, Terraform, Docker, or Kubernetes. From configuring CI to adding a local Vagrant testing environment to automatically generating a Kubernetes resource specification, Rug can automate common DevOps tasks across all repositories, improving standardization and compliance along the way.
At Atomist, we are proposing a new way to think about the relationship between developers and software. Please join the conversation and help us shape the future of software.