Software That Writes And Evolves Software

I’ve been quiet lately, largely because I’ve been writing a lot of code. Today, I’m delighted to start talking about some of what we’ve been working on at Atomist.

You may have seen Jessica Kerr’s excellent elm-conf presentation on creating and editing Elm projects with Atomist. If you haven’t, it’s well worth 23 minutes of your time.

This is the first of a series of blogs in which I’ll explain the concepts in the Atomist project behind Jessica’s compelling demo, and why we think this project is important.

How We Got Here

Code generation has a long and mixed history. At times it’s been used to try to hide complexity that should have been removed in a language or framework. Sometimes it has resulted in a mess of magical code that humans are forbidden to modify. Yet it has proven its value time and time again, from Rails scaffolding to Yeoman generators to Spring Initializr, to the recent Create React App.

With the rise of microservices, project creation is more and more important to individuals and organizations, as is maintaining consistency between a potentially large number of services.

Creating new projects is hard, and it doesn’t make sense to do it by hand. Or to do what most of us have been doing: Copy from the most similar project on hand and add, delete and modify to arrive at a the correct starting point.

At Atomist, we thought long and hard about what works in code generation and what doesn’t. And we believe we’ve found a better way, which can be applied to many developer technologies.

“Editors”: The Key Differentiator of This Approach

Traditional code generators offer a one way trip. They create a project, allowing basic parameterization (such as project name, base package name or dependencies) and then their contribution ends. The generation mechanism is often opaque: For example, a mix of static and template content and code in the generator itself. Our first attempt at project generation worked like this.

This approach has major problems. Editing and customizing templates is difficult, as it may require modifying generator code as well as template code. The sharp distinction between a project and a template means that it’s impossible to use normal tooling on templates, or easily verify that a change to a template hasn’t broken anything.

The solution: Model project creation as a sequence of transformations on a starting point, which is itself a normal, running project in its target technology. We call these transformations “editors.”

Some existing technologies create and evolve projects. But they tend to be platform-specific special purpose tools such as Spring Roo, which are only useful within a single community and are tied to a particular architectural style.

It’s possible, and much better, to enable the creation and evolution of projects in a way that’s independent of a particular platform, and independent of the origin of the project.

Fundamental Concepts

Here are the key concepts in our approach:

• A project template is simply a working project. Thus any project can be used as a template. The world is instantly full of templates, ready to be transformed to meet our needs.
• Editing is scriptable and composable, to maximize reuse.
• Different kinds of projects and content can be edited with a consistent approach. The approach that works for Elm should work for Java, JavaScript and Clojure. This is an ambitious goal, but leads to a superior solution that offers value across the polyglot landscape increasingly found within organizations.
• A change made by an editor is indistinguishable from a change made by a developer. No magic markers as to what can and can’t be modified by hand; editors able to follow project style guidelines and produce clean diffs preserving formatting and comments. The aim is not to minimize the role of developers, but to make developers more powerful by scaling them up via a polite, invited, automated helper that follows their preferences.
• Editors can act on projects that weren’t created by Atomist, or have been modified by hand. Projects remain non-magical at all points. This also avoids lock-in. More editors will be applicable if projects follow conventions, but conventions help people, too. In other words, editors can help more with a Spring Boot project than, say, an arbitrary Java project; more with an Angular or React project than with an arbitrary JavaScript project.
• Editors should be able to run anywhere. Any editor should be applicable locally via a command line or IDE, or in the cloud via a web form or ChatOps interaction, modifying a GitHub repo.

How To Get There

These goals mean that editors must be able to parse and understand projects (both at a language and platform level), and make it easy to work with that understanding.

This enables code like the following, which adds Netflix Hystrix support to a Spring Boot project using the appropriate Spring Boot “starter”:

The effect on the Java source tree of running this editor is:

Note how related operations are combined in one reusable editor; how it’s possible to operate on a class only if it meets certain criteria; and the use of operations such as “addAnnotation” that work on the structure of Java source files. In the cloud, Atomist expresses edits as pull requests. I’ve omitted the change to the Maven POM, bundled in the same commit.

To illustrate the consistency of approach across languages, here’s an example from Jessica’s Elm demo, which drills deeper into language structure to change the type, name, and return value of a function in a standard Elm project:

This editor requires a “module” parameter, which can be requested from the user via a CLI, web form or the Atomist Slack bot.

Projects are made of more than source code, and this approach works across file types. The following editor modifies a POM to make it meet the needs of a new user project, avoiding the need for the traditional game of “copy and edit until all the third party stuff is replaced by my own”:

As an example of modeling project generation as a transformation chain, our Spring REST Service generator works like this:

1. Start with a valid Spring Boot REST service project. This will typically be a GitHub project, and can contain any amount of custom content that should be common to all new projects.
2. Change the Maven POM as above to customize it
3. Change the root package and application class names based on user input
4. Change README.md to refer to the new project name.

Bingo! A fresh, clean, custom project. We can write code as with any other project, and at any point apply the Hystrix editor we’ve seen, user-defined editors or editors bringing in additional Spring projects such as Spring Cloud Stream that modify dependencies, code and config to provide a best practice starting point.

Jessica’s Elm template/editor demonstrates more sophisticated possibilities, using the ability to make wide-ranging modifications to an existing project to achieve a single goal, such as evolving a beginner program in accordance with the Elm Architecture.

This shows a key benefit: knowledge can be embodied in and transferred via generators and editors. This can be valuable inside an organization to spread best practice, or for an open source team who want to help their community keep up with best practice with minimal effort. It may prove a compelling alternative to traditional tutorials.

Where Next

The Atomist editor DSL and infrastructure — written largely in Scala — will soon become an open source project. I’m delighted to be starting another open source project, and am looking forward to working with the community to bring the benefit of this approach to a variety of technologies.

We’re adding comprehensive support for Java and Spring Boot, followed by other popular platforms. Of course many features, such as file operations, are valuable across all platforms.

We’ll also be enabling project creation directly from an Atomist button that can be hosted on any page. This will create a new GitHub repo for the created project, saving even the effort of setting up Git.

As Jessica showed, Atomist can also create projects via Slack. We’re excited by the potential of ChatOps to help developers streamline their daily work; more on that in a later post.

In my next posts, I’ll talk in more detail about the syntax of our editor DSL and the thinking behind it; how to extend it using JavaScript and custom types; and how to work with “microgrammars” to transform arbitrary content.

Atomist aims to free developers of as much busywork as possible via automation. Editors are a step on that journey. Please sign up for updates on what we’re building.