Evolving Projects Using Atomist Editors

In my last post, I discussed a new approach to code generation, introducing the idea of “editors” as transforms on a running project.

Today I’ll show how Atomist editors running in the cloud make it easy to embody and share knowledge, and easier to evaluate and adopt new technologies.

Consider the important problem of evaluating an unfamiliar technology. How do we add the technology to an existing project without wasting a lot of time? If we can eliminate unnecessary overhead, we make it easier to evaluate new technologies, unleashing productivity and creativity.

In this post, I’ll focus on Spring Boot — although the Atomist approach isn’t specific to Java or Spring. We’ve seen its applicability to Elm and Clojure. Many tutorials can benefit from such an approach, across a range of technologies.

There are many developer technologies out there worth evaluating, but the effort needed to take the first step can be daunting. For this post, assume I’m a Java developer who may benefit from using a graph database to solve a connected data problem. How do I get started with Neo4j, the most popular product in this space?

Step 1: Create a New Spring Boot Project

Let’s start by creating a new Spring Boot project. Using a popular, high quality framework like Spring Boot not only makes life easier through the functionality you get for free; it makes your application easier to comprehend and manipulate, by hand or by machine. If you have an existing Spring Boot project, created by Spring Initializr or by hand, you can go on to Step 2.

The quickest route to a new Spring Boot project on GitHub is the Atomist Spring Boot template. You can use the button in the repo’s README or click on this button to create a new, customized Spring Boot project:

You’ll be asked to authorize Atomist on GitHub. This not only allows Atomist to create the initial repo, but enables Atomist editors to suggest further modifications via pull requests.

Provide the few parameters specified in the template, and you’ll quickly have a shiny new, personalized GitHub project. As I explained in my last post, project generation works by starting with a running (template) project and transforming it using editors.

Step 2: Run an Editor to Bring in Neo

One of the Spring guides explains how to add Neo4j to a Spring Boot project. Spring has always had excellent documentation, and it’s a fine guide, but we can do even better.

Think about what we’d want to happen at this point in an ideal world. We’d want to keep our package structure and any existing code we’d written, with Neo4j magically appearing in our project. Specifically:

1. The Maven POM would need the dependencies for the Neo4j client and its excellent Spring integration.
2. We’d need a Spring configuration class to bootstrap Neo.
3. We’d want a sample domain class and Spring Data repository to illustrate basic use of Neo4j. We don’t want to have to waste time figuring out what code to start with. Ideally, we’d also want some integration tests so that we are able to work in an agile manner.
4. We’d like some material in README.md explaining how to install and start the database, and core Neo4j concepts.

Whereas a traditional tutorial would tell you what to cut and paste or offer a new project to clone, unrelated to your own work, the Atomist approach is executable: At each point, you can click to perform the action being described.

For example, simply hit this link and your original project will be updated to add Neo4j.

The result of running this editor will be a Pull Request with contents like this:

(The actual pull request involves more files than I’ve shown here.) When existing files are edited, the diffs are clean. No loss of formatting; no peculiarities to distinguish robot editor from a human developer.

Git’s change logging and GitHub’s social coding are valuable here. We get a log of the changes that can be reviewed by others, and if we don’t pursue the new technology, we can simply delete the new branch and life on “master” goes on as before.

Implementation

Let’s look at how this editor works.

Our Neo editor, like the original Spring Boot template, is a running project. This makes it easy to keep up to date using your favorite tooling for the target stack.

The normality of the toolchain and lack of magic enables domain experts to contribute their knowledge. In this case, Michael Hunger of Neo Technology kindly wrote us an up-to-the-minute example of Neo usage that can continue to serve as the basis of our editor.

Thus editors and templates can be versioned on GitHub, with the content easily understandable. If you have an improvement that will benefit all users, you can fork an editor repo and send a pull request to the owner. If your improvement benefits only you and your colleagues or contains private content, you can fork it and register a distinct new editor or generator repository: I’ll talk about that in a future post.

Editors provide an executable way to embody and share knowledge, either publicly or within an organization. With the Neo editor, you are effectively pairing on your project with a member of the Neo4j core team.

Some editors don’t need backing artifacts: For example, editors that update files rather than introduce new content. Others, like our Neo editor, need resources to draw on. Its starting point is a normal Spring Boot project using Neo4j. It grafts content from that project into existing target projects. Here’s the Spring/Neo project structure in the root:

Unlike in the case of a project generator, this content won’t automatically be copied into target projects. Editors will copy resources selectively.

The .atomist directory contains our editors, which can access any content in the archive. Its structure looks like this:

It contains:

• A .rug file (AddNeo.rug) that uses our DSL to manipulate target projects to the desired state
• Several test scripts with a .rt extension that test our editors
• Backing projects under /tests/targets that are used by the tests to verify behavior given different starting points. Such projects typically contain the minimum content needed to test the scenario.

Note: for current information on Editors, see our docs — the Rug DSL has been deprecated

Let’s look at the top level editor. It is tagged to indicate the technologies it relates to. Most editors declare parameters that will be gathered from user input, but this editor doesn’t require any. Its implementation first checks that the project is eligible for this transformation — it must be a Spring Boot project built with Maven — before invoking four lower-level editors that perform the transformation:

The heavy lifting is the Java project manipulation performed by the AddJavaSrcFiles editor:

Our DSL aims to make the expression of the steps required as close as possible to how we naturally think about them:

1. Copy the new Java files from the Neo project backing the editor into the project being edited. Because of the eligibility tests, we know that the basic directory structure will be the same.
2. Rename the org.neo4j.example.person package to be a domain sub-package of the package in the target project containing its Spring Boot application class. (We know there is a Spring Boot application class if we get this far.) We change the package name here as the user will add more domain classes and won’t want a distinct package name for each.
3. Add a Neo4j dependency in the project’s POM.

The AddJavaSrcFiles editor uses two Rug types — spring.bootProject and pom — that allow manipulation of Spring Boot projects and Maven POMs respectively. As you’d expect, the spring.bootProject type is a specialization of the Java project type, allowing us to perform generic Java operations as well as ones specific to Spring Boot.

Our DSL uses a model exposing comprehension of project language, framework and structure.

Building this project model means parsing project artifacts where necessary to build ASTs for source artifacts, POM files etc. Regular expressions are great for many tasks, but there are important things that can only be accomplished safely with the deeper understanding that comes from a context-free grammar or equivalent. In the case of Java, we were able to build on JavaParser project. For Elm, we needed to write our own parser, using Scala parser combinators. In other cases, we are using Antlr grammars or user-defined microgrammars.

It’s possible to build custom “types” for many technologies and file types. This enables a consistent approach in all cases.

We love test-driven development, so we provide a BDD-style testing framework to allow editors to be tested before they act on actual code. In the following test scenario, we are making assertions about what packages are present and absent in the finished project, given input from the .atomist/test/targets/simple directory:

Such tests enable us to be confident that editors work before we try them on the local file system or GitHub projects. They also allow us to make changes to non-editor content — in this case, the Spring/Neo project serving as code and configuration donor — with confidence.

Good editors make as few assumptions about project structure as possible; good tests verify that those assumptions still hold. The testing support also allows us to use CI for Atomist generators and editors.

Roadmap

We’re steadily adding more generators and editors. Beyond Spring, Clojure, Elm and Python, we aim to support many more languages and frameworks. We’re in the process of open sourcing our editor support, and are excited about the potential for community contribution.

As infrastructure as code becomes increasingly important, the benefits don’t stop in development. In the words of Martin Fowler:

Having your infrastructure defined as version-controlled code aids with compliance and audit. Every change to your configuration can be logged and isn’t susceptible to faulty record keeping. … All of this increases in importance as you need to handle more servers, making infrastructure as code a necessary capability if you’re moving to a serious adoption of microservices.

Community members have already noted the wide potential for Atomist generators and editors:

Ideas on what editors you could write or wish others would write? The Atomist Community Slack is a great place to hang out and discuss possibilities. I look forward to seeing you there!

[Disclosure: Neo4j is a technology I’m personally excited about, which is why I became a Neo Technology board member.]