Codemod idea to make your codebase evergreen (part 2/2 — practice)

Real code samples and practical explanations on how to keep your code evergreen: the full end-to-end process from examing your code to get AST of it to writing tests to validate the correctness of your modifications.

Small recap of the previous part: jscodeshift is a convenient wrapper on codemod idea, which helps to keep our codebase evergreen.

I will take as an exercise for our practical part a real case from ING.

ING uses Web Components. By now, Web Components have quite a history: specifications changed, libraries around them changed, the ecosystem around them changed, and their support by browsers changed. Or course, all of this requires some changes to the codebase.

ING uses Web Components for a long time. It started back in a day with Polymer version 1. When I joined ING, it had already finished migrating to Polymer 2.

The Polymer 2 Element looks like this:

But the time goes by.

After Polymer 2, there was Polymer 3, which looks like this:

The main differences between Polymer 2 and Polymer 3:

• Easier to control what goes to Shadow DOM and what’s not.
• Lifecycle events are standards-based.
• ES6 Modules instead of HTML Imports and NPM instead of Bower.

The last one is quite crucial. I will go into detail about why a bit later.

Polymer 3, however, according to the official website, is also outdated, and the offered replacement is Lit.

Lit element:

The main differences between Polymer 3 and Lit

• Properties definitions
• Styles definition
• Internals of libraries like size, features, and how it does its job

Therefore, to keep our code evergreen, we need to migrate from Polymer 2 to Lit.

I promised to tell more about why migration from HTML Imports to ES Modules in Polymer 3 is significant.

jscodeshift works with JavaScript and not with HTML. It cannot parse HTML to migrate it. Thus we need something else to even start working with jscodeshift.

Jobs that go beyond jscodeshift are marked with an asterisk (*) so they can be skipped to focus on jscodeshift rather than migration from Polymer 2.

HTML Imports to ES Modules (*)

Thanks, Polymer team, for Polymer modulizer. It automates the process of migration from Polymer 2 to Polymer 3.
I don’t know how it works internally, but one of its dependencies is jscodeshift(wink). After the migration, we can start working on modernizing our codebase.

Simplest Polymer 3 Element to Lit

This is the most interesting part where I will show the full power of jscodeshift.

To understand what steps we need to perform the complete transformation, we can compare a Polymer element and a Lit element line by line. We can break down the migration plan into the following steps:

1. Migrate the imports (first line).
2. Migrate the class definitions (fourth line).
3. Migrate the properties (not present in the example but it would be the fifth line).
4. Migrate the template (line 7-8) (*).

1. Migrate the imports

This:

To this:

Understand the workflow

We learned the process in Part 1 of this series and even created our first codemod. The common process is:

1. Parse the code with thejscodeshift helper api.jscodeshift(source).
2. Find parts of AST that we want to change with .find utility.
3. Update these parts or return new ones with .forEach or .replaceWith utilities.
4. Convert AST back to source code to enjoy the results with .toSource method.

Let’s dive one level deeper into what happens on each step from the jscodeshift

1. api.jscodeshift(source) returns an instance of Collection. It allows having an iteration over the paths or the nodes of the AST of the source code. According to core concepts, Node is a pure representation of AST Node when NodePath has additional meta-information, including a reference to a parent AST Node.
2. jscodeshift is highly extensible. You can register your own methods and utilities there. This extensibility is used by jscodeshift as well. That’s why Collection does not contain a .find utility — it’s inside Node type.
   .find creates a new Collection from the Nodes that match the provided filter.
3. .forEach executes a given function for each NodePath in Collection without changing it.
   .replaceWith modifies the paths without the possibility to exclude any path
   .map modifies the paths with a possibility to exclude a path if null is returned.
4. .toSource is a method of a Collection that goes to the very top of the tree and then converts the AST back to the source code with the help of recast.print.

At that moment, we know the utilities from jscodeshift that help us locate the Nodes in the code we want to convert and utilities that help with our machinations.

However, apart from knowing what utilities to use, we also need to know how to use them.

Since we are working with Abstract Syntax Trees, for both location and changing tasks, we need to learn AST of the code first. For this, we can use https://astexplorer.net/. This tool conveniently shows the AST of a given JavaScript.

To understand how to locate the Node with astexplorer.net, let’s examine the following code that is used for import:

import {PolymerElement} from '@polymer/polymer/polymer-element.js'

The AST of it, according to astexplorer.net (with a chosen parser acorn):

{
  "type": "Program",
  "start": 0,
  "end": 67,
  "body": [
    {
      "type": "ImportDeclaration",
      "start": 0,
      "end": 67,
      "specifiers": [
        {
          "type": "ImportSpecifier",
          "start": 8,
          "end": 22,
          "imported": {
            "type": "Identifier",
            "start": 8,
            "end": 22,
            "name": "PolymerElement"
          },
          "local": {
            "type": "Identifier",
            "start": 8,
            "end": 22,
            "name": "PolymerElement"
          }
        }
      ],
      "source": {
        "type": "Literal",
        "start": 29,
        "end": 66,
        "value": "@polymer/polymer/polymer-element.js",
        "raw": "'@polymer/polymer/polymer-element.js'"
      }
    }
  ],
  "sourceType": "module"
}

I marked with bold the most important parts: they distinguish this particular import from others which we don’t want to touch.
Like the following:

// local PolymerElement that we don't want to rename
import { PolymerElement } from './my-polymer-element.js'

There are other edge cases like import '@polymer/polymer/polymer-element.js', but I skip changing them in this article for the sake of simplicity.

The first argument of .find is a type of Node (first bold text in the AST above), and the second argument is a filter to specify the criteria of a Node you want to find. Here we use a path of the import:

.find(jscodeshift.ImportDeclaration, {
  source: {
    type: 'Literal',
    value: '@polymer/polymer/polymer-element.js'
  }
})

Now, let’s focus on how to make a new import instead of the previous one.

We already know a method of jscodeshift that allows us to replace the found nodes:

.replaceWith((nodePath) => {
})

Here is a code we want to construct:

import {LitElement} from 'lit'

An AST of it, according to AST Explorer:

{
  "type": "Program",
  "start": 0,
  "end": 30,
  "body": [
    {
      "type": "ImportDeclaration",
      "start": 0,
      "end": 30,
      "specifiers": [
        {
          "type": "ImportSpecifier",
          "start": 8,
          "end": 18,
          "imported": {
            "type": "Identifier",
            "start": 8,
            "end": 18,
            "name": "LitElement"
          },
          "local": {
            "type": "Identifier",
            "start": 8,
            "end": 18,
            "name": "LitElement"
          }
        }
      ],
      "source": {
        "type": "Literal",
        "start": 25,
        "end": 30,
        "value": "lit",
        "raw": "'lit'"
      }
    }
  ],
  "sourceType": "module"
}

Again, the essential parts are bold. We need to construct ImportDeclaration and to pass there the specifiers and the source. But what are these things?

ImportDeclaration is a part of ECMAScript language specification. At https://github.com/benjamn/ast-types/blob/master/def/es6.ts#L235 we can find what import declaration is and what it consists of. We need this information to be able to construct it.

An import declaration consists of the following fields: Array specifiers, Literal source, and Enum importKind

jscodeshift proxies builders of AST-types from recast: builders have corresponding names as the types they construct except the small letter in the beginning: ImportDeclaration becomes importDeclaration:

jscodeshift.importDeclaration()

Each item of thefields becomes an argument:

jscodeshift.importDeclaration(
  Array specifiers,
  Literal source,
  Enum importKind
)

Each specifier can be either ImportSpecifier, ImportNamespaceSpecifier, or ImportDefaultSpecifier .

We can check the result AST Explorer for what we need body[0].specifiers[0].type and see there ImportSpecifier

AST Types, ES6, line 220 tells us that an ImportSpecifier has one field Identifier

In its turn, anIdentifier definition can be found on AST Types, Core, line 323. It has only two fields: String name and Boolean optional.

jscodeshift can build Identifier like this:

jscodeshift.identifier(String name)

in our example, we need:

jscodeshift.identifier('LitElement')

then an array of ImportSpecifier ‘s

[jscodeshift.importSpecifier(jscodeshift.identifier('LitElement'))],

following the same process, Literal for a path can be built like this:

jscodeshift.literal('lit')

an to build an ImportDeclaration for our case, we do:

jscodeshift.importDeclaration(
  [
    jscodeshift.importSpecifier(
      jscodeshift.identifier('LitElement')
    )
  ],
  jscodeshift.literal('lit')
)

The resulting script

2. Migrate the class definition

At the moment, our file looks like this:

LitElement is imported but never used. To fix this, we need to rename the PolymerElement to LitElement.

The migration track is the same:

1. Inspect the Polymer code in AST Explorer to learn how to locate the piece of code we want to rewrite.
2. Use .find to locate it.
3. Inspect the Lit code in AST Explorer to learn the code we need as a result.
4. Use builders from jscodeshift to build it.
5. Use .replaceWith to replace the located Nodes with the new ones

And at the end, our transform looks like this:

3. Migrate the properties

There are quite some differences in the properties definition:

1. No shortcut for a type in Lit.
2. No default value in Lit properties: should be set in the element constructor.
3. No read-only properties in Lit.
4. No observers and computed values in Lit.
5. Attribute names of the property are different in Polymer and Lit.
6. Etc.

As a beginning, we can focus on the value shortcut.

For us, that means we need to change:

// Polymer 
static get properties() {
  return {
    name: String
  }
}

To this:

// Lit
static get properties() {
  return {
    name: { type: String }
  }
}

As usually, our workflow is the following:

1. Learn the AST of the given pierce to be able to locate it
2. Learn the AST of the target piece to construct it
3. Combine two things above in the code and happily run it

This one is quite big:

{
  "type": "MethodDefinition",
  "static": true,
  "computed": false,
  "key": {
    "type": "Identifier",
    "name": "properties"
  },
  "kind": "get",
  "value": {
    "type": "FunctionExpression",
    "id": null,
    "expression": false,
    "generator": false,
    "async": false,
    "params": [],
    "body": {
      "type": "BlockStatement",
      "body": [
        {
          "type": "ReturnStatement",
          "argument": {
            "type": "ObjectExpression",
            "properties": [
              {
                "type": "Property",
                "method": false,
                "shorthand": false,
                "computed": false,
                "key": {
                  "type": "Identifier",
                  "name": "name"
                },
                "value": {
                  "type": "Identifier",
                  "name": "String"
                },
                "kind": "init"
              }
            ]
          }
        }
      ]
    }
  }
},

Finding this Node is a bit trickier. We want to change an object but there are no unique qualities to it that can help us to locate it. The only difference of this object is that it’s returned inside a static get properties getter.

Fortunately, it’s covered in jscodeshift. For finding this object we can use another built-in method — .filter. The idea is to find all objects but after check whether an object is returned inside properties getter.

Locate the method static get properties with the help of filter

To build Lit properties definitions, we need to go over the properties definitions and check whether the value is an object. If not — this is a shortcut which we need to replace:

The construction of new properties definition is a bit more complex this time. We need property and objectExpression.

The resulting code is:

4. Migrate the template (*)

One of the most challenging parts of all of this is to rewrite the template. The scope of it is outside JavaScript codemod and more like an interesting exercise.

The simplest case of template migration is exceptionally straightforward:

And the Lit Element template is this:

So the only thing we need to do is to change the static get template to render:

However, the juicy part is inside the template itself.

Let’s take the real-world example of the template:

The main points of attention are the following:

1. on-click="viewPluginDetails" — Polymer is quite forgiving. It knows that the handler of a click event is a function. So it treats it as a method of the current element instead of plain string. Small reminder: special tokens should be wrapped into [[]] or {{}} so the official way to write it is on-click=[[viewPluginDetails()]]
2. <template is="dom-if" if="[[]]"> is used to show conditional data in the template. The tricky part of the template element is that it’s not rendered in the DOM until it’s needed. The double-tricky part is that Polymer does not even parse the content of the template element until it’s needed.
3. <template is="dom-repeat" items="[[]]"> is used to iterate over the list of items.
4. class="plugin-type-[[item.type]]" — bindings can be part of other strings to form values of attributes and properties.
5. on-click="[[selectPlugin(item)]]" handlers of the events can have arguments.

Why all of the above is important? Because in Lit, that’s all different:

Apart from all of these parts, there is another crucial difference: Lit Element uses Tagged Template Literals with full power. This leaves all the heavy-lifting of parsing a template to the browser. Polymer uses its own syntax and parses templates on its own.

Unfortunately, this means that we need to reimplement the entire parsing mechanism of Polymer, which is a big part of it. This is the end of our template migration journey, right?

Well, at least unless we use something that parses the template in exact same way as Polymer. Like, hmm, Polymer!

The problem here is that Polymer works in a browser while the code migration happens in the file system.

So the challenge is whether we can make Polymer work for us in the NodeJS environment, use it to parse the template, and use the resulting internal structures of Polymer to our advantage.

Long (and cumbersome, and bumpy, and tiring) story short — it’s possible.

The internal structures of Polymer contain all the information about each binding from the element. For example, if it’s a function, its arguments, the attribute or property it connects to, and so on.

Along with the migration of the template, we can change the elements themselves. For example, a paper-button depends on Polymer. So if we want to replace the Polymer library instead of adding Lit on top, we need to replace the elements as well.

ING made an open-source library called lion-web. The beauty of it, besides thorough-tested good-quality components, is that it is white-label. Meaning you can make your styles to preserve the look and feel of your project. Therefore, we use lion for components replacement.

This task contains two fundamental groups. An example of the first group is a replacement paper-button with lion-button. This is a direct replacement. The behavior of the elements is similar, and the only thing we need to care about is the corresponding names of attributes from paper-button to lion-button. This is an easy group. An example of the second group is iron-ajax. The whole idea of sending AJAX requests was changed, and thus there is no direct replacement of the element. This is a complex group. It’s actually so complex that we can call it impossible and focus on the easy one.

To migrate the components with direct replacements we can define the map of the source elements and their targets:

And the code which uses this map:

Tests

Tests not only validate your correctness of migration code. They also show potential users how exactly the code changes.

jscodeshift has some handy helpers here as well. The section in the README about unit testing explains it in detail.

The first helper defineTest seems what we need. We can cover with test the step-1__imports. Other steps are no different from those.

To define the test, we need to create a folder __tests__ in the folder with our codemod and put there a file step-1__imports.test.js:

This automatically loads the transform, applies it and compares the input and the output.

To provide the input and the output, we need to create a folder __testfixtures__ with the files step-1__imports.input.js and step-1__imports.output.js that contain an expected outcome of the transform:

That’s it. Now we can run the tests with jest and get the output:

In the actual code, the complexity of the tests is way higher. One transform can do many different things: change imports, rename classes, create or delete files. Moreover, some of the code runs asynchronously. That’s why sometimes using test helpers like defineTest or defineInlineTest is not enough. Additionally, these helpers do not support .only or .skip methods on the tests.

To have asynchronous tests or select what tests to run/skip, we can run another helper from jscodeshift. It’s a bit lower level (meaning we need to do some things ourselves) but also gives more flexibility. This helper is applyTransform:

And the output of it:

Conclusion

At ING, I made a list of codemods that migrated the entire real-world Polymer 2 to Lit Element of 15 components (50 files). It automated 65–75% of migration. The rest of the cases are either not implemented yet or cannot be implemented at all. Also, there are rare cases that happen once in 1-2 components (out of our 15) and have a cost-to-benefit ratio greater than one. In other words, 5–10 min of manually rewriting such code is still faster than making a transform for it if it requires a few days of effort. Of course, this varies from the percentage of occurrences, the total number of components, and the effort to make a codemod.

I want to thank my team “Ewoks” and my department “Fraud prevention and investigation” for helping me to make this happen. I also want to thank Thijs Louisse for contacting me after the first part and sharing many ideas. There was no section about tests without his input.

I hope this article can help you to extend your toolbox with a new approach. Migrating the codebase to have it evergreen is not an easy task. Even in the examples above, we migrated the simplest possible element omitting many edge cases, and it’s already hundreds of lines of code. However, if you have a large codebase, the effort of making a codemod can be worth it.

Happy coding and wish you smooth migrations and evergreen code!