Metaprogramming in JavaScript with jscodeshift

Part 1: How to analyse and modify your programs

Metaprogramming is a powerful but under-appreciated tool. Even though many widely used development tools use it under the hood, interest in metaprogramming is steadily declining (at least according to Google Trends data):

Google Trends data shows that metaprogramming is not in its prime. July 2019. Source

There could be many reasons behind the trend. It used to be a fairly popular tool in the fields that nowadays Machine Learning is thriving in. As development tooling gets more and more sophisticated, the need to investigate such topics decreases. Whatever the reason, I strongly believe that this forgotten art is still relevant and can bring a lot of benefits once mastered.

What is metaprogramming?

Wikipedia defines metaprogramming as a “programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running”.

In other words we can use metaprogramming to inspect, modify, or even generate code from scratch. We can also use it to write self-modifying programs — a use-case outside of the scope of jscodeshift, but one I covered briefly in a previous article:

Python-like list/array operator in JavaScript

One of the tools we can use for metaprogramming is jscodeshift: a toolkit created by Facebook in 2015 that provides an easy-to-use API for traversing, querying and modifying the abstract syntax tree (AST) of the JavaScript code. It’s built on top of recast and it uses Esprima as a JavaScript parser.

Code which modifies original source code is commonly called a codemod. Plenty of libraries have their codemods publicly available on GitHub — this is usually a good place to start and learn how to write your own.

The following diagram shows the general overview of how jscodeshift works in practice. Firstly, the source code we wish to modify is transformed to an AST, then the codemod is applied to add/remove/modify the nodes. As a result, another tree is created. We can then convert the tree back to code form and save it.

What is an abstract syntax tree (AST)?

In order to understand the whole process of metaprogramming and write our own codemod, first we have to understand how the JavaScript program is read by the machine.

The JavaScript engine does not read our code the same way we do. Instead it transforms the code to an abstract syntax tree. The AST is a tree-based data structure representing all the variables, calls and flow-control structures used within our code. When the program runs, the JavaScript engine traverses the tree and executes the commands represented by each node. If you want to learn more about the ASTs and discover the different parsers available, head to this brilliant article by Juan Picado.

AST Examples

Let’s start with a simple example:

Simple as 2+2

and see an AST representation of it:

We can see that our code has been transformed into simple tree with a BinaryExpression (which represents addition) as a node and two arguments (left-hand and right-hand) as the child nodes.

Now let’s consider some more complex code and its AST representation:

One declaration, one conditional statement and two console.logs

AST of the short code example. It gets complicated quite fast.

In this example the tree grew quite significantly. As you can see our conditional expression has been transformed into the node containing 3 children nodes:

• test: the condition under which the “consequent” expression will be executed
• consequent: code that gets executed if the test subtree evaluates to true
• alternate: code that gets executed otherwise

We can explore the tree further and inspect each of these paths separately. Our x > 3 statement has been converted to BinaryExpression subtree with two arguments — similar to our 2+2 example.

Our console.log(x) expression got expanded into MemberExpression which is an AST representation of taking a property from the object (console.log) and then combining it with the arguments (x) under the CallExpression node.

Our example code (on the left) and the AST (on the right)

If you want to inspect your code’s AST, you can use AST Explorer to generate an AST representation. You can explore the tree in the JSON format on the right. When you hover over a node the corresponding code will get highlighted.

This form of code is really easy for machines to analyze and modify. In fact it’s exactly what jscodeshift uses to modify!

Real life examples

Before we start writing our own code modifications, we should take a look at what codemods can do and where they are useful. Lots of them are documented really well.

Migrating to Jest

The Jest migration guide recommends using jscodeshift to automate the migration process from popular test frameworks like AVA, Chai, Expect.js, Jasmine or Mocha to Jest. The codemods themselves can be found on the jest-codemods GitHub. There are many great stories describing how this codemods helped developers to migrate their projects faster, such as when Kent C. Dodds migrated a codebase from AVA to Jest at PayPal.

React breaking changes

React includes codemods as part of the migration guides between versions that introduce breaking changes. These guides (in theory) allow you to migrate between versions with just a single command. In practice, some manual fixing is usually needed but having a codemod speeds up the process significantly.

Gatsby migration v1 -> v2, aka codemods are hard

On the other hand, codemods are not always that helpful. Recently I tried to migrate an old Gatsby v1 project to the latest version using the official codemods. Unfortunately, even though our codebase was fairly simple, some codemods didn’t work and others generated invalid code. Writing a good codemod is not straightforward given that it needs to run on all possible codebases. In our case migrating Link components broke the app because we had an abstraction on top of Gatsby’s class. After the migration all of the imports were duplicated and we had to fix them manually.

Migrating lodash / underscore to native functions

Another interesting codemod I have found migrates a codebase from lodash and underscore to ES6 equivalents. In the past using lodash and underscore was justified by the fact that most of the methods they provided weren’t part of JavaScript’s functionality. Nowadays most of these methods are included in the language itself. Unfortunately migrating by hand would be time consuming and error-prone. Thanks to codemods we can do it with a single command:

Other useful codemods and resources

There are many other codemod resources, notably:

• React codemods — a repository for migrating React projects
• JS Codemod — codemods for migrating to new JavaScript syntax like arrow functions, let/const instead of vars, Object.assign and more
• “Awesome” jscodeshift — a repository with plenty of articles, talks, tools and codemods

Stay Tuned

This is article is just an introduction to codemods. Stay tuned for the second part in which we will create three codemods that format code correctly and search for data leaks.