Metaprogramming in JavaScript with jscodeshift

Part 1: How to analyse and modify your programs

Kacper Kula
Aug 8, 2019 · 6 min read
Image for post
Image for post

Metaprogramming is a powerful but under-appreciated tool. Even though many widely used development tools use it under the hood, interest in metaprogramming is steadily declining (at least according to Google Trends data):

Image for post
Image for post

There could be many reasons behind the trend. It used to be a fairly popular tool in the fields that nowadays Machine Learning is thriving in. As development tooling gets more and more sophisticated, the need to investigate such topics decreases. Whatever the reason, I strongly believe that this forgotten art is still relevant and can bring a lot of benefits once mastered.

What is metaprogramming?

Wikipedia defines metaprogramming as a “programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running”.

In other words we can use metaprogramming to inspect, modify, or even generate code from scratch. We can also use it to write self-modifying programs — a use-case outside of the scope of jscodeshift, but one I covered briefly in a previous article:

One of the tools we can use for metaprogramming is jscodeshift: a toolkit created by Facebook in 2015 that provides an easy-to-use API for traversing, querying and modifying the abstract syntax tree (AST) of the JavaScript code. It’s built on top of recast and it uses Esprima as a JavaScript parser.

Code which modifies original source code is commonly called a codemod. Plenty of libraries have their codemods publicly available on GitHub — this is usually a good place to start and learn how to write your own.

The following diagram shows the general overview of how jscodeshift works in practice. Firstly, the source code we wish to modify is transformed to an AST, then the codemod is applied to add/remove/modify the nodes. As a result, another tree is created. We can then convert the tree back to code form and save it.

Image for post
Image for post

What is an abstract syntax tree (AST)?

In order to understand the whole process of metaprogramming and write our own codemod, first we have to understand how the JavaScript program is read by the machine.

The JavaScript engine does not read our code the same way we do. Instead it transforms the code to an abstract syntax tree. The AST is a tree-based data structure representing all the variables, calls and flow-control structures used within our code. When the program runs, the JavaScript engine traverses the tree and executes the commands represented by each node. If you want to learn more about the ASTs and discover the different parsers available, head to this brilliant article by Juan Picado.

AST Examples

Let’s start with a simple example:

Image for post
Image for post

and see an AST representation of it:

Image for post
Image for post

We can see that our code has been transformed into simple tree with a BinaryExpression (which represents addition) as a node and two arguments (left-hand and right-hand) as the child nodes.

Now let’s consider some more complex code and its AST representation:

Image for post
Image for post
Image for post
Image for post

In this example the tree grew quite significantly. As you can see our conditional expression has been transformed into the node containing 3 children nodes:

  • test: the condition under which the “consequent” expression will be executed
  • consequent: code that gets executed if the test subtree evaluates to true
  • alternate: code that gets executed otherwise

We can explore the tree further and inspect each of these paths separately. Our x > 3 statement has been converted to BinaryExpression subtree with two arguments — similar to our 2+2 example.

Our console.log(x) expression got expanded into MemberExpression which is an AST representation of taking a property from the object (console.log) and then combining it with the arguments (x) under the CallExpression node.

Image for post
Image for post

If you want to inspect your code’s AST, you can use AST Explorer to generate an AST representation. You can explore the tree in the JSON format on the right. When you hover over a node the corresponding code will get highlighted.

This form of code is really easy for machines to analyze and modify. In fact it’s exactly what jscodeshift uses to modify!

Real life examples

Before we start writing our own code modifications, we should take a look at what codemods can do and where they are useful. Lots of them are documented really well.

Migrating to Jest

The Jest migration guide recommends using jscodeshift to automate the migration process from popular test frameworks like AVA, Chai, Expect.js, Jasmine or Mocha to Jest. The codemods themselves can be found on the jest-codemods GitHub. There are many great stories describing how this codemods helped developers to migrate their projects faster, such as when Kent C. Dodds migrated a codebase from AVA to Jest at PayPal.

React breaking changes

React includes codemods as part of the migration guides between versions that introduce breaking changes. These guides (in theory) allow you to migrate between versions with just a single command. In practice, some manual fixing is usually needed but having a codemod speeds up the process significantly.

Gatsby migration v1 -> v2, aka codemods are hard

On the other hand, codemods are not always that helpful. Recently I tried to migrate an old Gatsby v1 project to the latest version using the official codemods. Unfortunately, even though our codebase was fairly simple, some codemods didn’t work and others generated invalid code. Writing a good codemod is not straightforward given that it needs to run on all possible codebases. In our case migrating Link components broke the app because we had an abstraction on top of Gatsby’s class. After the migration all of the imports were duplicated and we had to fix them manually.

Migrating lodash / underscore to native functions

Another interesting codemod I have found migrates a codebase from lodash and underscore to ES6 equivalents. In the past using lodash and underscore was justified by the fact that most of the methods they provided weren’t part of JavaScript’s functionality. Nowadays most of these methods are included in the language itself. Unfortunately migrating by hand would be time consuming and error-prone. Thanks to codemods we can do it with a single command:

Image for post
Image for post

Other useful codemods and resources

There are many other codemod resources, notably:

  • React codemods — a repository for migrating React projects
  • JS Codemod — codemods for migrating to new JavaScript syntax like arrow functions, let/const instead of vars, Object.assign and more
  • “Awesome” jscodeshift — a repository with plenty of articles, talks, tools and codemods

Stay Tuned

This is article is just an introduction to codemods. Stay tuned for the second part in which we will create three codemods that format code correctly and search for data leaks.

Onfido Tech

Stories from Design, Engineering, Product and Research at…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium