How codemods motivated us to tackle some technical debt on our front-end

When you want to introduce new technologies or libraries, as we can see in Dan’s blog post about CSS variables, sometimes it can be quite hard. Dealing with legacy code and refactoring are two tedious tasks that could become easier thanks to codemods.

Codemod is a tool developed by Facebook to help you refactor large-scale codebases. Changing bundler, upgrading libraries or even switching to a new framework can be smoother than before. The Javascript version codemod, JSCodeshift, relies on the library Recast to parse and manipulate the Abstract Syntax Tree (AST) before outputting formatted code.

In this post, I’d like to share with you how our team used codemods to tackle
some technical debt around our build system, what we’ve learnt so far and how we will carry on using them in the future. This post will not cover details of AST-to-AST code transformations and how to write codemods scripts in details, but will focus on a real life application within Geckoboard’s codebase.

A common context

All front-end developers are facing some sort of Javascript fatigue.

I hear similar things from other developers when I go to Meetups or front-end events around London.

“We’re using X but we are rewriting with Y.”

And when applying this rewriting to a large codebase, it usually means running both in parallel or writing some sort of adaptors to plug them together while the rewriting take place in your day to day flow. Or your team is large enough to be split and you’ve got dedicated developers to work on a rewriting on the side for few months.

At Geckoboard, we started to use React and Flux, along with ES6 in parallel
and we have been migrating logic and components over the past two years. While a major part of our app now lives in this new ecosystem, code written four years ago is still run everyday on our TV dashboards and should receive the same level of attention as new code. That’s the reason why we started to investigate codemods after seeing interesting results in the open-source community.

Some libraries like Jest already offer codemods to automatically migrate your tests suite. Upgrading to the latest version of React can be simple thanks to official codemods as well.

History of the web application at Geckoboard

The story of our web application is a classic. It started as a few lines of
Javascript in a Rails application. When extracted, the core of our dashboard
logic has been structured around neat collections and models thanks to Backbone. Remember it was the holy grail, MV * !

In order to split the build and load only some JS per section (dashboard, admin, config), a custom JSON manifest had been setup. Grunt tasks were orchestrating the build by reading a manifest.json like this one:

{
 “dashboard”: [“src/base/models/dashboard”, …],
 “v4”: [“lib/backbone”, “src/v4/main.js”,…]
}

to produce different bundles. Alongside that a global namespace object called GB was used to hydrate some models and collections straight into our Rails templates.

GB.models.dashboard = GB.models.Dashboard({ id: 123 });

This namespace was also used as a module system. A module was registering itself to this global by doing

// src/base/collections/dashboards.js
GB.collections.Dashboards = GB.collections.Collection.extend({
 …
});

This custom module system was later deprecated by RequireJS for new code. Unifying our modules and build system by removing custom Grunt scripts in favour of Webpack is something all the frontend team wanted for a while but this was a tedious task with no chances to win against new features and product requirement. This was until we decided to try using codemods to attempt to automate this refactoring.

Codemods in action

In this section I’ll detail the approach we took to unify our bundler under
Webpack by wrapping modules under Asynchronous Module Definition (AMD) and getting rid of our global namespace GB.

Create a separate repository

Inspired by this excellent open source repository js-codemods we decided to create our own internal repo called js-tooling that will host codemods.

Having a separate repository is good for a few reasons.

• Testing
  It’s important to have a decent environment to write and test your codemods. The unit testing utility based on Jest provided by jscodeshift toolkit is amazing and was a great incentive for us to invest time in codemods.
• Sharing / reviewing
  It’s mandatory to be able to track, share helpers across codemods, allow peer-review and link them to commits on your main app.
• Coding style
  To make sure to keep a consistent style.

Moving to AMD

Our first task was to wrap all non AMD module like this:

Input

// src/base/collections/dashboards.js
GB.collections.Dashboards = GB.collections.Collection.extend({
 model: GB.models.Dashboard,
 …
});

Output

// src/base/collections/dashboards.js
define([
 ‘src/base/collections/collection’,
 ‘src/base/models/dashboard’
], function(Collection, Dashboard) {
 return Collection.extend({
   model: Dashboard,
   // …
 });
});

After expressing the code transformation requirements by creating an input and an output, you can use them as your unit test case. The following approach was a result of iterating in a Test Driven (TDD) manner.

The core of this codemod is to collect all usage of a member expression like GB.xxx that are not assignment. (Assignment will be handled differently). MemberExpression is a type of expression that represent accessing a property of an object like foo.bar where foo and bar are 2 identifiers.

Helper function to collect all our GB namespaced expressions

refStore will keep path of each usage to later replace them with correct identifier. In our case Collection and Dashboard.

store is a Map that holds a Set for each type models and collections.

In our example we’ll have this in our store:

store = {
 ‘models’: [‘Dashboard’],
 ‘collections’: [‘Collection’],
};

Later we’ll map them as our modules requirements in a new expression statement by creating an array of literals containing files’ paths
and another array of identifiers containing names.

Now we’ve got everything needed to create our new AMD expression statement

define([], function () {});

and we can replace our global namespace assignment, that we filtered out earlier, with this expression and the contained body.

Take away

In this post I expose only one codemod from this refactoring. We had to split our refactoring into small chunks and more codemods as we wanted to clean other global dependencies for libraries and some adaptors we have to bridge to the new part of the app. While this refactoring ended up working really well without any regressions, our journey wasn’t that easy and we learned some lessons that I’d like to put down as take aways.

First do create your own codemods repo! It’s a really good place to start, play with the AST-parser and having a quick feedback thanks to the test utility. Also don’t miss this visual explorer. As we said earlier, it’s good to keep your work under versioning to share and reuse.

Never update code manually which has already been updated by a codemod in the same commit! One codemod applied = one commit. Never try to manually fix small edge cases not handled by your codemod. This makes it hard to rebase, revert or simply change syntax, or a variable name. If the edge case is easy to fix automatically, simply add the edge case to your input test file and update the codemod to handle it. Sometimes that could be a real exception where it’s useless to spend time adapting your codemod and you should handle it manually in a separate commit.

You should treat codemods like a database migration. Database migrations are usually stored in separate files prefixed with timestamps. You don’t have to use timestamp but you should use a unique file name for your codemods. You can therefore link your codemod to a commit by prefixing the commit message with its unique name.

[codemods — gb-global-namespace] applied

Link codemods to a pull request and ask for a review. Even if it’s a one shot script, you should ask for a review. This will force your team to get familiar with codemods and help writing future codemods.

Unfortunately, I didn’t find a hosted version of jscodeshift documentation. I highly recommend to checkout a local version of the repo to browse the generated HTML documentation.

Conclusion and future

Codemods are incredible and allow you to discover or re-discover the core of
your language and how JS AST is parsed. Jumping into the world of AST before having these open-source project and tools available was quite complex and would stop most of us trying to learn to use it. Today we can add AST transformation thanks to codemods in our toolbox for day-to-day task at Geckoboard. We are already planing to use to migrate a test-suite written in Jasmine framework V1 to V2.

P.S Geckoboard is hiring! Check out our open positions.