Making the Leap to Require.js at Conductor

Much has been written about the benefits of Asynchronous Module Definition (AMD) script loaders like Require.js for large-scale JavaScript applications. There’s the boost to testability and maintainability, the clear dependency graphs, the deliverance from script tag sorting hell. As with any useful technology it has inspired a holy war or two, but it’s hard to argue with the value of a technology that makes code module organization practical in JavaScript, in the browser, right now. Other, better systems may come along in the future, but until they arrive (and until all of your customers upgrade to browsers that support them) AMD is a great stepping stone.

There’s just one problem: getting to that first stone can be quite a leap. It’s easy to start a new application from scratch with Require.js or another AMD loader (or any other module system, for that matter), but upgrading a large, existing code base is a different thing entirely. A module system is almost by definition a cross-cutting concern: it touches every JavaScript file, and then some. How can a team managing a large and fast-moving application (like Conductor Searchlight) upgrade every file without descending into branch-and-merge madness or stop-the-world paralysis? We have a business to run here, so upgrades like this have to be smooth and must be able to proceed in parallel with other projects. Is it even possible? We think so. What follows is the first chapter in Conductor’s ongoing transition to Require.js. Future articles will tell the rest of the story, after we know how it ends.

Motivation

Conductor Searchlight began life like many applications of its generation, with Spring MVC and predominantly server-side JSP page rendering. There were dynamic tables and a handful of JavaScript UI components, but fundamentally everything interesting happened on the server. That has since changed, and as Searchlight’s JavaScript-enabled functionality has grown, so has the need for discipline in its organization, development, and testing. We adopted Backbone, giving us a way to organize functionality and keep separable concerns separate, but we were still left with an increasingly unwieldy collection of script tags and the testing and management headaches that come with them. Whenever you can’t delete a JavaScript file because you don’t know if something is using it, it’s time to up your game. We turned to Require.js.

Strategy

Searchlight’s JavaScript wasn’t formally modularized, but it wasn’t chaos either: all meaningful code was (and still is) organized under a hierarchical namespace — a global JavaScript object tree — that allowed the code in any given file to reference its dependencies in a mostly predictable and consistent way. Each file defines new methods, classes, or data structures and hangs them on this growing tree for subsequent files to reference. The file load order is important but in no way dictated by the files themselves; the script tags just need to be in the right order. It’s a module system, but one where the dependency graph is implicit, hard to see, and easy to break. It’s far better than having no module system at all, though, and it forms the foundation for the transition to Require.js.

That transition proceeds in three phases:

1. Gradually convert JavaScript modules to AMD, without actually using Require.js in the application. In lieu of an actual AMD loader, a custom adapter allows both converted and unconverted modules to coexist and depend on each other, as long as they are still loaded in the correct order via script tags. Meanwhile, unit tests of converted modules use the real Require.js to load their code under test and other dependencies.
2. Once all modules are converted, drop the adapter and cut over to real Require.js.
3. Drop all the individual script tags and let Require.js (or rather r.js, since we will want concatenation and minification) do the work of loading.

The lengthiest phase by far is the first. It entails making structural modifications to every JavaScript file, and our policy is to make the transition opportunistically: a file is converted if and when it wouldn’t add undue merge complication to other work in progress. During this phase, both AMD and traditional, non-modularized JavaScript files exist in the system simultaneously, so using a real AMD loader is awkward. Instead, we use an adapter that implements define() and require() in a simpler way, without dynamic file loading:

// requirejs_conductor_adapter.js (function() { "use strict"; // a map of AMD module paths to corresponding global object paths var modules = { "jquery": "$", "underscore": "_", "backbone": "Backbone", "Conductor/Searchlight": "CONDUCTOR.Searchlight", "Conductor/WidgetTypes": "CONDUCTOR.WidgetTypes", "Conductor/graphs/distributionGraph": "CONDUCTOR.Domain.Distribution", "Conductor/views/WidgetView": "CONDUCTOR.Views.WidgetView", // ... and many more elided here }; function defineModule(name, module) { var pathString = modules[name]; if (!pathString) { throw new Error('Could not define module "' + name + '": no mapping.'); } var path = pathString.split('.'); var parent = window; for (var i = 0; i < path.length; i++) { var pathSeg = path[i]; if (i+1 == path.length) { if (parent[pathSeg]) { console.warn('Overwriting existing definition for %s at %s.', name, pathString); } parent[pathSeg] = module; return; } if (!parent[pathSeg]) { parent[pathSeg] = {}; } parent = parent[pathSeg]; } } function retrieveModule(name, definingModule) { var pathString = modules[name]; if (!pathString) { throw new Error('Could not retrieve module "' + name + '": no mapping.') } var path = pathString.split('.'); var context = window; var pathComponent; while (pathComponent = path.shift()) { context = context[pathComponent]; if (!context) { throw new Error('No definition for module "' + name + '" at ' + pathString + ' while defining ' + definingModule + '.'); } } return context; } // Implement define() and require(), just like a real AMD loader if (typeof window.define != "function") { window.define = function(name, deps, factory) { var resolvedDeps = _.map(deps, function(module) { return retrieveModule(module, name); }); var module = factory.apply(null, resolvedDeps); defineModule(name, module); }; window.require = function(deps, consumer) { var resolvedDeps = _.map(deps, retrieveModule); consumer.apply(null, resolvedDeps); }; } })();

More advanced versions of this adapter have evolved to support Handlebars template loading and other tricks, but this is the original core. Its function is simple: Be an AMD loader from the point of view of AMD modules while keeping everything available in a hierarchical namespace for those modules that haven’t yet been converted to AMD. For every AMD module there is a corresponding object somewhere under the global “CONDUCTOR” namespace, and the adapter includes a map (called “modules” above) from AMD module paths to the global object path where the module is defined. See, for example, the Backbone view called “WidgetView” mentioned in the code above. That module may be provided by a fully migrated AMD module located at “Conductor/views/WidgetView”, or it may be a legacy file that defines “WidgetView” within the “Views” object under the shared “CONDUCTOR” global. Other modules can depend on WidgetView without knowing or caring where it came from, and they can themselves be either legacy or migrated. The adapter maps the two paradigms together and lets them exist simultaneously.

Testing

Meanwhile, as the bulk of the code gradually migrates to AMD syntax supported by a fake AMD-esque loader, the tests are moving to the real thing. As each code module makes the jump to AMD, so does its corresponding unit test suite. The difference is that the tests load themselves — and their code under test — using the actual Require.js loader, not the adapter. Using the real loader keeps us honest, and provides a continuing impetus to migrate files quickly: It’s much easier to test things that are require() ‘able, so the incentive to keep moving towards the end goal is strong.

Complications

The adapter has been working well so far, but it pays to remember that it is fundamentally smoke and mirrors. The biggest gotcha is that files must still be included on the page in the correct order. The adapter doesn’t actually load anything itself — it can’t. Its purpose is to map between an asynchronous module system and a synchronous one, and there is just no way to synchronously satisfy the requirements of legacy files by asynchronously loading migrated AMD files. To maintain its contract, the define() function must resolve its entire list of dependencies, invoke the factory function, and map the resulting module definition back onto the global namespace, all in the same turn of the event loop, so there’s no time for ad-hoc loading.

The constraints on file ordering are actually far tighter when using the AMD adapter than with the legacy system. A properly-written AMD module wants its dependencies to be fully satisfied before it defines itself. With the legacy system we could be more cavalier about ordering, since as long as a dependency is loaded by the time it is used (generally after document ready), it doesn’t matter which file loads first. This new constraint may appear annoying, but it has served to illuminate several infelicities in our existing code and helped us move toward a cleaner and more natural factoring.

Conclusion

Conductor’s transition to Require.js is a work in progress, but so far it has been progressing smoothly. By implementing our own AMD-like loader that knows how to accommodate our legacy system we’ve been able to make the leap a gradual process of succession, rather than a stressful and risky all-or-nothing port. What’s not to like?

Check back later for the end of the story, where we’ll cover the final cut over process and share our thoughts in retrospect.