It’s Just So Meta — Making Frontend Devs’ Lives Easier

As O’Reilly has been growing its learning platform, we’ve been looking for ways to reduce the cost of doing development for our frontend developers. “Meta-modules” and “managed files” are just two ways we are doing so.

One of the things I love about frontend web development is how quickly things change. New libraries come out. New technologies and practices come to the fore. It’s pretty exciting! Of course, that all comes with a cost. As an engineer, you have to keep up to date — which is why a lot of people turn to O’Reilly in the first place! There’s a cost for engineering organizations as well. How do you keep teams informed of changes and keep individual projects, which may reside in dozens of different git repositories, up to date? How do you make updates happen quickly if a security vulnerability is discovered in a commonly used dependency?

There are lots of tools and approaches to help out with these challenges. For instance, GitHub offers security scanning, while some organizations keep related projects in monorepos to simplify maintenance. Frontend frameworks, like Angular and Vue, do this through a CLI they provide to bootstrap apps and roll out updates more easily. When it comes to the Angular CLI, typing ng update @angular/core and having it update a host of related dependencies and perform code migrations as well is pretty cool.

On the React front, which is what we use here at O’Reilly, there is the popular “Create React App” (or CRA) CLI, which helps with getting a project bootstrapped up. However, it’s not uncommon that a project needs to eject its configuration to do something CRA doesn’t support. At that point, the project is on its own for managing dependencies and configuration.

While only a few of our projects have used CRA, nearly all of them face the same problem of this “ejected” configuration. Whether this is in the form of the dependencies detailed in the package.json or in the configuration files needed by Babel, Jest, or some other bit of tooling. To try to address this, we’ve taken some inspiration from Angular’s update approach to automate updates. In our case, the approach consists of two key concepts: “meta-modules” and “managed files”.

What’s a “meta-module”?

That’s just a high-falutin’ way of saying, “a module that provides other modules.” That’s one of the things that CRA does, and all that we’ve done is provide a few modules that are specific to the needs of our apps. We’ve split our meta-modules along categories, so we have one that provides basic tooling every frontend JavaScript project may need, like Webpack, Babel, and Jest. Others are more tailored to specific needs, like a frontend tooling library that provides React and PostCSS.

With meta-modules, a project doesn’t need to worry about managing the versions of individual dependencies. They just install the latest version of the meta-module and they are good to go. Of course, if there are breaking changes, we provide guidance to the teams on how to make the upgrade. However, we have another tool that can help us with that and that’s “managed files”.

What’s a “managed file”?

Again, a term that makes the concept sound more complex than it really is. Essentially, it’s a file that lives in the meta-module and is something like Webpack, Babel, or Jest configuration. This managed file is copied into the consuming project when a developer installs or upgrades the meta-module. The interesting part comes from how we handle updating those files.

Obviously, you wouldn’t want to copy those files in every time the developer installs the module, as they may have made some changes specific to their project. To handle this, we add a simple header — just a blob of text — to the top of every the managed file. An example:

// : — -:
// version: 2
// priority: 0
// source: chassis-core
// : — -:

The meta-module then has a postinstall script that iterates over all the managed files it knows about. It looks at the header of the project’s managed file and compares it to its own. If the meta-module’s version number is higher and its priority is the same or greater, it lets the developer know it would like to copy an updated file in. The developer sees this sort of output:

Although the developer can refuse the update, we hope they won’t! Once they accept the updates, the developer would see the file as changed in source control and could compare this modified file against the last version checked in. This would allow them to restore any project-specific changes they’ve made to the files. And just in case the project doesn’t use source control or the developer had some local modifications, we create a backup copy of that file.

Some of the Benefits

What I hope you took away from the above is that this is a really simple system. There’s no complex logic or concepts here. However, for such a simple approach, we’ve already seen a number of benefits. For instance…

• We don’t have to fight for the developers’ attention. A Slack message or email saying, “Update this package!” will quickly get forgotten by a developer who already has enough to deal with. With this approach, they’ll get the updates simply as a part of managing their project.
• We no longer have to worry about getting configuration “perfect”. If we want to provide some configuration in the meta-module and need to change it in the future, we’ll just version it. Of course, we don’t want to do that too often, since developers have to resolve the files manually.
• Configuration isn’t hidden. Initially, we tried exposing the configuration as an export of the meta-module. The project would do a require or import to bring it in, but that made it tough for developers to know exactly what was going on when they ran Webpack or Jest, tools that typically need the most per project configuration. Now they can see it all, modify it easily, while still being able to get updates from the meta-module.

Now, it’s probably worth mentioning, getting all this to work does require an understanding of some of the inner workings of NPM scripts, but that’ll have to be for another article!

So, that’s the high-level overview of how we’ve tried to simplify this chore for frontend devs. What do you think? Are there other solutions you’ve tried or suggestions on how to improve our approach? Comment below or reach out on Twitter!