Faster Meteor Reloads

Edit: Checkout out Reval for one way of speeding up certain kinds of Meteor code reloads.

One of the biggest complaints about Meteor is how long it takes the server to restart after modifying code. In a fresh Meteor project it’s pretty speedy, usually <5 seconds. But for large Meteor projects (like ours) reload speeds can get very long, sometimes >20 seconds for a full server reload. If you include the time it takes for the browser itself to reload with the new page we often see reload times of >30 seconds.

The purpose of this document is to show techniques for profiling reloads, explore likely causes of the slowness, and list some potential optimizations. I think that we as community are capable of fixing these issues, and the main reason why we haven’t already is that getting started on reload optimization requires a lot of background knowledge. I hope that this document provides some of that background.

Bandaid Solutions

Before diving into the details of how we might fix reload speeds at the Meteor level, I’d like to go through some techniques that can speedup reloads in existing projects.

1. If you are building in React, and are already structuring your project with ES6 modules, then you can replace the Meteor build tool with Webpack. This gets you much faster rebuilds and HMR (Hot Module Reloads). However, migrating large existing projects to React+ES6 Modules+Webpack is not for the faint of heart.
2. Ensure that client-only code is only loaded on the client and server-only code is only loaded on the server. Meteor can rebuild the client and server bundles independently, so by carefully separating out client/server files, code changes that only effect the client or only effect the server will reload much faster.
3. Make sure that you are using an SSD. The build tool is pretty disk IO bound, so this can make a really big difference. If you are feeling ambitious you could also mount `~/.meteor` and your project’s `.meteor` directory on a RAM disk.
4. Make sure there isn’t anything in your code that takes a long time on startup. This can be a double-whammy for startup code that runs on the client and the server, since it will run both when the server restarts and when the browser restarts.
5. Move as much of your code as possible into many small Meteor packages. Packages rebuild somewhat separately from the main application and other packages, so you can get much faster reloads by breaking up your code. If you are using a large library (e.g. Semantic UI), make sure it is in its own package!
6. Don’t store large files in your source tree. This includes files in your `public/` directory and files added with `api.addAssets`.
7. Remove third-party packages that you aren’t using.
8. Try adding the environment variable `METEOR_OFFLINE_CATALOG=1`. Some projects spend several seconds on every reload looking for package updates, and this disables automatically checking for package updates.

Doing all of these things can really help speed things up a lot, but for large projects they are not enough.

Hacking On Meteor

For the remainder of this article it will be important that you can modify the Meteor build tool. The best way of doing this is running Meteor directly from the git repo:

git clone git://github.com/meteor/meteor.git
cd meteor
./meteor --help

Then you can just use the Meteor binary located in the checkout to run your project rather than the globally defined Meteor binary. Now any modifications you make to files in the checkout will be present when you run your application.

Profiling Reloads

In order to figure out what is making reloads so slow we need to be able to profile the build tool. There are two ways of doing this.

The easier approach is using Meteor’s built-in profiler. All you have to do is set the environment variable `METEOR_PROFILE=1`. So you might run your application like:

cd path/to/application
METEOR_PROFILE=1 path/to/meteor

On every reload your app would then output something like:

It takes some practice to learn how to read the profiler output, but it can be very informative. There are two stages to a server reload (1) `Prepare Project For Build` and (2) `Rebuild App`. The profiler output profiles these two steps separately. For each step, the profiler outputs a tree representing the call stack over time (each node has the amount of time spent in that node or its children and the number of times that node was entered) followed by summary statistics about the most expensive leaves of the tree.

David Greenspan provides a more coherent explanation of the profiler output here.

The second way of profiling the build tool is by using the v8-profiler NPM package. Then you can wrap blocks of code that you wish to profile like so:

var profiler = require('v8-profiler');
profiler.startProfiling('1', true);
// ... slow code ...
var profile = profiler.stopProfiling();
var fs = require('fs');
profile.export(function(error, result) {
  fs.writeFileSync('profile.cpuprofile', result);
  profile.delete();
});

You can then load this dump into the Chrome Developer Tools and analyze it using its CPU Profiler. This gives you a nice flame chart (with symbol names!), and the ability to sort the most expensive function calls. This output is pretty different from the Meteor profiler output because it actually samples the call stack, whereas the Meteor profiler just records how long manually annotated chunks of code take to run.

Potential Problems

Time for the exciting part! Let’s try to figure out what all is making the rebuilds slow. These observations are drawn from David Greenspan’s work on rebuild speeds last January (documented here and here), talks with MDG devs, and the many hours I’ve spent profiling Meteor.

The times included next to each issue are taken from Qualia’s main Meteor application running on a mid-2014 MacBook Pro with a 2.8GHz i7 processor and 16GB of DDR3 RAM. Keep in mind that these improvements aren’t all additive.

Reading Local Package Catalog [~2 seconds]

On a reload, Meteor scans each package’s metadata. It doesn’t actually load any of the package’s files, just the `package.js`. Here is a sample profile I took using the `v8-profiler`(load it into the Chrome CPU Profiler!). It is a profile of `project-context.js:620` and corresponds to `_initializeCatalog` in the normal Meteor profiler. I really don’t understand why this part is slow, it’s not loading all that many files.

File Watching [~3 Seconds]

When Meteor reloads it doesn’t keep track of which files triggered that change (i.e. which files changed). So when the reload first starts it has to figure out which files changed. It does this by loading every file in the source tree into memory (since this isn’t just the user source tree, but all the code used by Meteor this is often 10,000+ files), hashing it, and comparing it to the hash of the file from the last time Meteor restarted. This corresponds to `watch.isUpToDate` in the normal Meteor profiler. This could be avoided by remembering which files triggered a reload, or perhaps by checking a file’s `mtime` rather than computing a hash (that would turn all of the reads into stats). It would also help if Meteor only watched the files that it absolutely needed to (only app files rather than files in all packages that the app depends on). This could vastly decrease the number of files Meteor is watching.

CSS Concatenation/Minification [~2-8 seconds]

Meteor concatenates and minifies all of the CSS files in a project into a single file. For projects with a lot of CSS (e.g. any project including a large CSS framework like Semantic UI), this can take a long time. It seems like this should be unnecessary in development mode, but apparently not doing this can cause issues. Nevertheless, Qualia has been using this PR which disables concatenation/minification without any issue for 6+ months. And it saves us 2–3 seconds on every reload. This issue is further discussed here.

ES6 Compilation [~1.5 seconds]

This corresponds to `plugin ecmascript` in the Meteor profiler. It looks like like the ES6 compiler looks at every Javascript file that ends up in the bundle (based on the number of `files.stat`’s that are being called) and recompiling every Javascript file in the package that is being recompiled (or every file in the main app if it is the main app that changed, not a package). I think that the compiler makes heavy use of caching, so it’s not recompiling everything from scratch every single time, but there are enough files that this step is quite slow (I am not 100% that everything is being cached properly). The ES6 compiler is implemented here. I feel like there must be a way so that it doesn’t look at so many files.

JS Linking [~2 seconds]

This step is rather opaque to me (`PackageSourceBatch#_linkJS` in the Meteor profiler), but I believe that this is the step where all the Javascript files in a package are merged into one file, and a bunch of fancy stuff happens to fake support for ES6 modules. I don’t have many ideas on how to make this faster.

Writing Bundle To Disk [~1.5 seconds]

After the client and server bundles have been compiled they need to be written to disk. This corresponds to `bundler writeSiteArchive` in the Meteor profiler. The majority of the time spent in this section is time spent doing disk IO. Also note that significantly more time is spent writing the server bundle to disk than writing the client bundle to disk. I believe this is simply because the server bundle is much bigger (because e.g. it probably contains many NPM packages). If `meteor npm` has been used, then these times can be significantly higher. Just `meteor npm install — save bcrypt` adds an additional second to the reload time, and as of Meteor 1.4.1 it is strongly recommended that this package is installed. The effect of `meteor npm` can be seen in the Meteor profiler under `meteorNpm.runNpmCommand`. It might be possible to avoid a lot of disk IO if we modified the existing client+server bundles rather than deleting and rewriting them from scratch on every reload.

Server Reloads Regression [~3 seconds for server only changes]

If you change a file that is only available to the client then Meteor doesn’t bother to rebuild the server bundle. There used to be a similar optimization for when a server-only file changed, but this no longer seems to be true.

Potential Solutions

I mentioned potential solutions to many of the problems I listed above, but I’d also like to go over more general solutions that might help.

Parallelize client+server build

Constructing the client and server bundles are independent processes that are currently being done in serial. Seems ripe for parallelization.

Singe file change fast-path

Most reloads are triggered by changing single files. If we make a fast path for these changes that performs minimal modifications to the existing bundles we could save a loootttt of time. For inspiration, modify something in `.meteor/local/build/programs/web.browser/app/app.js` (replace `app` with whatever the name of your project is) and do a hard refresh in your browser (so it reloads the page without using the cache). The change you made shows up instantly! Magic! Imagine if changing a file in your main app code just modified `app.js` and invalidated the browser cache. You could get <1 second reloads…

Keep more things in memory

A lot of time is spent doing disk IO. If everything was more aggressively kept in memory then a lot of this would be unnecessary. For example, we might be able to avoid writing the bundle to disk if we kept the entire thing in memory. However, this is a little tricky since the build tool and the Meteor server are running in separate processes.

Be more selective about reading files

In many cases, files that could not have possibly changed are re-read or re-compiled during a reload. When the Meteor server reloads it should take advantage of the fact that it knows which files have changed (for more information on Meteor’s file watching mechanisms see here and here).

Conclusion

I think there is a lot of room for improvement in the performance of the Meteor build tool. Slow reloads are super-linearly bad; a 2 second reload doesn’t leave time for the developer to get distracted, whereas a 20 second reload guarantees that the developer will get distracted while waiting on a reload.

I think the community is capable of fixing these problems and I hope that this document will provide some guidance on how to get started. I am very happy to say that Meteor 1.4.2 will be entirely dedicated to speeding up reload times, and I am cautiously optimistic about the future.