Introducing Packem: a super fast experimental bundler written in Rust

Bukhari Muhammad
May 15 · 7 min read

Packem is an experimental precompiled JavaScript module bundler primarily implemented in Rust. It can also handle a variety of other file types like YAML/TOML, fragment shader files and a lot more. Checkout the website or the GitHub page to quickly get started.

Packem’s logo. Always soothes me.

Packem resolves a module’s dependencies and rehydrates them into a module graph, a flat list containing module interfaces which are essentially references to in-memory heap-based mutable data structures containing special metadata of a module in the module graph.

Most of the business logic is abstracted into Rust using FFI bindings to enable low level interactions between both ends. The Rusty binaries are available as precompiled Node C/C++ addons in Packem’s repo. A cloud-based CI is used to run a few scripts with pre-gyp installations, yielding OS-specific binaries with support for later Node versions (8, 9, 10).

This layer of Packem’s core is what is referred to as the Logical Context (LC). All the other operations that are not explicitly prioritized are regressed into Node’s general runtime, which in Packem’s terms is the Runtime Context (RC). Read more on contexts here.

Theoretically, the module graph is kept flat to avoid common pitfalls that would lead to unnecessary traversals if a tree was used in place. This allows the RC to keep track of cases such as deep circular dependencies or heavily nested dynamic imports (code splitting), amongst others, appropriately with minimum performance implications or side effects as possible.

An overview of the bundling cycle from contexts.

More details can be found at Packem’s README.md.

I’ve been having this idea in mind but never planned to execute it until I joined forces with Saddam M. It has really been in my interest to see module bundling as a concept safe for anyone to learn, understand and implement. Having people struggle with configurations, documentation and plugins was extremely horrendous and I’d like to take the chance to change that. With you. With Packem.

Quick history

I took some time to exhaust most of the bundlers written in a non-JavaScript environment. I found out that most of them forgot that they’re supposed to be a bundler and not a C/C++ library from the dark ‘ol 19s.

What I wanted was a bundler that does most of the heavy-lifting in a close-to-the-metal language for the user without requiring any interaction with its internals. Then I found Rust. A smart and concise systems language that shows off some laudable features like a fearless concurrency model, type safety, and more! I would expect as much from using C/C++ but I’d rather stick with Rust since it’s pretty straightforward when it comes to memory management.

Why another bundler?

So what’s the take here? Why do we need another build tool since we already have amazing ones like webpack, Parcel, Rollup, etc? I’ll take you along with a few reasons why. Perhaps you might have your own interests in having your development and production build times reduced heavily.

It’s 2019, we don’t need slow tools no more

Even though Packem is faster than webpack 4, it is more than twice as fast as Parcel (with multicore compilation). In a benchmark test, we bundled Lodash v4.17.1 with both Packem and Parcel and this was the result:

Benchmark test acquired by running Packem against Parcel.

Never take any benches at face value. You can test it out for yourself here.

The reason why I didn’t bother benchmarking Parcel against webpack was because webpack 4 is profoundly faster than Parcel. I proved this fact by using Sean T. Larkin’s own benches and a thread to it on Twitter can be found here.

Because we can. Anyone can, right?

Of course, what will make the most sense, is because we can. We had the idea of having faster bundle times with a Rusty interface either with FFI or WASM (was still unsure by then). FFI was more reasonable as far as speed and DX was concerned, so we went with having Packem implemented in Rust FFI bindings.

We experienced a few thread-related issues so we didn’t make much use of the available resources. As a result we used multiple node child processes (with node-worker-farm), the same technique Parcel uses for multicore compilation, but for larger module graphs since it adds a significant startup time on top of Node’s uptime when used with smaller module graphs.

Configuration style

This was a tricky part. There were a lot of questions that needed a good answer to make up to picking the right configuration style. Static or dynamic? JSON/YAML/TOML? Our choice was based entirely on whether we needed Packem to:

  1. Have a neater configuration style, and

Bottomline, we proceeded with a static configuration style since we found it to be exactly what we needed. Something that could declaratively tell Packem how to manage the bundle cycle. All the limits to having a static configuration were made clear.

Another aspect of interest was the type of file we should use for the configuration. JSON that is more common to an overwhelming majority of JavaScript developers or YAML/TOML/XML-style which are less common but have their own advantage(s). A suggestion was still made for JSON support (#5).

JSON just didn’t cut out because of all the unnecessary string quotes, curly & block braces, which makes sense since it’s a data interchanging format. An XML-ish approach deserves no respect with regards to being used as a configuration format since it makes things worse than JSON as far as unnecessary characters are concerned. TOML introduced a lot of new lines, and debugging nested options didn’t appear to be eye-appealing since we knew that Packem plugins could get really nesty.

The final winner was YAML! It was able to pass through all aspects of being a proper configuration format (for Packem at least). It:

  1. Makes configuration painless.

Here’s an example of a typical Packem configuration (packem.config.yml). Check for yourself and think about writing the same content in a JSON/TOML/XML-ish style.

FYI, only the first two options are necessary! 😉

Extending Packem

This feature is not yet implemented.

Sometimes we might need to use a feature that doesn’t yet exist, might not be implemented in Packem or is very specific to our project. For that case, you’d have two ways of solving your needs:

  1. Create a Packem plugin for your use case (which is the recommended option).

Using Rust gives us the chance to reform the LC into other binary formats, such as WebAssembly, which will enable Packem to exhibit multiple compile targets:

  1. A NAPI-based C/C++ addon with platform-specific binaries required by Packem’s default RC.

The last two are not yet on the radar since internal refactorings are still being sorted out.

The advanced guide is soon expected to show you how to build a custom build tool using Packem’s binaries to fit your own needs in case you need to use Packem outside the browser and Node environments. These binaries complete the entire graph generation, and duplicate filtering and other graph-related aspects. This means you can use your custom serializer, file watcher, plugin system, etc. It is much like how you can build your custom renderer over OpenGL.

  1. You can still embrace Packem’s plugin system since it will allow you to integrate Packem’s ecosystem of plugins with your custom bundler.

Current state

  • Code Splitting for development and production modes.

What’s next?

These are the features we’re hoping to have soon in the upcoming releases. With your efforts we could get bundling done the right way. When Packem is at 1.0, we’re expecting to have full support for all the features listed below and the others mentioned in Packem’s roadmap.

  • A browser-compatible standalone of Packem with the LC in WebAssembly for closer integration with the underlying system. Axel Rauschmayer already made a feature request to have a Node-compatible version in WASM. For the record, we’ll be working on both soon.

Resources

Packem hasn’t reach 1.0 yet. If you have found Packem to be interesting at all to you, try contributing to Packem itself by creating plugins, updating the documentation, supporting us financially, representing Packem at conferences or any other means. We appreciate your efforts!

Happy bundling! 🧡🧡🧡

freeCodeCamp.org

This is no longer updated. Go to https://freecodecamp.org/news instead

Bukhari Muhammad

Written by

A caffeine reliant being that happens to be good at React & Node related stuff. Coder/writer by day, super-napper at night. JS lover; IE hater; burden reducer.

freeCodeCamp.org

This is no longer updated. Go to https://freecodecamp.org/news instead

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade