What makes Putout so fast?

„Speed is not part of the true Way of strategy. Speed implies that things seem fast or slow, according to whether or not they are in rhythm. Whatever the Way, the master of strategy does not appear fast.“ — Miyamoto Musashi

Hi folks! The most exiting news I have for today, besides the Book of AST, is my speed adventures, so fasten your seatbelts, let’s go!

Cache

OK, let’s check 🐊Putout itself. At the moment of writing it has 2402 files that includes:
- ✅ JavaScript;
- ✅ JSON;
- ✅ Markdown (+ js, ts, json);
- ✅ yaml;
- ✅ `.gitignore` and `.npmignore`

Here is results on digital ocean droplet:

coderaiser@cloudcmd:~/putout$ time redrun lint
> putout . — raw — rulesdir rules
real 0m23.321s
user 0m21.971s
sys 0m1.137s

Why so fast you ask? The reason is: the cache located in

node_modules/.cache/putout/places

it’s created on a first correct with 3 unique things:
- version of node.js;
- version of 🐊Putout;
- options;

What we will when some of this parts changes is the same as using ` — fresh` flag:

coderaiser@cloudcmd:~/putout$ time redrun lint — — fresh
> putout . — raw — rulesdir rules “ — fresh”
real 4m24.870s
user 4m16.093s
sys 0m5.757s

4 minutes, so it about 264 seconds, so if we divide count of files on time we will receive:

2402 / 264 = 9

9 files / second without using a cache!

And this is result of running 🐊Putout + ESLint on so many supported formats! And this is only for a fresh run on such a big project.

Memoization

Memoization — an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.

© Wiki

The simplest plugins format is Replacer. It uses a lot of 🦎PutoutScript, here is example:

module.exports.report = () => 'any message here';

module.exports.replace = () => ({
    'let __a = __b': 'const __a = __b',
});

Expressions, like this one:

let __a = __b

Are parsed to AST form. This is expansive operation, but this short expressions are repeated often, so memoized version of parsing used to make things faster.

What can make things even more faster?

There is a couple operations that used in 🐊Putout most of the time:

• reading/writing files;
• parsing files;
• parsing 🦎PutoutScript;
• traversing;

Working with file system want give us any benefit, so we can skip it.

Parsing files is better, but we came back to it later. Traversing should be made on JavaScript side, since all the plugins JavaScript-based.

So we can parse 🦎PutoutScript faster. How?

napi-rs

A minimal library for building compiled Node.js add-ons in Rust.

The first thing that came on mind is using Rust to speed things up. But as was mentioned earlier memoization used for the purpose of parsing, and when we need to compare nodes this is also very fast operation (about 2ms), and for the purpose of using Rust we need do one of things:

• traverse JSON using Rust;
• convert to BSON and then pass result to Rust, and then convert again;

Both cases hardly can beat 2ms of JavaScript-based compare.

wasm-bindgen

Facilitating high-level interactions between Wasm modules and JavaScript.

Almost the same but WebAssembly based solution. Again there is no answer on a question, what should be written on Rust 🤷‍♂️.

swc

Rust-based platform for the Web

OK, that can be a solution, since we still have a parsing of files, which is definitely can be speeded up.

Here is the code we have to compare speed of a very big file (44k):

const {readFileSync} = require('fs');
const {parseSync} = require('@swc/core');
const {parse} = require('putout');
const source = readFileSync('./lib/cli/index.spec.js', 'utf8');
console.time('swc #1');
parseSync(source);
console.timeEnd('swc #1');
console.time('swc #2');
parseSync(source);
console.timeEnd('swc #2');
console.time('babel #1');
parse(source);
console.timeEnd('babel #1');
console.time('babel #2');
parse(source);
console.timeEnd('babel #2');

Here is results:

swc #1: 13.898ms
swc #2: 13.33ms
babel #1: 236.062ms
babel #2: 132.123ms

SWC more then 10 times faster than Babel! So we can use it for parsing.

But. It has a different internal format: AST. This can be solved actually, there is a tool swc-to-babel that aims to convert SWC format to Babel, but there is a lot of differences, and SWC doesn’t support:

• comments;
• location data (solved);
• partial parsing;
• all new syntax Babel supports

And even after making some conversion happen, such simple transformations ad removing the node just doesn’t work. Also using additional parser make things less stable, as we already using Recast, Babel and ESLint. Each of them brings own adventures.

Conclusion

Yes, there is things that can be improved, and it definitely will! And I appreciated any help an any of this directions. Anyways things still works very fast.

If you like what I’m doing you can support me on Patreon or ko-fi. And if you need any code transformation just create an issue, I always glad to help 🙂.