Introducing esprint: a fast, open source eslint CLI
Arthur Lee & Allen Kleiner | Pinterest engineers, Core Experience
At Pinterest, we care deeply about developer productivity. We want to empower engineers to move fast while maintaining high code quality. Our web codebase contains more than 4,000 JavaScript files, each rigorously linted using eslint and a combination of Pinterest’s and Airbnb’s JavaScript lint rules. We believe a robust and comprehensive set of rules encourages our engineers to write consistently readable and maintainable code without falling into syntactical traps. However, with a growing codebase and a large set of rules, lint times quickly became the slowest path in our build. This was starting to slow down the development process, so we built esprint, a fast linting solution for both CI and development environments that we’re open sourcing today.
Multi-threaded eslint
The biggest bottleneck we identified was that eslint was running on a single thread. We used the worker-farm package to parallelize eslint runs across multiple cores. We then took the list of files that needed linting and distributed them to multiple worker processes. This gave us immediate gains with speedups of ~3x on an 8-core machine. This was reasonable given the overhead involved in starting worker processes in node.
This was a significant improvement, especially for our continuous integration (CI) jobs. However, we run eslint every time a file changes in our development environments so engineers have regular feedback about the state of their code. Running a CPU-consuming process every time a file changes would be too slow while also possibly slowing down more important processes, such as webpack. It became clear we needed a better solution for the dev environment.
Inspired by flow
Flow is Facebook’s type-checker for JavaScript. While flow isn’t a style checker like eslint, it also needs to read many files, process them and return results. Flow does this elegantly by performing a large amount of work in parallel during its initial run and keeping a background server running to cache results so that subsequent runs are almost instantaneous. We drew inspiration from flow’s architecture, spinning up a background server that caches lint results in memory and watches source files for changes.
Introducing esprint
We called the new eslint cli esprint (pronounced E-S-sprint). It uses a multi-threaded lint runner to lint files and spins up a background server to watch the source files using Facebook’s watchman. Whenever files change, watchman picks up the change, and esprint preemptively lints those changed files. Running the esprint command simply queries the background server for the latest lint status. Since linting each additional file takes very little time, the runtime of esprint is in the order of seconds, with the exception of the initial run.
As a result, we have a fast linting solution that is suitable for both CI and development environments. Looking ahead, we plan to improve the reliability of the background server as well as performance using techniques like reusing eslint engine instances. You can find the code for the beta release on GitHub. We welcome any issues and pull requests!
Acknowledgements: Thank you to the Core Experience team and our web engineers for testing and giving feedback.
