State of package.json dependencies
GitHub repositories contain tonnes of package.json files, extracting all dependencies from them with the help of BigQuery brings us a lot of valuable analytic data. Let's explore them.
Data sample
For this analysis, I selected 559 968 repositories containing package.json on GitHub. I wanted to correlate this data somehow with dates, and the way I did it was by extracting the date from the last commit that modified package.json. The next step was to get all dependencies from package.json, which gave me 8.1 million dependencies (containing both dependencies and devDependencies).
dependencies or devDependencies?
If you take a look at npm trends you get download data. That’s great, but do those downloads come from dependencies or from devDependencies? How often do projects only contain dependencies, devDependencies, or both?
The chart featured above presents situations where only these types of dependencies occur in the repository. It gets more interesting if we try to calculate the average amount of dependencies per project:
Currently, projects on average contain more than 20 dependencies and
maintain linear growth. The trend of increasing amounts of packages is:
+ 0.9 dependency/year
+ 2.0 devDependency/year
Some of the packages are specific for dependencies like express or body-parser and other for devDependencies like mocha or karma. Let’s check where the top 30 most popular packages occur:
The chart above represents data ranging from 2012.01 to 2018.08. But the tool ecosystem for web development is quickly evolving, so it can be more relevant to present data from last year only (2017.08–2018.08):
There can be some confusion as to where some of packages should be put, dependencies or devDependencies? For example, jquery is equally likely to appear in both dependency types, as some projects use it as a normal dependency, while others use jquery as a helper for tests and thus include it in devDependencies.
Let’s examine the most popular problematic packages (when they are equally likely to appear in both types of dependency).
Frameworks
Now things are getting more interesting. Everyone loves to compare the popularity of frameworks, and so do I. But I can be justified by the obligation to reliably present a full analysis of the data.
Grey bars represent the number of projects on GitHub with package.json in the root of the project. The date is calculated based on last commit modifying the above mentioned package.json. Lines represent the percentage of each framework in package.json. You can read this chart like this:
In January, 2017 there were 10 000 projects with the last commit modifying
package.json. 14% of them contained React, 6% Angular, and 3% Vue".
No surprisingly, React is leading with a peak of usage at 16% (May 2017). The beginning of Angular and React/Vue lines are quite different, where the popularity of Angular drastically grew in May 2016 (this is a date of first RC version). This can be explained by the release of Angular, which was a big event in the web industry, especially considering that the previous version was well established in a community. But why Angular 1.x popularity is so low in this plot? It’s due to the way it was distributed. At the beginning, the usage of the npm package system wasn’t popular for Angular 1.x. So on this chart, you mostly see the adoption of downloading Angular 1.x via npm. The same goes for jQuery. It’s surprising that it reached 12% of downloads on npm (yes I know that jQuery is not a framework, but I could not resist including it).
If you are interested why I choose the representation of frameworks in percentages, take a look at the number of framework occurrence each month:
With this representation, framework lines are highly coupled with grey bars, this can hide information about trends. If you compare drops of frameworks in this plot and the previous one, you can see on both plots drop at the beginning of 2018. What does this mean? My explanation for this is that most of the framework related projects can be either:
- abandoned one year ago (peak in October 2017)
- or still maintained (during last year
package.jsonis modified a few times, so it’s peak is shifted towards the end of a chart).
Above plots can be compared with npm tends. But a lot of people try to identify stars on the GitHub as a determinant of the popularity of the framework. If I present the same data in the integral form, it will be easier to compare it with GitHub stars:
It looks like GitHub stars do not translate into popularity in GitHub repositories. Currently, Vue has 113k stars and is used (according to this research) in 7 092 projects, and React has 110k stars and is used in 45 706 projects (it’s 6x more than Vue).
Tasks runners and transpilers (build tools)
These types of dependencies are usually downloaded from npm, so there is no problem like including Angular 1.x from CDN. But some of the projects can still rely on globally installed packages. So in reality, those numbers can be bigger.
Not surprisingly, this graph demonstrates the change in leadership from grunt -> gulp -> webpack. Browserify never managed to break into the mainstream, during its greatest popularity (middle of 2015). Webpack gained more attention and started to dominate. Webpack and babel seem to be inseparable, just look at this ridiculously closely correlated plot lines.
Analyzing the number of packages in projects is fun, but it’s more interesting if we compare their use with the frameworks:
Note how to read this chart: 3% of projects containing React contain also grunt in package.json. You must also adjust for the fact that this chart is calculated based on all data from the beginning of GitHub’s existence until August, 2018.
Based on Angular 1.x creation time, it is clear why it’s mostly used with task runners (42% gulp), not transpilers. React, Angular, and Vue have obviously related transpilers, but little help from gulp is also common in these frameworks (14% — 18%). Browserify is not popular in newer frameworks like Angular or Vue, and mostly appears in React and Angular 1.x projects (12% — 17%). Angular practically does not exist without typescript (also zone.js and rxjs as you later find out). Rollup surprisingly found its place in Angular projects.
Unit tests
It was difficult to choose a representative group of testing tools since they are test runners, test frameworks, assertion libraries, and testing plugins. A lot of types but only one graph to show trends among them.
There are not too many changes in the testing tool field, as frontend projects constantly increasing their usage. Only Angular related jasmine and karma lost some momentum in the recent time. The trend lines for mocha and chai look very similar with a shift of around 10%. These tools are complementary for each other and that's why there can be a positive correlation (next chart will verify it). Facebook’s jest took off rapidly, with the adoption in projects increasing even during a time period when usage of React was slightly declining!
Generally, test workflows are based on multiple tools, thus we have the opportunity to measure a correlation between these packages:
As you can expect, jasmine is used with karma and mocha with chai. Mocha and karma are the most unlikely to be used together in the same project.
Probably, the information above is nothing new to you. But did you know about Pearson correlation coefficients for testing tools? 😎
Jasmine/karmaandmocha/chainever pretended to be framework related projects, but the market decided where they should be used. Stillmochais more popular for React projects than Facebook’sjest. In the official Vue documentation, a recommendation forkarmais featured, but it isn’t actually reflected in this data.
E2E tests
After unit tests, we can try to extract some data about end-to-end test tools. But it’s difficult to analyze, since the most popular ones like selenium are absent in package.json. Also, e2e tools only appear in full projects, and not in small projects like plugins (where this is the most popular case of repositories in GitHub).
As we can expect, the percentage of these packages are not as high as unit test tools. The popularity of protractor is caused by its inclusion in Angular projects. Puppeteer is not strictly an e2e tool, but can be used for testing. Nonetheless, it’s gained huge popularity and momentum in recent months.
Linters
Trends are well defined in this area: there are main 3 linters for JavaScript, only one for TypeScript, and “sort of a linter” prettier.
I’m not surprised by these results. It was always known that jshint is more popular than jscs. Jscs was merged with ESlint in April 2016 and eslint takes the linting industry by storm. Prettier is a new kind of approach here. It turns out that developers are adopting this idea. Additionally, prettier is not competing with eslint, since both tools can be used together.
There is eslint and nothing else in JavaScript linting market. Just some historical usage of jshint in Angular 1.x projects. Angular has its own tslint. Prettier is most likely to be used in React projects because it is created by the Facebook team.
State management
Nowadays, each modern framework must have some sort of state management. Some of them adopt independent packages like redux, and others develop solutions in-house like vuex. I was trying to replicate results from StateOfJS 2017, as they contain data for GraphQL. It’s hard to distinguish a project with GraphQL, since you don’t need any type of tool for a frontend project to use it. But we can somehow correlate graphql tools for creating backend GraphQL endpoints and the graphql-tag helper for dealing with GraphQL queries on the frontend.
You probably guessed where the popularity of redux comes from. It’s clear that the market is highly dominated by only one solution. Sadly mobx is not as popular as they are advertising it. It’s such a shame that graphql-tag values are so low — it’s hard to analyze this data.
Polarization for redux and vuex is obvious here. It’s interesting why firebase is so popular in Angular projects, maybe because both projects are created by Google? GraphQL related tools are most popular in React projects, but only slightly.
Comparisons of packages in React, Vue, and Angular
We can compare two frameworks by plotting the popularity of packages in both frameworks. At first glance, this chart can look complicated. That's why I include an example of how to read it.
Values on the diagonal present a perfect agreement in both frameworks. A point further from the diagonal presents specificity to one of the frameworks.
React and Angular
You can read this plot like this:
rxjsis used in 99.6% projects with Angular and in 1.2% with Reactlodashis used in 25.1% projects with Angular and in 26.3% with Reactclassnamesis used in 0.1% projects with Angular and in 18.1% with Reactxml2jsis used in 0.6% projects with Angular and in 0.6% with React
This plot contains a lot of data and I’m not going to explain all of them (it’s simply impossible). You should search for packages of interest to you.
The most obvious outcome from this plot is highly related to Angular project packages: zone.js, rxjs, core-js, lite-server and all TypeScript packages. Also, you find out highly React related packages like eslint, redux, classname, enzyme, and Babel packages.
React and Vue
Surprisingly or not (if you know that Laravel strongly supports Vue) packages like laravel-mix and laravel-elixir are specific to Vue. Classnames is closer to the diagonal line than on the previous chart. This is due to the fact that you can use JSX and classnames in Vue, and are really helpful used in tandem within Vue.
Angular and Vue
Axios is strongly distinct here — because it’s recommended for fetching data in Vue, while Angular has its own build in solutions. Jquery is also more popular in Vue since it’s easier to use Vue in legacy projects.
npm scripts
If you are interested in statistics on how other people write npm scripts, I’ve analyzed almost a million scripts on GitHub repositories. This post covers current npm scripts trends:
