Libraries.io December Progress Update
Continuing from last month with the regular progress updates on Libraries.io since securing a grant from the Sloan Foundation, here’s what I got up to in December.
Notable Features
SourceRank Breakdown for Repositories to give a high level score and some of the key metrics that go towards a good quality open source project.
Time travelling dependency tree resolution allows you to see what the resolved dependency tree of a library would have looked like on any given day, very handy for research where reproducibility of the exact software stack used to generate some data is important.
Here’s a video of the dependency tree growth of npm module “request” between Jan 2011 and July 2016 using that feature:
Added a way to see all the open issues across all the dependencies of a given GitHub repository, great for finding things to contribute to projects that matter to you.
Improved contribution guidelines and readme setup instructions to help new contributors hit the ground running.
Parallel sitemap generation to reduce the time to regenerate the 40 million link sitemap from 2 days to 11 hours.
Added direct tarball download links to many package managers so you can get the source code without having to install it via the CLI.
Support manifests from Go dependency management tools gpm, gb and govendor in
Started investigating how the value of a GitHub star decreases over time, more coming on that soon: https://github.com/librariesio/star-halflife
Created a tool to get a better overview of activity across a GitHub organisation, https://github.com/librariesio/org-pulse, which generates a markdown report of issues, pull requests and commits across all repos in a github org, for example: https://gist.github.com/andrew/fba81a6f8a173ffe540c76f0366f7e4b
We now have a public slack channel where you can get support or get involved in the development of the project: https://slack.libraries.io/
Statistics
140,513 new libraries and 203,145 new versions found, bringing the total to 2,079,755 libraries indexed.
12,661,330 background jobs processed in Sidekiq, down 50% from last month.
24,599,706 http requests and 280GB bandwidth served to 335,303 unique visitors through Cloudflare, of which 5,607,021 were made by Googlebot.
269 commits, 26pull requests and 290 issues opened across all Libraries.io repositories on GitHub: https://github.com/librariesio
Contributors
This month we had a quite a few new contributors whom we would like to thank:
Other changes and improvements
- Update rails to 5.0.1
- Update to ruby 2.3.3
- Only consider runtime dependencies in source rank
- Only show runtime dependency count in project statistics
- Show kind of dependency on repo page
- Show “Other” column on usage graph
- Handle never pushed repos better
- Only update extra repo info if repo has been pushed to since last sync
- Proper paginatable scope for dependent projects
- Show github user bios on profile page
- Only show usage stats for non-forked repos
- Improved unsubscribe pages for projects
- Regular tasks to update outdated information from GitHub
- Better project and version name validations across all package managers to reduce the chance of duplicates
- Fixed semver incompatibility issues between rubygems and npm
- Refactoring work towards supporting repositories on GitLab and BitBucket
- Numerous visual tweaks and fixes
- Improved code climate score to 3.71
- Improved test coverage to 57.7%
Plans for Next Month
I’ve started making milestones for each month’s work, here’s January’s: https://github.com/librariesio/libraries.io/milestone/8, some highlights include:
- More progress on Bitbucket and GitLab support
- Version level metadata
- Issue/PR statistics
- Investigating the impact of civic technology
As well as running 24 Pull Requests during December I also found some time to hack on a little project to help me stay on top of the hundreds of GitHub notifications that Libraries.io and my other open source projects generate.
The result is Octobox, an inbox for your GitHub notifications so you can see exactly what’s going on across all your projects, it’s open source and has already had contributions from 25 other people: https://github.com/octobox/octobox
As always, follow us on Twitter at @teabass, @benjam and @librariesio for more updates.
