Libraries.io February Progress Update
BitBucket and GitLab support
The biggest feature to ship was support for finding and parsing dependency information from source code repositories hosted on GitLab and Bitbucket.
Previously Libraries.io could only pull in information about GitHub repositories and it’s taken a few months to untangle the hard coded GitHub logic throughout the application.
We’re slowly spidering both services and so far have around around 300,000 repositories on GitLab and Bitbucket, this helps give a much better picture of the whole open source ecosystem rather than just the one on GitHub.
Other Notable Features
During the great repository refactoring, we also made switched the approach to parsing dependency information we were taking.
Previously we had a separate node.js web service call Librarian, which the background workers interacted with over http, but now we’ve switched to using the ruby library Bibliothecary for parsing dependencies, which runs directly in the background workers.
With this we reduced the amount of different web services we needed to run, simplified a whole lot of code and everything is in one language, this also means that Libraries.io is now sharing it’s parser with Dependency CI, so an improvement to the library benefits both services.
We’ve started to codify the infrastructure and server configuration that Libraries.io with Ansible, it’s all being done in the open on the new infrastructure repository: https://github.com/librariesio/infrastructure
I also rolled out a new small service to replace our usage of the npm rss feed, which was pretty terrible, instead npm-update-stream directly follows the changes feed and produces a list of recently updated project names for watcher to consume every 60 seconds.
Ben has been working to improve Libraries.io’s documentation throughout February. He re-wrote much of the old, developer-centred documentation to make Libraries.io a more welcoming project for designers, copywriters, and users alike. You can check it out at docs.libraries.io
A big part of this has involved re-writing our contributor’s handbook and the templates we use for issues and pull requests, which we’ve rolled out to all the repositories in the Libraries.io organisation.
In line with our improvements to documentation we’ve also improved our ticketing. Inquisitive, would-be contributors can now find issues to pick up by size, skill or type and can easily find issue to help others with. We even have a label for first-time contributors, explanations of each label can be found on the documentation repo.
And finally, in return for financial support we made a number of commitments to the Alfred P. Sloan Foundation and, as of January, the Ford Foundation. You can now see those applications and our commitments to these institutions in our new supporters repo.
Unseen Open Source Infrastructure
The blog post went down well and has helped kicked off a number of conversations around how we can use Libraries.io data to pinpoint important projects that might need more support.
42,594 new libraries and 265,766 new versions found, bringing the total to 2,154,734 libraries indexed.
701 commits, 33 pull requests and 220 issues opened across all Libraries.io repositories on GitHub: https://github.com/librariesio
Libraries.io has also discovered over 20 million open source repositories from GitHub, GitLab and Bitbucket.
This month we had a quite a few new contributors whom we would like to thank:
Plans for March
After that I’ll be focusing on beefing up the postgres and elasticsearch infrastructure and refactoring the organisation/user mapping to be less GitHub specific and bring in support for GitLab and Bitbucket Users and Teams.