Libraries.io November Progress Update

It’s been one month since Libraries.io secured a grant from the Sloan Foundation and I started working on it full time, I thought it would be interesting to keep track of my progress over the coming months starting with November 2016.

I’m also very excited that Ben is starting full time on Libraries.io from 1st December, I can’t wait to see what he brings to the project with his backgrounds in security, open source and fundraising.

Notable Features

Upgrade to rails 5 opening the door to some wonderful new features.

Version usage statistics for projects to help open source maintainers get a better picture of which releases their users depend on:

Added support for Haxelib, the package manager for Haxe with nearly 1,000 libraries: https://libraries.io/haxelib.

Asynchronous dependency tree resolution to ensure that libraries with large dependency trees still load correctly:

Created the watcher project to monitor rss and json feeds for new packages, reducing the delay in finding new versions to around 30 seconds for many package managers.

Created CocoaPods API project to make it much more efficient to pull information from the newly shared CocoaPods Specs repo.

Improved repository contributor page with contribution percentages:

Cache dependent repo counts onto projects which allows sorting and searching by “most used”.

Created pydeps project to collect dependency information from python libraries using pip, this is needed because parsing the dependency information from setup.py is not reliable enough, instead we have to run “pip install foo==0.1.0” for every version of every python module and see what dependencies it installs 🙈

Added “New” sections to Explore to highlight recently published libraries and repositories:

Statistics

172,489 new libraries and 657,833 new versions found, bring the total to 1,930,000 libraries indexed.

21,464,490 background jobs processed in Sidekiq, the daily job count has been reduced since the introduction of the watcher:

23,906,897 http requests and 297 GB bandwidth served to 341,202 unique visitors through Cloudflare, of which 3,634,707 were made by Googlebot.

426 commits, 26 pull requests and 374 issues opened across all Libraries.io repositories on GitHub: https://github.com/librariesio

Other changes and improvements

• Import CPAN dependencies
• Added support for yarn.lock
• Added support for dev deps in composer.lock and reparsed 80,000 files from GitHub
• Allow users to hide their own profiles
• Download dependencies of all old rubygems which were previously were not available in the Rubygems.org API
• Added last_synced_at timestamps to relevant pages
• Fix a couple bugs around checking out of date versions of dependencies in php
• Cache http requests to package managers in memcached where possible
• Fixed comparing version ranges in Elm
• Download CRAN dependencies
• Download Elm dependencies
• Load old versions of CRAN projects
• Delete removed manifests after syncing
• Removed dependency requirements for valid semver in source rank calculator
• Include latest_stable_release to project search api response
• Allow paginating through org and user contributors
• Improve code climate score to 3.6
• Fixed syncing many cache expiry bugs to ensure latest version numbers are always up to date.
• Improved test coverage to 19% (still terrible)
• Improved efficiency of a number of background jobs
• Various performance improvements around the site
• Fixed edge case syncing errors where github users or organisations have swapped usernames.
• Upgraded the Elasticsearch machine to handle more load, still running on 1.7 for now though.

Plans for Next Month

• Resolve dependency trees for Manifests
• GitLab and BitBucket support
• Searching against many projects at once
• Allow logged in users to request to “resync” projects from package managers
• Source rank for repos
• Version level metadata

and many more bugfixes and improvements as they pop up.