Sunsetting MethodsCount.com
So Long, and Thanks for All the Fish!
A month ago we celebrated methodscount.com 3 years anniversary, and it seems almost appropriate to communicate that the service will be taken down in the coming weeks.
This post was written collaboratively by me and my dear friend Nicola, who I thank again for having helped me creating this tool.
A brief history
For those of you that don’t know us or have never used it, MethodsCount is (was?) a service that allows to compute the number of methods of a given library for Android. Each library comes with the methods count, the byte size, and the list of dependencies, each with their own details.
It started like many of the things start, I (Sebastiano) needed it. The company I was working for had a great deal of users in markets where cheap Android devices are the most common ones, therefore I didn’t want to go over the infamous 65K methods limit. I was manually using Mihai Parparita’s excellent dex-methods-count tool, but it involved quite an amount of work: downloading libraries, extracting dependencies, running the tool and checking the output.
The idea was thus born: a service that was able to do such computations automatically, with a cache that provided instantaneous results for already computed libraries. Perhaps an integration with Android Studio, that would’ve been even more convenient.
Service architecture
The service has been created using Sinatra + Javascript, supported by a MySQL database. We will break it down and observe each component.
Core
At the core of the service we have the methods count. In order to compute this information, we use Gradle to orchestrate the different steps of the computation pipeline:
- Fetch the library
- Extract the
classes.jararchive - Run
dxto produce the DEX file - Inspect the DEX header and extract the methods count information
What’s important to know is that the computation for each library is quite CPU intensive, in particular if the dependencies have themselves not been computed yet: it’s a recursive procedure! This is also why we try to cache as much as possible (more details soon).
Backend
The backend is written in Sinatra (a Ruby web development framework). It’s been necessary to split the app into services, due to the different resource requirements of serving users (possibly at a big scale) and execute CPU intensive library computation tasks on demand.
The result is a 2 parts backend:
- the user facing component, which serves the web pages using already computed content (from MySQL) and forwards user requests to the background workers if needed (i.e., the library has not been cached yet);
- the background workers: they receive and compute asynchronously the user requests.
We chose Amazon Web Services to host our backend, mainly because Nicola was very experienced with it. The services are running on different EC2 instances, and have different scaling policies.
In order to simplify the operational effort to give shape to our infrastructure and maintain it, we have leveraged Elastic Beanstalk, keeping the database manually configured on RDS in order to be able to share it between the two services.
The frontend-server communicates with background workers via SQS (the AWS message queuing service).
The background-workers listen to the queue and process the items (i.e., the libraries) as they come, scaling out into multiple parallel workers in case of demand (the Elastic Beanstalk worker environment helped a lot in setting up the autoscaling policies). The frontend component is also autoscaled and multiple instances a load-balanced via an ELB.
Frontend
The web application is entirely written in Javascript (so to decouple the development of frontend and backend) and it consumes a set of APIs provided by the backend. The design language of choice is Google’s Material Design, and is rendered using the MaterializeCSS framework.
This architecture allows us keep the interface responsive while the user is waiting for the result to be computed. The waiting is implemented via polling; a push-based implementation was in the works, but never saw light of day. The frontend-server, once the library computation request has been forwarded to the workers, polls the database for the result.
This approach has saved us already once when we experienced a partial outage: a single background worker went out of memory (the library graph computation is also memory intensive), but it wasn't replaced by the autoscaling group due to a badly configured health check. The website kept working until we noticed the problem and, once the workers were back up, all the pending library requests in the queue have been processed. No data loss, user only partially affected (already computed library could still be fetched).
We use a bash script leveraging the AWS CLI in order to update the SDK we use in the workers. Given that the update takes quite a few resources and time, it would not be ideal to execute it every time a new instance is created. We therefore bake the updated SDK into an Amazon Machine Image and essentially update the Elastic Beanstalk configuration for the EC2 instance.
We just recently dropped the support for the Android Studio integration, due to lack of time for maintenance and excess of consumed resources.
Statistics
This is a brief summary of what MethodsCount has been able to achieve.
Total number of libraries (unique by FQN): 55,364
Total number of unique libraries (version is discarded): 16,667
Total number of computed methods: 71,968,176
Total pageviews: 6,044,143
Total sessions: 573,596
Total users: 143,702
Average session duration: 00:05:16
We have also calculated some tables that might be interesting for you to take a look at:
Takedown
There are a few reasons why we decided to end support for MethodsCount.
The cost
Keeping in mind the architecture that we described before and the CPU intensive operations that we needed to run to compute the number of methods, MethodsCount was not cheap to maintain.
We tried a donation-based sustainability model, but it clearly didn’t work (but thanks for those of you who did it anyway!).
We also tried to seek sponsorships from Google and Amazon, but without much success.
The original motivation
Two years ago, when the website was launched, managing to stay below the 65K methods limit was a big deal and required a big effort. Now, as we move towards a brighter future, this need is no longer a major priority for the Android community.
The time
This is mainly a consequence of the previous point. Given that the motivation was beginning to fade, so was our time committed to this project.
FOSS
We decided to open-source the code for MethodsCount, both for frontend and backend, as well as the database that contains all the statistics that we have collected so far. Feel free to take a look around!
https://github.com/dextorer/MethodsCount
What’s next
The website is now in read-only mode, meaning that no new libraries can be computed. We have no immediate plans for a total shut down, but that might change in the future.
We would like to thank you all for showing appreciation to the service and to us! It was really heartwarming to see that something we created was used by more than 2000 people on a daily basis.
Even more so, a huge thanks to all of you who donated to keep the service running. No matter how small the donation was, it meant a lot to us!
But in all the sadness, a silver lining should brighten our day:
Brace yourselves
minSdk21 is coming
