How fast does my internet need to be to use the Gradle Remote Build Cache? (Part 1)
Is it worth downloading artifacts from a remote build cache node on a slow internet connection?
This question is becoming ever more important as we all switch from our blazing fast office internet connections to our home internet connections.
This is part 1 of a 3 part series. Part 2 is now published.
Downloading a Single Task
How do we know if a remote build cache is worth it? Let’s take a simple Gradle task and calculate whether having a remote build cache is worth it.
If a task takes 10 seconds to execute and generates 1 MB of output, you would need an internet connection faster than 0.1 MB/s in order to save time. An internet connection of 1MB/s would mean that you could download the output of the task in 1 second, saving you 9 seconds of execution time.
Calculation for a real Gradle task
To do this for a Gradle task, you can print its outputs by adding the following to your build.gradle file:
tasks.named("compileDebugKotlin").configure {
outputs.files.forEach {
println(it)
}
}For the KotlinCompile task, the outputs are placed in that module’s build/tmp/kotlin-classes directory.
$ du -sh libs/analytics/build/tmp/kotlin-classes/
136KThis KotlinCompile task produces 136KB of output and takes about 5 seconds to run on my computer. This would require a build cache connection of 0.136MB/5s = .0272 MB/s in order to be able to download the task faster than it would take to run locally.
The download size is actually smaller than 136KB because Gradle compresses artifacts before placing them in the build cache. This speeds up download time but adds decompression time. We’ll dive in to this later.
Let’s check a build scan
A build scan will tell us exactly how much time we saved by reusing the outputs of the task from the build cache. There is one caveat, it uses the amount of time that it took to run the task on a different machine when the remote build cache was seeded. In this case, the build cache was seeded from a significantly slower machine. This task only takes 5 seconds to run on my machine, but took 15 seconds when seeding the cache. To get a more accurate estimate, we need to estimate the minimum connection speed based on the local machines power.
Calculating parallel computation
Machines these days can perform multiple Gradle tasks at once. To perform the same calculation for two tasks run in parallel, we just take the length of the overall build, and divide it by sum of the two task’s outputs.
If two 10 second tasks run in parallel, the length of the overall build would still be 10 seconds. If both tasks produce 1MB of output, the sum of the two task’s outputs would be 2MB.
2MB/10s = 0.2MB/s minimum connection speed
The more tasks our local machine can run in parallel, the faster our connection to the build cache node needs to be!
Compression
Gradle compresses artifacts before storing them in the build cache. In the case of the above KotlinCompile task, the compressed artifacts are only 20KB instead of 136KB. This means our minimum connection speed to make the build cache worth it is much lower: 0.02MB / 5 seconds = 0.004 MB/s. This assumes the artifact decompression time is insignificant.
To calculate the size of the compressed artifact, we need to know the build cache key. We find this in the build scan by clicking on the task.
The build cache stores artifacts by default in the ~/.gradle/caches/build-cache-1 directory where the name of the file is the build cache key, for example:
du -h ~/.gradle/caches/build-cache-1/b59e646b297605c55f49a251bdd7814
20KCalculating a typical build
We want to know if the build cache is worth it in a real scenario like assembling our app. To make this calculation, we would divide the amount of time a build would take by the total amount of compressed artifacts that the build produces. For the build time, we should only include the execution phase of the Gradle build. We exclude configuration time since tasks are neither executed nor downloaded during that time.
We run into a bit of a problem when trying to find the total size of all the compressed build cache artifacts. A Gradle build may run hundreds if not thousands of tasks. There is no public API to get the build cache key for a Gradle task. The only way to find the cache key is to manually look at a build scan. If we had to do that for 1000 tasks, we would be here all day.
In order to provide a real estimate, we will dive in to the internals of Gradle in part 2 of this post.
If you want to run an automatic estimation in your project, apply the Gradle Doctor plugin. Then simply run any gradle task with -PbenchmarkRemoteCache.
./gradlew :app:assembleDebug -PbenchmarkRemoteCacheYou’ll get a nice report like the one below:
=================Gradle Doctor Prescriptions========================
= Remote Build Cache Benchmark Report =
Executed tasks created compressed artifacts of size 159,93 MB
Total Task execution time was 208,85 sIn order for a remote build cache to save you time, you would need an internet connection to your node of at least 0,77 MB/s.
Check a build scan to see your connection speed to the build cache node.
Build cache node throughput may be different than your internet connection speed.A 1 MB/s connection would save you 48,92 s. A 2 MB/s connection would save you 128,88 s.
A 10 MB/s connection would save you 192,86 s.
Note: This is an estimate. Real world performance may vary. This estimate does not take in to account time spent decompressing cached artifacts or roundtrip communication time to the cache node. ====================================================================
Let me know how it works for you and your workload. Is using the remote build cache worth it?
Thank you to Júlio Zynger for proofreading this post.
