Supercharging Gradle Builds in CircleCI with S3 Build Cache
TL;DR
Discover how we boosted CI/CD efficiency at Fiverr by overcoming Gradle cache server connectivity issues in CircleCI cloud using the gradle-s3-build-cache plugin, reducing CI execution times by up to 70%.
Introduction
Efficiency is key in CI/CD, and at fiverr, we’re always striving to enhance our development workflows. Recently, our journey into Gradle build caching took an unexpected turn due to connectivity issues with the Gradle cache server in our CircleCI cloud environment.
Understanding the Gradle Build Cache
The Gradle Remote Build Cache accelerates build times by storing and retrieving previous build outputs from a central server. Instead of recomputing tasks, Gradle checks the remote cache during subsequent builds, retrieving compatible outputs. This approach reduces redundant computations, particularly beneficial in collaborative and CI/CD environments. The cache ensures consistency by considering task inputs and configuration details. Overall, it streamlines the development process, providing faster and more efficient builds.
Challenges and Discovering the Solution
Our enthusiasm for integrating the Gradle build cache into our CI pipeline faced a substantial challenge as we intentionally refrained from exposing it outside the VPC due to security considerations. Notably, the global nature of the CircleCI IP enable list at that time posed difficulties aligning with our specific security requirements.
Undeterred, we sought an alternative solution that could seamlessly integrate with our cloud-based CI setup. After thorough research, we found the gradle-s3-build-cache plugin, a game-changer for our CI optimization journey. This plugin allowed us to upload build outputs to S3, introducing flexibility and scalability to our build caching strategy.
Configuring the S3 Build Cache Plugin
val isCI = System.getenv("CI") != null
settings.gradle.settingsEvaluated {
settings.pluginManager.apply(AwsS3Plugin::class.java)
settings.buildCache {
local {
// Local build cache is dangerous as it might produce inconsistent results
// in case developer modifies files while the build is running
isEnabled = false
}
remote<AwsS3BuildCache>().apply {
region = System.getenv("AWS_DEFAULT_REGION_DEV")
bucket = System.getenv("AWS_CI_BUCKET")
prefix = "gradle-cache/${settings.rootProject.name}/"
awsAccessKeyId = System.getenv("AWS_KEY")
awsSecretKey = System.getenv("AWS_SECRET")
isEnabled = isCI
isPush = isCI
}
}
}Benefits and Results
The S3 build cache solution reduced CI execution times by over 70%, with a two-week retention policy and reduced redundancy in the S3 bucket contributing to cost-effectiveness. Leveraging Amazon S3’s low storage costs and scalable pricing model further enhanced the overall efficiency.
Visualising the impact
Conclusion
Our journey to optimize Gradle builds in a cloud CI environment was marked by challenges, discovery, and success. The gradle-s3-build-cache plugin’s adaptability proved invaluable, overcoming limitations with the Gradle cache server in CircleCI. As the tech landscape evolves, finding tailored solutions for specific CI environments becomes increasingly crucial.
