Preventing Log Creep

In early 2020, the logs from AndroidX CI runs were huge. For example, this log has 1,840 lines and this log has 1,566,722 lines.

We found we could shorten these logs via post processing, and filter out previously expected warnings with a list of regexes, which today includes things like this:

Hiding the expected warnings addressed the immediate problem but left us with two more:

• We needed a way to keep this list of regexes up to date.
• We didn’t want to encourage our team to pay less attention to important warnings.

The way we keep this list of regexes up to date now is by validating the list itself on every change. If a change introduces a new warning, the warning must be addressed or exempted before the change can be submitted.

Having to update an exemptions file also seems to be an effective way to remind us to handle these warnings. If the author of a change overlooks a warning, code reviewers will often ask about it (example). If we can’t easily fix a warning, we might ask for support from a dependency (example), or file a bug to a teammate (example). We think that we’re actually looking more carefully at our build warnings now than we were before (example).

We’ve also found that having an up-to-date warnings exemption file helps us to review past warnings because it allows us to know when those warnings were introduced (example warning).

A few details about how we make this work:

We modified gradlew to save build output into a file that we process here. Because this disables Gradle’s status display, we put these features behind a couple of flags, which are enabled in CI and can be enabled by users too. We also normalize some file paths (showing these shorter paths to the user instead) and remove any leftover color characters before doing the regex matching.

If a new warning is detected and triggers a failure, we indent the warning text, suggest an autogenerated exemptions file, and offer a guess of a simpler way to reproduce the failure than the full CI build.

This process gives us the following recent build failure log, short enough to inline here:

Buildbot Version: androidbuild_buildbot_20230919.00_RC00
Machine: prod-2004–5534
DIST_DIR=/buildbot/dist_dirs/aosp-androidx-main-linux-androidx_incremental/P63279351 CHANGE_INFO=/buildbot/tmp/P63279351-changeInfo frameworks/support/busytown/androidx_incremental.sh

############################################################################
Attempting to locate the relevant error messages via build_log_simplifier.py
############################################################################

Detected these failing tasks: []
FAILURE: Build failed with an exception.

* What went wrong:
Output file $DIST_DIR/inspection was declared as an output of multiple tasks: task ':work:work-inspection:copyInspectionArtifacts' and task ':sqlite:sqlite-inspection:copyInspectionArtifacts'

> Get more help at https://help.gradle.org.

* Exception is:
org.gradle.api.GradleException: Output file $DIST_DIR/inspection was declared as an output of multiple tasks: task ':work:work-inspection:copyInspectionArtifacts' and task ':sqlite:sqlite-inspection:copyInspectionArtifacts'
at androidx.build.ListTaskOutputsTaskKt.findAllTasksByOutput(ListTaskOutputsTask.kt:139)
at androidx.build.ListTaskOutputsTask.computeOutputText(ListTaskOutputsTask.kt:95)
at androidx.build.ListTaskOutputsTask$outputText$2.invoke(ListTaskOutputsTask.kt:37)
at androidx.build.ListTaskOutputsTask$outputText$2.invoke(ListTaskOutputsTask.kt:37)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at androidx.build.ListTaskOutputsTask.getOutputText(ListTaskOutputsTask.kt:37)
at androidx.build.ListTaskOutputsTask._init_$lambda$0(ListTaskOutputsTask.kt:43)
at jdk.proxy1/jdk.proxy1.$Proxy89.graphPopulated(Unknown Source)

BUILD FAILED in 5m 23s


Gradle command failed:
./gradlew \
 - ci \
buildOnServer \
checkExternalLicenses \
listTaskOutputs \
exportSboms \
 - profile \
-Pkotlin.incremental=false