Crafting Android bytecode analysis tooling using a secret ingredient (Part 1)

During the development process, we often focus on the source code but rarely inspect the compiled bytecode. This means we’re missing out on a valuable source of information and data for analysis. How? Let’s delve into Dex file inspection and build a tool that demonstrates how source code changes impact the compiled binary.

Have you ever set R8 rules to obfuscate your app? Have you used an APK analyzer or a diffuse tool to understand how the code is compiled? Are you confident that debug code hasn’t leaked into production?

There’s another potential pitfall: libraries may provide obfuscation rules (e.g., Gson) that merge with the ones in your project. This means third-party dependencies can alter configurations for the entire app.

We often assume everything is fine and that we’ll notice if something isn’t right. But will we? Does this make you feel uneasy? Does it concern you? Because it certainly concerns me.

Here, we’ll attempt to enhance the situation and enable you to see precisely how your code changes impact the compiled binary. To better comprehend this, let’s start with the code compilation process:

1. It all begins with the Java and/or Kotlin source code, which is then compiled into JVM .class files.
   Note that at this stage, Java and Kotlin compilers can execute annotation processing tools (APT/KAPT) to generate source code (e.g., Dagger), and Kotlin compiler can run plugins to modify the internal code representation.
2. Then, the D8 compiler takes these compiled classes, third-party libraries (JARs, AARs), and converts them into .dex files.
3. However, if obfuscation and/or minification are enabled (which is almost always the case for release builds), R8 comes into play after D8. R8 obfuscates and optimises the bytecode, and additionally, R8 produces a source map file — a special file listing all the changes and replacements.

Obfuscation replaces human-readable names of various entities (classes, functions, fields, etc.) with very short, yet still unique names, for example, com.example.myapp.Utils might become a.a.b.A.

Bytecode optimization is a complex process that employs various techniques to make the code smaller and more performant. The first and basic optimization is called Tree-Shaking (sometimes referred to as Dead Code Elimination — DCE), which removes unused and unreachable code. More complex optimizations involve changes in the code structure, such as code inlining or outlining. The optimizer can merge the function body with the caller function (similar to the Kotlin ‘inline’ modifier), but it can also do the opposite — outline, extracting similar repetitive code patterns into a separate reusable function.

There are other optimization techniques, but we won’t delve into the details here. It’s important to grasp that the result of optimised bytecode may be very different from what was originally written.

At this point, it’s evident that the only reliable method to comprehend the contents of this “black box” is to inspect the bytecode itself directly. The dex file format is elucidated in this document, so let’s delve into it.

Any dex file initiates with a header where the metadata is stored. This header encompasses a special magic number signifying it as a dex file, along with details like its version, checksum, SHA-1 hash, and pointers indicating the locations of other data structures.

To obtain the actual class names, we must follow these steps:

1. Read the header and store string IDs, type IDs, and data sections.
2. Retrieve entries from the type IDs section.
3. These entries will reference entries in the string IDs section.
4. Retrieve these entries from the string IDs section.
5. Read the values from the data section.

Here’s a simplified diagram of the dex file structure, illustrating how types are stored.

Note that the class names are stored as type descriptors (for more details, click here) which is the standard way to represent any type in a dex file.

Does it seem complex? Well, it should. Remember, the first commercially available Android phone, the HTC Dream, was released back in 2008. It featured a single-core CPU, 192 MB of RAM, and 156 MB of internal storage. The Dex format was designed to be as efficient as possible so it could run even on that hardware.

I’m not sure if this is a good thing or not, but there are several implementations of dex parsing available. One that stands out as particularly interesting is the apkparser project. This project is a hidden gem for a few reasons:

• It is published as a library (com.android.tools.apkparser:apkanalyzer).
• It is actively maintained and kept up to date.
• Although there is no official documentation or quick start guide, there is a comprehensive set of tests that do an excellent job of helping users understand how it should be used.

With this library in hand, it’s possible to start experimenting. Let’s create a basic Java project with JDK 17 toolchain and the following dependencies:

compileOnly("com.android.tools:common:31.1.1")
implementation("com.android.tools.apkparser:apkanalyzer:31.1.1")
implementation("com.android.tools.smali:smali-dexlib2:3.0.3")

Create and build a basic Android app with the following code using ./gradlew assemble.
As a result, there will be apk files in %project_root%/app/build/outputs/apk/

import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.ui.Modifier
import com.example.dex.ui.theme.DexDemoProjectTheme


class MainActivity : ComponentActivity() {
   override fun onCreate(savedInstanceState: Bundle?) {
       super.onCreate(savedInstanceState)
       setContent {
           DexDemoProjectTheme {
               Surface(
                   modifier = Modifier.fillMaxSize(),
                   color = MaterialTheme.colorScheme.background
               ) {
                   Column {
                       Greeting("Android")
                   }
               }
           }
       }
   }
}


@Composable
fun Greeting(name: String, modifier: Modifier = Modifier) {
   Text(
       text = "Hello $name!",
       modifier = modifier
   )
}

And let’s try to build a Java app that lists all the classes in that apk. I’ll start with debug apk first

import com.android.tools.apk.analyzer.dex.DexFiles
import java.io.File
import java.util.zip.ZipFile


fun main() {
   val apk = File(“app.aab”) // use your apk here
   classes(apk)
           .onEach { println(it.type) }
           .count().let { println("In total $it entries") }
}


fun classes(apk: File): List<DexBackedClassDef> {
   return ZipFile(apk).use { zip ->
       zip.entries()
           .toList()
           .filter { it.name.endsWith(".dex") }
           .map { entry ->
               zip.getInputStream(entry).use { input ->
                   DexFiles.getDexFile(input.readBytes())
               }
           }
           .flatMap { it.classes }
   }
}

If you use an IDE like Intellij Idea you can execute it simply by pressing a green triangle button, other IDEs usually provide a similar way of doing it.

After executing it, the list of all the classes in the app will be printed to output:

Landroid/support/v4/app/INotificationSideChannel$_Parcel;
Landroid/support/v4/app/INotificationSideChannel;
Landroid/support/v4/os/IResultReceiver$_Parcel;
Landroid/support/v4/os/IResultReceiver;
Landroid/support/v4/os/ResultReceiver$1;
…
Lkotlin/text/StringsKt;
In total 10198 entries

As we can see, even the most basic app has over 10 thousand classes because the app template includes Kotlin, compose, and several other libraries.

Here’s a simple helper function to transform type descriptors into something more familiar:

private fun String.typeToHumanName() = typeToQualifiedName() ?: this

private fun String.typeToQualifiedName(): String? =
   when (first()) {
       '[' -> substring(1).typeToQualifiedName() + "[]" // array
       'B' -> "byte"
       'C' -> "char"
       'D' -> "double"
       'F' -> "float"
       'I' -> "int"
       'J' -> "long"
       'L' -> substring(1, length - 1).replace('/', '.') // Class path
       'S' -> "short"
       'V' -> "void"
       'Z' -> "boolean"
       else -> null
   }

Let’s add an additional filter so that only the relevant code is analyzed:

fun main() {
   val apk = File(...) // use your apk path from the previous step
   classes(apk)
           .filter { it.type.typeToHumanName().startsWith("com.example.dex.main") }
           .onEach { println(it.type.typeToHumanName()) }
           .count().let(::println)
}

Output:

com.example.dex.main.ComposableSingletons$MainActivityKt$lambda-1$1
com.example.dex.main.ComposableSingletons$MainActivityKt$lambda-2$1
com.example.dex.main.ComposableSingletons$MainActivityKt$lambda-3$1
com.example.dex.main.ComposableSingletons$MainActivityKt
com.example.dex.main.LiveLiterals$MainActivityKt
com.example.dex.main.MainActivity
com.example.dex.main.MainActivityKt$Greeting$1
com.example.dex.main.MainActivityKt
In total 8 entries

See this strange class MainActivityKt? That’s how Kotlin compiles top-level functions. The thing is, functions must belong to a class. If there is no class — which Kotlin allows — the compiler generates one.

So, that’s already enough to calculate a diff of classes (only classes so far!) added and deleted to the binary. Now, let’s slightly modify the app and add another class: Hello which returns one of 2 available greetings randomly.

class MainActivity : ComponentActivity() {
   override fun onCreate(savedInstanceState: Bundle?) {
       super.onCreate(savedInstanceState)
       setContent {
           DexDemoProjectTheme {
               // A surface container using the 'background' color from the theme
               Surface(
                   modifier = Modifier.fillMaxSize(),
                   color = MaterialTheme.colorScheme.background
               ) {
                   Column {
                       Greeting(Hello().getGreetingName())
                   }
               }
           }
       }
   }
}


class Hello {
   val greeting = "Android"


   fun getGreetingName(): String {
       return if (Random.nextBoolean()) greeting else "Another android"
   }
}


@Composable
fun Greeting(name: String, modifier: Modifier = Modifier) {
   Text(
       text = "Hello $name!",
       modifier = modifier
   )
}

Then collect classes from baseline and updated apks and calculate the diff:

fun main() {
   val baselineApk = File(...)
   val changedApk = File(...)


   val classesBaseline = classes(baselineApk)
       .map { it.type.typeToHumanName() }
       .toSet()


   val classesChanged = classes(changedApk)
       .map { it.type.typeToHumanName() }
       .toSet()


   val addedClasses = classesChanged - classesBaseline
   val deletedClasses = classesBaseline - classesChanged


   println("Added classes: ${addedClasses.size}")
   addedClasses.forEach(::println)


   println("Deleted classes: ${deletedClasses.size}")
   deletedClasses.forEach(::println)
}

Output:

Added classes: 1
com.example.dex.main.Hello
Deleted classes: 0

And let’s see how it works with the release app:

Output:
Added classes: 0
Deleted classes: 0

It turns out that the R8 optimizer is efficient. It simply inlines the function and removes the Hello class entirely, so we don’t see any difference here.

It is possible to add rules to R8 on how to process the code. For instance, you can obfuscate a class without applying optimizations by adding the following rule to proguard-rules.pro.

-keepclassmembers,allowobfuscation class com.example.dex.main.Hello {
 *;
}

And the rule works, if we run the analyzer again for release builds, the output will be the following:

Added classes: 1
z1.b
Deleted classes: 0

That’s better, but it’s still not clear which class was added. Yes, it’s obvious in this example, but what if it were a large app with dozens of classes added?

This is where the com.android.tools:r8 library comes to the rescue. It’s another underappreciated library that makes deobfuscation as simple as possible.

Remember, the release build also produces a mapping file. In the case of an APK, it’s saved to %project_root%/app/build/outputs/mapping/release/mapping.txt. For an AAB, it’s bundled right into the archive.

Again, thanks to the great set of tests in the project, retracing class names is as simple as that:

val sourceMap = File("...")
val retracerBaseline = Retracer.createDefault(
   ProguardMapProducer.fromPath(sourceMap.toPath()),
   object : DiagnosticsHandler {}
)
val ref = Reference.classFromDescriptor("Lz1/b;") // Lz1.b; -> obfuscated class descriptor we got on the previous step
val retraced = retracerBaseline.retraceClass(ref).stream().toList()

Combined with analyzer:

val baselineApk = File("...")
val baselineMap = File("...")
val classesBaseline = classes(baselineApk)
   .map { Reference.classFromDescriptor(it.type) }
   .flatMap { retracerBaseline.retraceClass(it).stream().toList() }
   .filterNot { it.isCompilerSynthesized }
   .map { it.retracedClass.typeName.typeToHumanName() }
   .toSet()

Note the important filter — the source mapping file has special notes on synthesized classes. It just makes sense to filter it out if you want to analyze only the code you’ve written. It doesn’t mean that it wouldn’t work without it, but it will produce noise which is not relevant at the moment. Let’s see how it works:

Added classes: 1
com.example.dex.main.Hello
Deleted classes: 0

We’ve just created a very simple yet very powerful tool. Just think about it:

• It’s possible to see exactly what has been added or deleted in the compiled binary with every source code change because the source code can be altered or optimized unpredictably. Comparing the binary provides the most accurate information.
• Whenever a new library is added or an existing one is updated, you’ll be alerted if the library is doing something with obfuscation that it shouldn’t.
• It can also be a great aid in setting up obfuscation rules for R8.

I believe you can come up with many other great use cases.

What’s crucial about this solution is its versatility. You could:

• Create a Gradle plugin to run the analysis on CI and generate a report, similar to what we do in Changeset analysis.
• Potentially halt the CI pipeline if debug dependencies accidentally leak into a production build.
• Or simply develop a JVM app that you use whenever there’s a need to understand what has changed.

Thank you for reading this far, and stay tuned, as I have more to share. In the meantime, let me know if you’ve adopted it in your project, share your feedback, and feel free to ask any questions — I’ll do my best to respond!

Keep an eye out for the next part where I’m going to explore how to analyze class members and get more information about obfuscation, stay tuned!