The internals of Android APK build process — Article

Understanding the flow of the Android APK build process, the execution environment, and code compilation

Android Dev Notes
Sep 7, 2020 · 8 min read

Table Of Contents

  • CPU Architecture and the need for Virtual Machine
  • Understanding the Java Virtual Machine
  • Compiling the Source Code
  • Android Virtual Machine
  • Compilation Process to .dex
  • ART over Dalvik
  • Understanding each part of the build process.
  • Source Code
  • Resource Files
  • AIDL Files
  • Library Modules
  • AAR Libraries
  • JAR Libraries
  • Android Asset Packaging Tool
  • resources.arsc
  • D8 and R8
  • Dex and Multidex
  • Signing the APK
  • References

Understanding the flow of the Android APK build process, the execution environment, and code compilation blog post aims to be the starting point for developers to get familiar with the build process of Android APK.

CPU Architecture and the need for Virtual Machine

Unveiled in 2007, Android has undergone lots of changes related to its build process, the execution environment, and performance improvements.

There are many fascinating characteristics in Android and one of them is different CPU architectures like ARM64 and x86

It is not realistic to compile code that supports each and every architecture. This is where Java Virtual Machine is used.

Understanding the Java Virtual Machine

JVM is a virtual machine that enables a computer to run applications that are compiled to Java bytecode. It basically helps us in converting the compiled java code to machine code.

By using the JVM, the issue of dealing with different types of CPU architecture is resolved.

JVM provides portability and it also allows Java code to be executed in a virtual environment rather than directly on the underlying hardware.

But JVM is designed for systems with huge storages and power, whereas Android has comparatively low memory and battery capacity.

For this reason, Google has adopted an Android JVM called Dalvik.

Compiling the Source Code

Our Java source code for the Android app is compiled into a .class file bytecode by the javac compiler and executed on the JVM.

For Kotlin source code, when targeting JVM, Kotlin produces Java-compatible bytecode, thanks to kotlinc compiler.

To understand bytecode, it is a form of instruction set designed for efficient execution by a software interpreter.

Whereas Java bytecode is the instruction set of the Java virtual machine.

Android Virtual Machine

Each Android app runs on its own virtual machine. From version 1.0 to 4.4, it was ‘Dalvik’. In Android 4.4, along with Dalvik, Google experimentally introduced a new Android Runtime called ‘ART’.

Android users had the option to choose either Dalvik or ART runtime in Android 4.4.

The .class files generated contains the JVM Java bytecodes.

But Android has its own optimized bytecode format called Dalvik from version 1.0 to 4.4. Dalvik bytecodes, like JVM bytecodes, are machine-code instructions for a processor.

Compilation Process to .dex

The compilation process converts the .class files and .jar libraries into a single classes.dex file containing Dalvik byte-codes. This is possible with the dx command.

The dx command turns all of the .class and .jar files together into a single classes.dex file is written in Dalvik bytecode format.

To note, dex means Dalvik Executable.

ART over Dalvik

Since Android 4.4, Android migrated to ART, the Android runtime from Dalvik. This execution environment executes .dex as well.

The benefit of ART over Dalvik is that the app runs and launches faster on ART, this is because DEX bytecode has been translated into machine code during installation, no extra time is needed to compile it during the runtime.

ART and Dalvik are compatible runtimes running Dex bytecode, so apps developed for Dalvik should work when running with ART.

The JIT based compilation in the previously used Dalvik has disadvantages of poor battery life, application lag, and performance.

This is the reason Google created Android Runtime(ART).

ART is based on Ahead — Of — Time (AOT) based compilation process where compilation happens before application starts.

In ART, the compilation process happens during the app installation process itself. Even though this leads to higher app installation time, it reduces app lag, increases battery usage efficiency, etc.

Even though dalvik was replaced as the default runtime, dalvik bytecode format is still in use (.dex)

In Android version 7.0, JIT came back. The hybrid environment combining features from both a JIT compiler and ART was introduced.

The bytecode execution environment of Android is important as it is involved in the application startup and installation process.

Understanding each part of the process.

Source Code

Source code is the Java and Kotlin files in the src folder.

Resource Files

The resource files are the ones in the res folder.

AIDL Files

Android Interface Definition Language (AIDL) allows you to define the programming interface for client and service to communicate using IPC.

IPC is interprocess communication.

AIDL can be used between any process in Android.

Library Modules

Library module contains Java or Kotlin classes, Android components, and resources though assets are not supported.

The code and resources of the library project are compiled and packaged together with the application.

Therefore a library module can be considered to be a compile-time artifact.

AAR Libraries

Android library compiles into an Android Archive (AAR) file that you can use as a dependency for an Android app module.

AAR files can contain Android resources and a manifest file, which allows you to bundle in shared resources like layouts and drawables in addition to Java or Kotlin classes and methods.

JAR Libraries

JAR is a Java library and unlike AAR it cannot contain Android resources and manifests.

Android Asset Packaging Tool

Android Asset Packaging Tool (aapt2) compiles the AndroidManifest and resource files into a single APK.

At this point, it is divided into two steps, compiling and linking. It improves performance, since if only one file changes, you only need to recompile that one file and link all the intermediate files with the ‘link’ command.

AAPT2 supports the compilation of all Android resource types, such as drawables and XML files.

When you invoke AAPT2 for compilation, you should pass a single resource file as an input per invocation.

AAPT2 then parses the file and generates an intermediate binary file with a .flat extension.

The link phase merges all the intermediate files generated in the compile phase and outputs one .apk file. You can also generate R.java and proguard-rules at this time.

resources.arsc

The output .apk file does not include the DEX file, so the DEX file is not included, and since it is not signed, it is an APK that cannot be executed.

This APK contains the AndroidManifest, binary XML files, and resources.arsc.

This resource.arsc contains all meta-information about a resource, such as an index of all resources in the package.

It is a binary file, and the APK that can be actually executed, and the APK that you often build and execute are uncompressed and can be used simply by expanding it in memory.

The R.java that is output with the APK is assigned a unique ID, which allows the Java code to use the resource during compilation.

arsc is the index of the resource used when executing the application.

D8 and R8

Starting from android studio 3.1 onwards, D8 was made the default compiler.

D8 produces smaller dex files with better performance when compared with the old dx.

R8 is used to compile the code. R8 is an optimized version of D8.

D8 plays the role of dexer that converts class files into DEX files and the role of desugar that converts Java 8 functions into bytecode that can be executed by Android.

R8 further optimizes the dex bytecode. R8 provides features like optimization, obfuscation, remove unused classes.

Obfuscation reduces the size of your app by shortening the names of classes, methods, and fields.

Obfuscation has other benefits to prevent easy reverse engineering, but the goal is to reduce size.

Optimization reduces the DEX file size by rewriting unnecessary parts and inlining.

By doing Desugaring we can use the convenient language features of Java 8 in older devices.

Dex and Multidex

R8 outputs one DEX file called classes.dex.

If you are using Multidex, that is not the case, but multiple DEX files will appear, but for the time being, classes.dex will be created.

If the number of application methods exceeds 65,536 including the referenced library, a build error will occur.

The method ID range is 0 to 0xFFFF.

In other words, you can only refer to 65,536, or 0 to 65,535 in terms of serial numbers.

This was the cause of the build error that occurred above 64K.

In order to avoid this, it is useful to review the dependency of the application and use R8 to remove unused code or use Multidex.

Signing the APK

All APKs require a digital signature before they can be installed or updated on your device.

For Debug builds, Android Studio automatically signs the app using the debug certificate generated by the Android SDK tools when we run.

A debug Keystore and a debug certificate is automatically created.

For release builds, you need a Keystore and upload the key to build a signed app. You can either make an APK file with apkbuilder and finally optimize with zipalign on cmd or have Android Studio handle it for you with the ‘Generated Signed Apk option’.

References

https://developer.android.com/studio/build

https://github.com/dogriffiths/HeadFirstAndroid/wiki/How-Android-Apps-are-Built-and-Run

https://logmi.jp/tech/articles/322851

https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html

https://speakerdeck.com/devpicon/uncovering-the-magic-behind-android-builds-droidfestival-2018

by androiddevnotes on GitHub

Find us at:

Discord: https://discord.gg/HqTbjss

GitHub: https://github.com/androiddevnotes/

YouTube: https://www.youtube.com/channel/UCQATLaT0xKkSm-KKVQzpu0Q

Twitter: https://twitter.com/androiddevnotes

Instagram: https://www.instagram.com/androiddevnotes

Contributions:

Contributions and Pull requests are welcomed at https://github.com/androiddevnotes repositories!

🐣

Android Dev Notes

Android open-source projects, notes, tutorials, and tips and tricks.