Boosting AOSP Build Speed: Pigz and Tar for Optimal Compression

This article explains on leveraging compression time of build artifacts

Photo by Todd Pham on Unsplash

Optimizing the way software is built is crucial for efficient development. It helps speed up the process, get products to market faster, and improve productivity. In AOSP, optimization not only reduces build times but also helps save resources, making the development pipeline more sustainable.

AOSP artifacts are of huge size, and so the compression also takes time with the traditional zip, so lets look at the alternative which I found to speed up the compression of build artifacts.

Pigz — A tool

Pigz is a parallel implementation of the gzip compression program. It allows for faster compression by using multiple processors or cores simultaneously.

When using pigz in conjunction with tar, you can create compressed tarballs for efficient storage and transportation of files.

1. Combining tar and pigz:
- Command Syntax: To create a compressed tarball using pigz, you can use a command like:

tar --use-compress-program="pigz -1" -cf output.tar.gz <source_directory>

• tar: The command-line utility for creating, extracting, and managing tar archives.
• --use-compress-program="pigz -1":
  - --use-compress-program: Specifies an external compression program.
  - "pigz -1": Instructs tar to use the pigz utility with compression level 1 (-1). Compression levels range from 1 (fastest, least compression) to 9 (slowest, best compression).
• -cf:
  - -c: Creates a new archive.
  - -f: Specifies the file name of the archive.
• output.tar.gz:
  - output.tar.gz: The name of the output archive file. In this case, it's a tarball (tar.gz) that will be created by combining the specified directory or files.
• <source_directory>:
  - <source_directory>: The source directory or file(s) to be included in the tar archive.

- This command will create a tar archive of the specified directory and then pipe it to pigz for compression, producing a gzipped tarball.

2. Parallel Compression:
- Advantages: The combination of tar and pigz allows for parallel compression, leveraging multiple processors for faster compression of the archive.

3. Decompression:
- Command Syntax: To decompress a pigz-compressed tarball, you can use a command like:

pigz -d -c input.tar.gz | tar -xvf -

• pigz:
  - -d: Decompresses the input file.
  - -c: Writes the decompressed output to the standard output (stdout).
  - input.tar.gz: The name of the input file to be decompressed. In this case, it's a gzip-compressed tarball named "input.tar.gz."
• | (Pipe):
  - Sends the decompressed output of the pigz command as input to the next command.
• tar:
  - -x: Extracts files from an archive.
  - -v: Verbosely shows the file progress during the extraction process.
  - -f -: Specifies that the input for the extraction comes from the standard input (stdin) rather than a file.
  - -: Indicates that the archive is read from the standard input.

- This command will decompress the gzip file using pigz and then extract the contents using tar.

4. Options and Customization:
- Both tar and pigz come with various options for customization. For example, you can specify compression levels with pigz or include additional tar options as needed.

5. Use Cases:
- Combining tar with pigz is particularly useful for compressing and archiving large directories or datasets, where parallel compression can significantly improve performance.

6. Efficiency and Performance:
- The parallel processing capabilities of pigz complement tar's archiving functionality, making the overall process more efficient, especially when dealing with substantial amounts of data.

7. Compatibility:
- Compressed tarballs created with pigz can be decompressed using standard tools like tar and gzip, ensuring compatibility across systems.

8. Example:

Creating compressed files: tar --use-compress-program="pigz -1" -cf build_artifact.tar.gz build_artifact

Extracting compressed files: pigz -d -c build_artifact.tar.gz | tar -xvf -

While zip has been a standard for a long time, pigz offers parallel processing power.

I hope this article is helpful. Please comment for any discussion in the comment box.