Open in app

Sign in

Write

Sign in

Unpacking Cloud Native Buildpacks

Domenico Luciani
Buildpacks
Published in
10 min readSep 18, 2023

You can quickly run buildpacks using platforms like pack and kpack, and for most people, this works great. But some buildpacks users, especially those maintaining their own platforms, may want more control over how the underlying buildpack lifecycle phases are executed.

The goal of this tutorial is to build a Bash application without using platform tools like pack or kpack. We will leverage the individual phases of the buildpack lifecycle to produce the application-run image.

Note: Depending on your computer architecture, the tutorial will have slightly different instructions.

Let’s get started! 📚

Getting started

To begin this tutorial, we’ll need a local copy of the lifecycle and the official Buildpack.io samples:

git clone https://github.com/buildpacks/lifecycle
git clone https://github.com/buildpacks/samples

The lifecycle orchestrates buildpack execution and then assembles the resulting artifacts into a final app image. It is composed of multiple phases that need to be executed to have the final image built and exported.

Of course, we need to build the lifecycle in order to use all its phases. You can do it by going into the lifecycle directory and executing

For AMD64 architectures

make build

For ARM64 architectures


 make build-darwin-arm64

Important Environment Variables

In order to execute the various phases correctly, we need to first set these environment variables:

CNB_USER_ID
CNB_GROUP_ID
CNB_PLATFORM_API
CNB_SAMPLES_PATH
CNB_LIFECYCLE_PATH
  • CNB_USER_ID,CNB_GROUP_ID: The user and group ID are arbitrary values that need to be consistent. The usual value for both is 1000, which we can use through this tutorial.
  • CNB_PLATFORM_API: Platform API version will vary depending on your use case. This tutorial makes use of v0.12, which is the most recent at the time of writing this post.
  • CNB_SAMPLES_PATH: The path of our local samples directory
  • CNB_LIFECYCLE_PATH: The path of our local compiled lifecycle dir

Run the following commands in the terminal to set these values:

export CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.12
export CNB_SAMPLES_PATH="/<your-path>/samples"
export CNB_LIFECYCLE_PATH="/<your-path/lifecycle/out/<your-arch>/lifecycle"`

Phases

We are going to go through each of the lifecycle’s phases except for the Creator which combines all of them in one go:

  1. Analyze
  2. Detect
  3. Restore
  4. Build
  5. Export

Analyze

The analyze phase runs before the detect phase in order to validate registry access for all images that are used during the build as early as possible. In this way it provides faster failures for end users

We need to create some directories

mkdir -p apps/bash-script
mkdir -p layers

Those commands will create two directories in the root called apps, which contains a bash-script directory, and layers. The layers directory contains subdirectories representing each layer created by the buildpack in the final image or build cache. Essentially it's our working directory

Then we need to copy the bash-script samples into our apps/bash-script directory. Here is where our app's source code will stay for simplicity.

cp -r "$CNB_SAMPLES_PATH/apps/bash-script/" ./apps/bash-script

Let’s call the analyzer:

For AMD64 architectures

$CNB_LIFECYCLE_PATH/analyzer -log-level debug -daemon -layers="./layers" -run-image cnbs/sample-stack-run:bionic apps/bash-script

For ARM64 architectures — the one I’m using in this tutorial

$CNB_LIFECYCLE_PATH/analyzer -log-level debug -daemon -layers="./layers" -run-image arm64v8/ubuntu:latest apps/bash-script

This command will run the analyzer with:

  • a debug level of logging
  • pointing to our local docker daemon
  • pointing to our layers directory, which is the main lifecycle working dir
  • running the specified image
  • with the path to the app that we are analyzing

Output

The output we will see looks like this one:

❯ $CNB_LIFECYCLE_PATH/analyzer -log-level debug -daemon -layers="./layers" -run-image cnbs/sample-stack-run:bionic apps/bash-script
Starting analyzer...
Parsing inputs...
Ensuring privileges...
Executing command...
Timer: Analyzer started at 2023-08-29T11:22:26+02:00
Image with name "apps/bash-script" not found
Found image with identifier "96f1071770b9f69d475d73d2b033ac869e76fcf6f944116df94f1ffa4cb33e36"
Timer: Analyzer ran for 216.833µs and ended at 2023-08-29T11:22:26+02:00
Run image info in analyzed metadata is:
{"Reference":"96f1071770b9f69d475d73d2b033ac869e76fcf6f944116df94f1ffa4cb33e36","Image":"cnbs/sample-stack-run:bionic","Extend":false,"target":{"os":"linux","arch":"amd64"}}

Since we are using the Platform API v0.12, it is going to check if it has the right privileges, and now inside our layers directory we should have a analyzer.toml file with a bunch of null entries.

Detector

The detector looks for an ordered group of buildpacks that will be used during the build phase

The detector requires an order.toml being present in the root dir, let's craft a new one from samples builder.toml removing the stack section:

cat "$CNB_SAMPLES_PATH/builders/jammy/builder.toml" | grep -v -i "stack" | sed 's/\.\.\/\.\./\./' > order.toml

An order.toml file contains a list of groups, and each group itself contains a list of buildpacks. The detector reads order.toml and looks for the first group that passes the detection process.

Set buildpacks layout directory

Let’s create a buildpacks dir in the root:

mkdir -p buildpacks

Then we need to populate it with the buildpacks we want or need. We have to follow the directory layout defined in the buildpack spec, where each top-level directory is a buildpack ID and each second-level directory is a buildpack version.

For example, the samples/java-maven buildpack:

mkdir -p buildpacks/samples_java-maven/0.0.1
cp -r $CNB_SAMPLES_PATH/buildpacks/java-maven/* buildpacks/samples_java-maven/0.0.1/

Let’s do that for every buildpack in the samples/buildpacks directory:

for f in `ls --color=no $CNB_SAMPLES_PATH/buildpacks | grep -v README`
do
    mkdir -p ./buildpacks/samples_"$f"/0.0.1
    cp -r "$CNB_SAMPLES_PATH/buildpacks/$f/" ./buildpacks/samples_"$f"/0.0.1/
done

Run the Detector

$CNB_LIFECYCLE_PATH/detector -log-level debug -layers="./layers" -order="./order.toml" -buildpacks="./buildpacks" -app apps/bash-script

Output

We should get something like this:

$CNB_LIFECYCLE_PATH/detector -log-level debug -layers="./layers" -order="./order.toml" -buildpacks="./buildpacks" -app apps/bash-script
Starting detector...
Parsing inputs...
Ensuring privileges...
Executing command...
Timer: Detector started at 2023-08-29T12:27:19+02:00
======== Results ========
fail: samples/java-maven@0.0.1
======== Results ========
fail: samples/kotlin-gradle@0.0.1
======== Results ========
fail: samples/ruby-bundler@0.0.1
======== Results ========
pass: samples/hello-world@0.0.1
pass: samples/hello-moon@0.0.1
Resolving plan... (try #1)
samples/hello-world 0.0.1
samples/hello-moon  0.0.1
Timer: Detector ran for 974.819292ms and ended at 2023-08-29T12:27:20+02:00

Between the output files, we should have a group.toml and a plan.toml file

The groups that have passed the detection have been written into the group.tomlfile writing its build plan into the plan.tomlfile.

Here if you want to read more about the group and plan toml files.

Restorer

This phase is mostly about bringing things out of the cache if anything is in there.

Let’s create the cache directory

mkdir cache

and then let’s run the restorer:

$CNB_LIFECYCLE_PATH/restorer -log-level debug -layers="./layers" -group="./layers/group.toml" -cache-dir="./cache" -analyzed="./layers/analyzed.toml"

Output

Getting back something like this:

Starting restorer...
Parsing inputs...
Ensuring privileges...
Executing command...
Timer: Restorer started at 2023-08-30T17:10:54+02:00
Restoring Layer Metadata
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon
Reading Buildpack Layers directory <REDACTED>/github.com/dlion/unpackingCNB/layers
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world
Reading Buildpack Layers directory <REDACTED>/github.com/dlion/unpackingCNB/layers
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon
Timer: Restorer ran for 832.458µs and ended at 2023-08-30T17:10:54+02:00

The cache directory should now be populated by 2 sub-dirs:committed and staging.

Builder

The builder transforms application source code into runnable artifacts that can be packaged into a container.

As a pre-request to run the builder, we first need to create two directories:

mkdir -p platform
mkdir -p workspace
  • platform is where we store configurations and env variables
  • workspace is where we store our app source code and where we build it

Let’s copy the source code from the app directory to the workspace dir:

cp -r apps/bash-script/* ./workspace

Let’s create a launcher file with the instructions to run our application:

cat << EOF > ./layers/samples_hello-moon/launch.toml
[[processes]]
  type = "shell"
  command = ["./app.sh"]
EOF

Let’s run the builder:

$CNB_LIFECYCLE_PATH/builder -log-level debug -layers="./layers" -group="./layers/group.toml" -analyzed="./layers/analyzed.toml" -plan="./layers/plan.toml" -buildpacks="./buildpacks" -app="./workspace" -platform="./platform"

Output

Starting builder...
Parsing inputs...
Ensuring privileges...
Executing command...
Timer: Builder started at 2023-08-30T17:16:43+02:00
Running build for buildpack samples/hello-world@0.0.1
Looking up buildpack
Finding plan
Creating plan directory
Preparing paths
Running build command
---> Hello World buildpack
     platform_dir files:
       total 0
       drwxr-xr-x  2 dluciani  staff   64 Aug 30 17:14 .
       drwxr-xr-x  9 dluciani  staff  288 Aug 30 17:14 ..
     env_dir: <REDACTED>/github.com/dlion/unpackingCNB/platform/env
     env vars:
       declare -x CNB_BP_PLAN_PATH="/var/folders/y5/tcnd_s55369g9v5crnmg2t040000gp/T/samples_hello-world-3790526309/samples_hello-world/plan.toml"
       declare -x CNB_BUILDPACK_DIR="<REDACTED>/github.com/dlion/unpackingCNB/buildpacks/samples_hello-world/0.0.1"
       declare -x CNB_LAYERS_DIR="<REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world"
       declare -x CNB_PLATFORM_DIR="<REDACTED>/github.com/dlion/unpackingCNB/platform"
       declare -x CNB_TARGET_ARCH="amd64"
       declare -x CNB_TARGET_ARCH_VARIANT=""
       declare -x CNB_TARGET_DISTRO_NAME=""
       declare -x CNB_TARGET_DISTRO_VERSION=""
       declare -x CNB_TARGET_OS="linux"
       declare -x HOME="<REDACTED>"
       declare -x OLDPWD
       declare -x PATH="<REDACTED>"
       declare -x PWD="<REDACTED>/github.com/dlion/unpackingCNB/workspace"
       declare -x SHLVL="1"
     layers_dir: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world
     plan_path: /var/folders/y5/tcnd_s55369g9v5crnmg2t040000gp/T/samples_hello-world-3790526309/samples_hello-world/plan.toml
     plan contents:
       [[entries]]
         name = "some-world"       [[entries]]
         name = "some-world"
         [entries.metadata]
           world = "Earth-616"
---> Done
Processing layers
Updating environment
Reading output files
Updating buildpack processes
Updating process list
Finished running build for buildpack samples/hello-world@0.0.1
Running build for buildpack samples/hello-moon@0.0.1
Looking up buildpack
Finding plan
Creating plan directory
Preparing paths
Running build command
---> Hello Moon buildpack
     env_dir: <REDACTED>/github.com/dlion/unpackingCNB/platform/env
     env vars:
       declare -x CNB_BP_PLAN_PATH="/var/folders/y5/tcnd_s55369g9v5crnmg2t040000gp/T/samples_hello-moon-3263588455/samples_hello-moon/plan.toml"
       declare -x CNB_BUILDPACK_DIR="<REDACTED>/github.com/dlion/unpackingCNB/buildpacks/samples_hello-moon/0.0.1"
       declare -x CNB_LAYERS_DIR="<REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon"
       declare -x CNB_PLATFORM_DIR="<REDACTED>/github.com/dlion/unpackingCNB/platform"
       declare -x CNB_TARGET_ARCH="amd64"
       declare -x CNB_TARGET_ARCH_VARIANT=""
       declare -x CNB_TARGET_DISTRO_NAME=""
       declare -x CNB_TARGET_DISTRO_VERSION=""
       declare -x CNB_TARGET_OS="linux"
       declare -x HOME="<REDACTED>"
       declare -x OLDPWD
       declare -x PATH="<REDACTED>"
       declare -x PWD="<REDACTED>/github.com/dlion/unpackingCNB/workspace"
       declare -x SHLVL="1"
     layers_dir: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon
     plan_path: /var/folders/y5/tcnd_s55369g9v5crnmg2t040000gp/T/samples_hello-moon-3263588455/samples_hello-moon/plan.toml
     plan contents:
---> Done
Processing layers
Updating environment
Reading output files
Updating buildpack processes
Updating process list
Finished running build for buildpack samples/hello-moon@0.0.1
Copying SBOM files
Creating SBOM files for legacy BOM
Listing processes
Timer: Builder ran for 2.304770458s and ended at 2023-08-30T17:16:45+02:00

As you can read from the output we have built the two buildpacks that we need in order to run our bash-script application. And now in our layers directory we should find other directories like the two from our buildpacks, a config and a sbom ones.

Export

We need to export the artifacts built by the builder, but first, we need to specify the path of the launcher that our image is going to run:

For AMD64 architectures

export CNB_LINUX_LAUNCHER_PATH=/<your-path>/lifecycle/out/linux-amd64/lifecycle/launcher

For ARM64 Architectures

export CNB_LINUX_LAUNCHER_PATH=/<your-path>/lifecycle/out/linux-arm64/lifecycle/launcher

And then let’s run the exporter:

$CNB_LIFECYCLE_PATH/exporter --log-level debug -launch-cache "./cache" -daemon -cache-dir "./cache" -analyzed "./layers/analyzed.toml" -group "./layers/group.toml" -layers="./layers" -app "./workspace" -launcher="$CNB_LINUX_LAUNCHER_PATH" -process-type="shell" apps/bash-script

The output should be something like this one:

Starting exporter...
Parsing inputs...
Ensuring privileges...
Executing command...
no project metadata found at path './layers/project-metadata.toml', project metadata will not be exported
no run metadata found at path '/cnb/run.toml'
Timer: Exporter started at 2023-08-30T17:28:46+02:00
Warning: Did not find SBOM lifecycle.sbom.cdx.json in /cnb/lifecycle
Warning: Did not find SBOM lifecycle.sbom.spdx.json in /cnb/lifecycle
Warning: Did not find SBOM lifecycle.sbom.syft.json in /cnb/lifecycle
Warning: Did not find SBOM launcher.sbom.cdx.json in /cnb/lifecycle
Warning: Did not find SBOM launcher.sbom.spdx.json in /cnb/lifecycle
Warning: Did not find SBOM launcher.sbom.syft.json in /cnb/lifecycle
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world
Processing buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon
Processing buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon
Found SBOM of type launch for at <REDACTED>/github.com/dlion/unpackingCNB/layers/sbom/launch
Reusing tarball for layer "buildpacksio/lifecycle:launch.sbom" with SHA: sha256:1ad83289f52becaf87159706f663d11b0871e0889d14b6f9fcff7938bb7d7a18
Adding layer 'buildpacksio/lifecycle:launch.sbom'
Layer 'buildpacksio/lifecycle:launch.sbom' SHA: sha256:1ad83289f52becaf87159706f663d11b0871e0889d14b6f9fcff7938bb7d7a18
Layer 'slice-1' SHA: sha256:fff5756b235cc98a7c89c9d059e26847773151a60f0ef269ef9721606ac16f8a
Adding 1/1 app layer(s)
Reusing tarball for layer "buildpacksio/lifecycle:launcher" with SHA: sha256:ef23d3a8631f2cc87faf9fadca65fcb1bcea64ab217c614f28d52c54bfffbbbd
Adding layer 'buildpacksio/lifecycle:launcher'
Layer 'buildpacksio/lifecycle:launcher' SHA: sha256:ef23d3a8631f2cc87faf9fadca65fcb1bcea64ab217c614f28d52c54bfffbbbd
Reusing tarball for layer "buildpacksio/lifecycle:config" with SHA: sha256:87f8a5a44bf99c09d22f06e7a0cede09dfefb3abd057071c150687bf01e0e645
Adding layer 'buildpacksio/lifecycle:config'
Layer 'buildpacksio/lifecycle:config' SHA: sha256:87f8a5a44bf99c09d22f06e7a0cede09dfefb3abd057071c150687bf01e0e645
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting CNB_LAYERS_DIR=<REDACTED>/github.com/dlion/unpackingCNB/layers
Setting CNB_APP_DIR=<REDACTED>/github.com/dlion/unpackingCNB/workspace
Setting CNB_PLATFORM_API=0.12
Setting CNB_DEPRECATION_MODE=quiet
Prepending /cnb/process and /cnb/lifecycle to PATH
Setting WORKDIR: '<REDACTED>/github.com/dlion/unpackingCNB/workspace'
no default process type
Setting ENTRYPOINT: '/cnb/lifecycle/launcher'
Timer: Saving apps/bash-script... started at 2023-08-30T17:28:46+02:00
*** Images (adaa8f978697):
      apps/bash-script*** Image ID: adaa8f9786976dabc271ab280daabf11d3fa55c667e36c0b7d05c9977a6c8365
Timer: Saving apps/bash-script... ran for 66.341292ms and ended at 2023-08-30T17:28:46+02:00
Timer: Exporter ran for 92.022541ms and ended at 2023-08-30T17:28:46+02:00
Timer: Cache started at 2023-08-30T17:28:46+02:00
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-world
Reading buildpack directory: <REDACTED>/github.com/dlion/unpackingCNB/layers/samples_hello-moon
Timer: Cache ran for 1.154042ms and ended at 2023-08-30T17:28:46+02:00
Warning: Failed to export cache: setting cache metadata: creating metadata file '<REDACTED>/github.com/dlion/unpackingCNB/cache/staging/io.buildpacks.lifecycle.cache.metadata': open <REDACTED>/github.com/dlion/unpackingCNB/cache/staging/io.buildpacks.lifecycle.cache.metadata: no such file or directory

Finally, our image has been exported correctly, which we can verify by running the docker images command.

Let’s run our image:

❯ docker run -it apps/bash-script ./app.sh
    |'-_ _-'|       ____          _  _      _                      _             _
    |   |   |      |  _ \        (_)| |    | |                    | |           (_)
     '-_|_-'       | |_) | _   _  _ | |  __| | _ __    __ _   ___ | | __ ___     _   ___
|'-_ _-'|'-_ _-'|  |  _ < | | | || || | / _` ||'_ \  / _\ | / __|| |/ // __|   | | / _ \
|   |   |   |   |  | |_) || |_| || || || (_| || |_) || (_| || (__ |   < \__ \ _ | || (_) |
 '-_|_-' '-_|_-'   |____/  \__,_||_||_| \__,_|| .__/  \__,_| \___||_|\_\|___/(_)|_| \___/
                                              | |
                                              |_|
Here are the contents of the current working directory:
.:
total 24
drwxr-xr-x 3 1000    1000 4096 Jan  1  1980 .
drwxr-xr-x 1  502 dialout 4096 Jan  1  1980 ..
-rw-r--r-- 1 1000    1000  692 Jan  1  1980 README.md
-rwxr-xr-x 1 1000    1000  736 Jan  1  1980 app.sh
drwxr-xr-x 3 1000    1000 4096 Jan  1  1980 bash-script-buildpack
-rw-r--r-- 1 1000    1000  202 Jan  1  1980 project.toml./bash-script-buildpack:
total 16
drwxr-xr-x 3 1000 1000 4096 Jan  1  1980 .
drwxr-xr-x 3 1000 1000 4096 Jan  1  1980 ..
drwxr-xr-x 2 1000 1000 4096 Jan  1  1980 bin
-rw-r--r-- 1 1000 1000  350 Jan  1  1980 buildpack.toml./bash-script-buildpack/bin:
total 16
drwxr-xr-x 2 1000 1000 4096 Jan  1  1980 .
drwxr-xr-x 3 1000 1000 4096 Jan  1  1980 ..
-rwxr-xr-x 1 1000 1000  330 Jan  1  1980 build
-rwxr-xr-x 1 1000 1000  242 Jan  1  1980 detect

This tutorial should have provided you with a comprehensive overview of the process of using Buildpacks to create container images. You are now ready to explore this technology further and adapt it to your application development and deployment needs. 🚀

Thank you

I’d like to mention Joe Kimmel who drafted the first version of this tutorial. Navdeep Pama, Natalie Arellano, Joe Kutner, and Juan Bustamante who reviewed and contributed to this tutorial, their insights and expertise were invaluable in bringing this tutorial to fruition.

If you have any questions or feedback, please don’t hesitate to reach out. Your engagement is what keeps our community vibrant and thriving. Thank you for joining us on this educational adventure, and we look forward to sharing more insights with you in the future. 👋🏻

Domenico Luciani, Sr. Software Engineer @ VMware

Cloud Native Buildpacks
Cloud Native
Buildpack

--

--

Domenico Luciani
Buildpacks

Written by Domenico Luciani

4 Followers
Writer for

Senior Software Engineer @ VMware | https://domenicoluciani.com

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams