Do you know Google Buildpacks? Compare it with the general version.

Stefanie Lai

Follow

Nov 16, 2020 · 6 min read

Since being included in the CNCF Sandbox Project at the end of 2019, Buildpacks¹ has become a shortcut for more people to get rid of the cumbersome Dockerfile. Indeed, Dockerfile sucks!

It is only built every time the project starts and then followed by ordinary templates, and many people would often forget how to write a reasonable Dockerfile. With the need to spend more time reviewing the Best Practice manual or search online, a twenty-minute job extends to two hours or more because of the unskillfulness or tiny mistakes.

Why We Need Buildpacks?

If you have already known or used Buildpacks, maybe this paragraph is not necessary for you, and please skip to the next section.

These problems will always bother you as long as you write Dockerfile.

• DRY principle. When we write similar Dockerfile repeatedly, DRY is broken.
• Triviality. Different languages and different environments require different Dockerfiles, which kills our pitiful hair.👩‍🦲
• Security. When you search for vulnerable Dockerfile in Google, you find so much information that makes you wonder if you get your Dockerfiles right, and even the titles may scare you away.

The most apparent advantages of using Buildpacks are:

• Improves development efficiency
• Reduces the learning cost of complex configurations and supports different languages
• Provides safe images, completing the upgrade and the patch of security vulnerabilities quickly without the user’s perception in the future
• Rebase, performs incremental updates on existing images swiftly

Convention over configuration (also known as coding by convention) is a software design paradigm used by software frameworks that attempts to decrease the number of decisions that a developer using the framework is required to make without necessarily losing flexibility. — From WIKI

Convention over configuration is a significant development principle in the Cloud era. All optimizations that lessen application developers’ burden on Infrastructure are worthwhile.

Buildpacks Internal

Skipping the process of installation, let’s look directly at how Buildpacks works. There are three steps in general.

• Detect the code, using the code to determine whether Buildpacks can run in the entire project. Through a set of tests, and when the first test hits, the current locale is fixed. For instance,Maven pom files,dependencies, main method entry, /target/*.jar can help to determine the Java environment. And if packages.json is found, it is a JS project.
• Build. Simulate Dockerfile and generate the required image by building different layers. And pack suggest-builders can help to view detailed information.
• Export. Upload the build image to local Docker, or remote docker hub, etc.

Buildpacks runs in the builder container. It performs related operations, relying on the scripts in the bin/ directory, /detect, /build, /release directories.

Use Buildpacks in Google Cloud

Google just released its internal Buildpacks opensource framework², which supports most mainstream programming languages, including Java, Go, Php, etc. As a developer who often deals with various Google products, I am very interested in trying and understanding it.

Compared with CNCF Buildpacks(CNB)

What are Google Buildpacks’ upsides?

• Build the code in Google-style and support GCP products like Google Cloud, Cloud Build, Cloud Run, Cloud Shell, Cloud Functions, etc.

Currently, CNB supports CI/CD tools, including CircleCI, Giblab ops, Tekton. If you want to integrate other CI tools, you need to customize, for which Google Buildpacks offers a solution.

• Easy Migrating, which is obvious. If you want to migrate to GCP or are already using GCP while still rely on the original Dockerfile, use Buildpacks directly and delete the previous Dockerfile. You can continue your work now without much effort, minimizing the disruption from the migration.
• In integrating Google Shell, you can simply use gcloud to build. Who does not love CLI?
• Faster. I am not confirmed, but the official document does mention it.

Practice

Try with a simple Go app example, triggering a build through Google Cloud Build and Google Shell, respectively. The first few steps are similar to official examples³

I have a local Go app and use it as an example. If you want to try, you can follow the instructions from the Github page.

• Build, initialize the Buildpacks in the project.

pack build --builder gcr.io/buildpacks/builder:v1 go-web

• Run

docker run --rm -p 9099:9099 go-web

• Deploy through Google Cloud Run

gcloud builds submit --pack \ image=gcr.io/${myGCPProject}/${imageName}
gcloud run deploy --image gcr.io/${myGCPProject}/${imageName}

• If you know skaffold⁴,You can deploy the image through skaffold as below 👇

steps:
- name: 'gcr.io/k8s-skaffold/pack'
  entrypoint: 'pack'
  args: ['build', '--builder=gcr.io/buildpacks/builder', '--publish', 'gcr.io/${myGCPProject}/${imageName}:${tag}']

I don’t use skaffold myself, so I simply put the shell commands into my Git post-update⁵ hooks. Then I can trigger the Google Cloud Run immediately after a successful push.

Principles

I am curious about open source projects, not satisfied with just using them. After reading the source code of Google Buildpacks, I have some reflections.

Google Buildpacks seamlessly integrate CNB, encapsulating CNB’s layer and libcnb in the code to define some common dependencies and image layers. And it also uses the pack tool to execute related commands.

So you’ll easily find that many commands in the code are implemented through a similar logic detect -> build function as above.

For example, let’s look at the implementation of code in thegolang/build package. There are two functions: detectFn and buildFn .

gcp.Main(detectFn, buildFn)

• detectFn is simple, it finds the *.go files.

func detectFn(ctx *gcp.Context) (gcp.DetectResult, error) {
 if !ctx.HasAtLeastOne("*.go") {
  return gcp.OptOut("no .go files found"), nil
 }
 return gcp.OptIn("found .go files"), nil
}

• The build method assembles each layer of the image with code and finally syncs the result to a directory.

cl := ctx.Layer("gocache", gcp.BuildLayer, gcp.LaunchLayerIfDevMode) // find possible cache
bl := ctx.Layer("bin", gcp.LaunchLayer)  // prepare for compiled binary files
bl.LaunchEnvironment.PrependPath("PATH", bl.Path)// compile go files with various env variables
bld := []string{"go", "build"}
bld = append(bld, goBuildFlags()...)
bld = append(bld, "-o", outBin)
bld = append(bld, buildable)
// BuildDirEnv should only be set by App Engine buildpacks.
workdir := os.Getenv(golang.BuildDirEnv)
if workdir == "" {
workdir = ctx.ApplicationRoot()
}
ctx.Exec(bld, gcp.WithEnv("GOCACHE="+cl.Path), gcp.WithWorkDir(workdir), gcp.WithMessageProducer(printTipsAndKeepStderrTail(ctx)), gcp.WithUserAttribution)// Configure the entrypoint and metadata for dev mode.
devmode.AddFileWatcherProcess(ctx, devmode.Config{
  BuildCmd: bld,
  RunCmd:   []string{outBin},
  Ext:      devmode.GoWatchedExtensions,
})devmode.AddSyncMetadata(ctx, devmode.GoSyncRules)

The build commands of some other languages are in the same way. For example, Java will distinguish gradle and maven via build tools and then use gradle and mvn, respectively, to compile related files.

Another prominent feature of Google Buildpacks is the integration of GCP products. Under tools/cloudbuild, there is a set of related shell scripts that achieve their goals by tools like docker and pack, defining the dependencies and commands Cloud Build requires through YAML.

Security and version are reflected in the build directory, which encapsulates the different Buildpacks versions that various languages and different versions depend on, as well as the configuration of templates, such as Golang’s supportive from Go111 to Go115.

From the source code, Google Buildpacks has the same concept and process as Buildpacks, but is not closely related to CNB actually and is relatively lightweight. And Google Buildpacks added a lot of general env variables to help the entire process execute faster, such as directly defining the variable of the entry GOOGLE_ENTRYPOINT.

To sum up

This article is a brief note when I am reading and practicing Google Buildpacks. Tools like Buildpacks are being used in more and more teams and companies for the convenience they bring. Understanding their usages and certain internal principles is a must-have skill for developers in the future.

Perhaps some content of this article may be inaccurate. I will try to perfect my later notes during my continuous practice with Google Buildpacks.

Thanks for reading!

Reference

[1] Buildpacks

[2] Google Buildpacks

[3] Quick start

[4] Skaffold

[5] Git post-update hooks