A walk through Kubernetes build process

6 min readNov 2, 2017

Building Kubernetes binaries by hand can be a difficult operation and you will probably end with small differences for every new try. Luckily for us, the Kubernetes community provides us with a set of tools to make things easier and we will be able to have reproducible builds, every run, even compared to the official ones. This is possible because the official releases are built using Docker containers which will be a requirement to build k8s (usually a local installation, but could also be a remote one).

Before going deep into the build process, worth noticing that thekube-buildDocker image is built based on build/build-image/Dockerfile and we will have 3 different containers during the build process using that image: the “build” and “rsync” containers are used for the build action and to transfer data between the container and the host; both will be deleted after every run. The “data” container will store everything necessary to support incremental builds so it will not be destroyed after each use. All this images and data will be stored in the build/directory, which will be also used for testing purposes (out of the scope of this post).

There are some works on the way to use Bazel (the open source release of the Blaze project that Google use internally), but for now, we will be using the all-mighty make. And as always, everything starts with the Makefile.

Makefile

A simple make help will show us all the options available, from building targets to testing, using bazel, verifying… (full output at make help). We will be focusing on make quick-release, but as you will see almost everything is related and we will be, at the end, decomposing the make all command.

define RELEASE_SKIP_TESTS_HELP_INFO
# Build a release, but skip tests
#
# Args:
#   KUBE_RELEASE_RUN_TESTS: Whether to run tests. Set to 'y' to run tests anyways.
#   KUBE_FASTBUILD: Whether to cross-compile for other architectures. Set to 'true' to do so.
#
# Example:
#   make release-skip-tests
#   make quick-release
endef
.PHONY: release-skip-tests quick-release
ifeq ($(PRINT_HELP),y)
release-skip-tests quick-release:
    @echo "$$RELEASE_SKIP_TESTS_HELP_INFO"
else
release-skip-tests quick-release: KUBE_RELEASE_RUN_TESTS = n
release-skip-tests quick-release: KUBE_FASTBUILD = true
release-skip-tests quick-release:
    build/release.sh
endif

As we can see from the code snippet above, the quick-release target (aka release-skip-tests) is the same as the release one, with two changes: We will not run any tests on it (KUBE_RELEASE_RUN_TESTS = n), and we will not compile the binaries for other architectures (KUBE_FASTBUILD = true); only linux/amd64 binaries will be created.

build/release.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
source "${KUBE_ROOT}/build/common.sh"
source "${KUBE_ROOT}/build/lib/release.sh"KUBE_RELEASE_RUN_TESTS=${KUBE_RELEASE_RUN_TESTS-y}kube::build::verify_prereqs
kube::build::build_image
kube::build::run_build_command make crossif [[ $KUBE_RELEASE_RUN_TESTS =~ ^[yY]$ ]]; then
  kube::build::run_build_command make test
  kube::build::run_build_command make test-integration
fikube::build::copy_outputkube::release::package_tarballs

All the kube::build functions are located in the file ${KUBE_ROOT}/build/common.sh. verify_prereqs and build_image will check that we have a running docker and will build the docker image (using build/build-image/Dockerfile) that we need for our purpose.

After running kube::build::run_build_command make cross, we will again ignore any tests, and will copy the results from the container to our hosts using the rsync container (kube::build::copy_output) and pack the binary into the _output dir (kube::release::package_tarballs).

build/common.sh

# Run a command in the kube-build image. This assumes that the image has
# already been built.
function kube::build::run_build_command() {
  kube::log::status "Running build command..."
  kube::build::run_build_command_ex "${KUBE_BUILD_CONTAINER_NAME}" -- "$@"
}# Run a command in the kube-build image. This assumes that the image has
# already been built.
#
# Arguments are in the form of
# <container name> <extra docker args> -- <command>
function kube::build::run_build_command_ex() {
  [[ $# != 0 ]] || { echo "Invalid input - please specify a container name." >&2; return 4; }
  local container_name="${1}"
  shift  local -a docker_run_opts=(
    "--name=${container_name}"
    "--user=$(id -u):$(id -g)"
    "--hostname=${HOSTNAME}"
    "${DOCKER_MOUNT_ARGS[@]}"
  )  local detach=false  [[ $# != 0 ]] || { echo "Invalid input - please specify docker arguments followed by --." >&2; return 4; }
  # Everything before "--" is an arg to docker
  until [ -z "${1-}" ] ;do
    if [[ "$1"=="--" ]];then
      shift
      break
    fi
    docker_run_opts+=("$1")
    if [[ "$1"=="-d"||"$1"=="--detach" ]] ;then
      detach=true
    fi
    shift
  done  # Everything after "--" is the command to run
  [[ $# != 0 ]] || { echo "Invalid input - please specify a command to run." >&2; return 4; }
  local -a cmd=()
  until [ -z "${1-}" ] ;do
    cmd+=("$1")
    shift
  done
 
  docker_run_opts+=(
    --env "KUBE_FASTBUILD=${KUBE_FASTBUILD:-false}"
    --env "KUBE_BUILDER_OS=${OSTYPE:-notdetected}"
    --env "KUBE_VERBOSE=${KUBE_VERBOSE}"
    --env "GOFLAGS=${GOFLAGS:-}"
    --env "GOLDFLAGS=${GOLDFLAGS:-}"
    --env "GOGCFLAGS=${GOGCFLAGS:-}"
  )  # If we have stdin we can run interactive. This allows things like 'shell.sh'
  # to work. However, if we run this way and don't have stdin, then it ends up
  # running in a daemon-ish mode. So if we don't have a stdin, we explicitly
  # attach stderr/stdout but don't bother asking for a tty.  if [[ -t 0 ]];then
    docker_run_opts+=(--interactive --tty)
  elif [[ "${detach}"==false ]];then
    docker_run_opts+=(--attach=stdout --attach=stderr)
  fi  local -ra docker_cmd=(
    "${DOCKER[@]}" run "${docker_run_opts[@]}""${KUBE_BUILD_IMAGE}")  # Clean up container from any previous run
  kube::build::destroy_container "${container_name}"  "${docker_cmd[@]}""${cmd[@]}"  if [[ "${detach}"==false ]];then
    kube::build::destroy_container "${container_name}"
  fi
}

Inside kube::build::run_build_command, we will call kube::build::run_build_command_ex "${KUBE_BUILD_CONTAINER_NAME}" -- "$@" where KUBE_BUILD_CONTAINER_NAME is the name of the container we got in the previous kube::build::verify_prereqs call and “$@” will be “make cross” in this case.

Then, we will get all the extra options that will be passed to the docker image (docker_run_opts) that will form the final docker run command, docker_cmd ( "${DOCKER[@]}" run "${docker_run_opts[@]}""${KUBE_BUILD_IMAGE}")

And with that, and the cmd to run inside the container (make cross), we have the final command we will execute: "${docker_cmd[@]}""${cmd[@]}"

From now on, everything will be executed inside the kube-build container. And we are back to the main Makefile (there will be an exact copy of our kubernetes root folder inside the container), but this time, we will just execute the cross target.

make cross (inside the kube-build container)

define CROSS_HELP_INFO
# Cross-compile for all platforms
# Use the 'cross-in-a-container' target to cross build when already in
# a container vs. creating a new container to build from (build-image)
# Useful for running in GCB.
#
# Example:
#   make cross
#   make cross-in-a-container
endef
.PHONY: cross cross-in-a-container
ifeq ($(PRINT_HELP),y)
cross cross-in-a-container:
    @echo "$$CROSS_HELP_INFO"
else
cross:
    hack/make-rules/cross.sh
cross-in-a-container: KUBE_OUTPUT_SUBPATH = $(OUT_DIR)/dockerized
cross-in-a-container:
ifeq (,$(wildcard /.dockerenv))
    @echo -e "\nThe 'cross-in-a-container' target can only be used from within a docker container.\n"
else
    hack/make-rules/cross.sh
endif
endif

This is pretty straightforward, just a call to hack/make-rules/cross.sh

hack/make-rules/cross.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
source "${KUBE_ROOT}/hack/lib/init.sh"# NOTE: Using "${array[*]}" here is correct. [@] becomes distinct words (in
# bash parlance).make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"make all WHAT="${KUBE_TEST_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_SERVER_PLATFORMS[*]}"

In hack/lib/init.sh we will source the file hack/lib/golang.sh which will be responsible to set all the KUBE_*_TARGETS and KUBE_*_PLATFORMS.

...
elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then
  readonly KUBE_SERVER_PLATFORMS=(linux/amd64)
  readonly KUBE_NODE_PLATFORMS=(linux/amd64)
...

If you remember from our first call to make quick-release, it set KUBE_FASTBUILD = true so we will just compile the linux/amd64 binaries.

The targets (KUBE_SERVER_TARGETS) will also be specified by hack/lib/golang.sh

# The set of server targets that we are only building for Linux
# If you update this list, please also update build/BUILD.
kube::golang::server_targets() {
  local targets=(
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    vendor/k8s.io/kube-aggregator
    vendor/k8s.io/apiextensions-apiserver
    plugin/cmd/kube-scheduler
  )
  echo"${targets[@]}"
}

And with all that set we call, again, the Makefile with the almighty all target. (Remember that we are now inside the kube-build container)

make all

define ALL_HELP_INFO
# Build code.
#
# Args:
#   WHAT: Directory names to build. If any of these directories has a 'main'
#     package, the build will produce executable files under $(OUT_DIR)/go/bin.
#     If not specified, "everything" will be built.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags passed to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make
#   make all
#   make all WHAT=cmd/kubelet GOFLAGS=-v
#   make all GOGCFLAGS="-N -l"
#     Note: Use the -N -l options to disable compiler optimizations an inlining.
#           Using these build options allows you to subsequently use source
#           debugging tools like delve.
endef
.PHONY: all
ifeq ($(PRINT_HELP),y)
all:
    @echo "$$ALL_HELP_INFO"
else
all: generated_files
    hack/make-rules/build.sh $(WHAT)
endif

Again, a simple make entry with the help and a call to hack/make-rules/build.sh with the WHAT variable being the KUBE_SERVER_TARGETS provided by hack/lib/golang.sh

hack/make-rules/build.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
KUBE_VERBOSE="${KUBE_VERBOSE:-1}"
source "${KUBE_ROOT}/hack/lib/init.sh"kube::golang::build_binaries "$@"
kube::golang::place_bins

And from now on, in kube::golang::build_binaries, is where the Go magic happens. It sets up the environment, some Go flags, builds the kube toolchain (github.com/jteeuwen/go-bindata/go-bindata and hack/cmd/teststale) and finally builds the targets.

Conclusions

As you can see, although it looks complicated at first, everything can be reduced to a simple call to make all with a bunch of environment variables to get the output we want. Making use of a common docker image with the same set of tools across the whole community will also help us to reduce any differences that can be a real nightmare if we need to debug any faulty service.