Avengers of Container World — Episode 2: Buildah & Skopeo Hands On

Published in

The Startup

9 min readJan 26, 2021

In the last Episode (Episode 1: Podman Hands on), we got Podman working on CentOS/VirtualBox. We also pulled tomcat image and got it running. In this episode we will explore the advantages of Buildah and Skopeo and build a complete custom image with our sample web application.

Why Buildah?

Docker provided a very sophisticated configuration file based provisioning with Dockerfile & Docker compose. It provided a simple configuration file, that docker daemon would use to build custom images, configure and provision the container. Docker Daemon has the functionality to build, pull, push, run and manage containers.

So why do we need Buildah? or Skopeo?

As mentioned in Episode 1 Single point of failure, root access, memory footprint, performance are some of the shortcomings of Docker approach. The next generation OCI compliant tools like Podman, Buildah, Skopeo along with CRI-O provide a more modular, daemonless approach, to keep it simple.

Why is Daemonless & Rootless a big deal for building images & running containers?

Docker Daemon runs with root access. Dockerfile has the configuration on how to build an image, which means to perform docker builds, we need root access, which is too permissive in security/compliance sensitive enterprises. Docker build involves pulling images, installing packages, building artifacts etc, which might require root access.

Podman & Buildah use user namespaces to overcome this root access problem. Here is how it works

User Namespaces provides the isolation of Linux processes (refer here for more details about namespaces). User namespaces isolate user IDs, group IDs, root directory, keys etc. Because of this process isolation, a process can have a full privilege within the namespace, but have different privilege outside the namespace.
What this means, is that the container will have complete privileges within the container, but will not have access in the host system, providing complete security.

This is a big deal from the security point of view, and also avoids a lot of vulnerabilities on the host system. The Host system is completely protected by the user and group privileges, which does not affect the container user & group privileges and vice versa, and the container will have access to the host resources, as per the userid/groupid prevelatges of the user, who performs the build & run

Another cool feature of buildah is, it completely supports Dockerfile builds buildah bud (bud=build using dockerfile) & provides building from the scratch step by step using buildah from scratch. buildah from <imagename> helps building new images from the existing image.

Buildah does not need a container runtime daemon, it uses runx.

Buildah from scratch helps in building the image step by step, and test it, before commiting and creating the image. This also helps cleanup temporary files, and create the most optimum image, while Docker internally does this with every command in Dockerfile, Buildah provides better flexibility here.

Registries

Podman, Buildah & Skopeo use registries to search, pull and push the images. typically we will find the registry configurations in /etc/containers/regitries.conf. Here is a typical snapshot of how it looks

The red circle highlights the list of all registries to search for, in the order of sequence, it can be redhat registry, quay.io, docker.io or any other custom registries. We can include custom user registries or enterprise registries, in this file.

The user who is trying to pull or push the images from the respective registries should have access to those registries, and typically we use podman login or buildah login, before we can access the registries.

Now let's jump in and create a custom image.

Please refer to chapter 1 for steps to install VirtualBox, Vagrant, CentOS. We are going to use the same environment to setup Buildah and Skopeo. Please note that I am creating this environment on Virtualmachine (so that I don’t mess up my local macos docker environment). If you want to do directly on your machine, you can ignore the vagrant commands.

Lets build a simple nodejs application running on node:10 image from docker.io . We will use buildah to build the container, test it, and then commit it to create a image out of it. We will then use podman to run that image and test. Let's get started!!!

Buildah

Install Buildah

sudo yum -y install buildah
# These are optional steps to install 'Untested' updates from Kubic project
sudo dnf -y module disable container-tools
sudo dnf -y install 'dnf-command(copr)'
sudo dnf -y copr enable rhcontainerbot/container-selinux
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo
sudo dnf -y install buildah

to check if the buildah is installed, try

buildah version

Now that we know buildah is installed, let’s use builah from command to start building our message. buildah from is exactly like the FROM that we give in Dockerfile. We have to provide the based image on top of which we want to build our container. buildah from has various options.

buildah from scratch — As the name suggests, We can build a container from scratch, using this command.
buildah from --pull <image> — This is used to build a container from an existing image. Buildah searches for that image name on the configured repositories, in the order we have specified in registries.conf file, as mentioned above. We can also specify the URL of the repository from where we want to pull.
buildah bud: buildah can also build the container from an existing Dockerfile, using bud (build with docker) command.

So lets get starting building our image

Creating a container with nodejs base

Let’s build a container with base node:10 image from docker.io, we will be using buildah from image command.

buildah from --pull --name “myNodeJSContainer” node:10

As you can see in the screenshot, buildah interactively provides the options of imaged we have, from the various repositories we have configured in repositories.conf.

Lets select docker.io standard image. --name argument is used to name the container here.

let's check if the container is created using buildah containers

Now lets create a directory in the container for deploying our app, and make it the default working directory.

buildah run myNodeJSContainer mkdir -p /usr/src/mynodeapp
buildah config --workingdir /usr/src/mynodeapp myNodeJSContainer

Now lets exit the vagrant shell and create a simple nodejs app, on our local machine.

npm init
npm install express --save

here is how the package.json looks

{
   "name": "buildah",
   "version": "1.0.0",
   "description": "",
   "main": "index.js",
   "scripts": {
       "test": "echo \"Error: no test specified\" && exit 1"
    },
    "author": "a b vijay kumar",
    "license": "ISC",
    "dependencies": {
        "express": "^4.17.1"
    }
}

and here a sample application (index.js)

const express = require('express')
const app = express()
const port = 3000app.get('/', (req, res) => {
  res.send('Hello World of buildah and podman!')
})app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`)
})

Now lets copy our sample application files to vagrant shell, The easier option to upload a file is using vagrant upload index.js

vagrant upload index.js
vagrant upload package.json

There is an alternate way to upload using SCP. Before we do that, get out of the vagrant shell to the host. execute ssh-config . I just thought it will be useful to learn this.

vagrant ssh-config

look for the location where IndentityFile is stored. Here is what I got when I executed this

Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/vijaykumarab/AB-Home/blogs/buildah/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

now lets copy index.js and package.json to the vagrant shell using scp (secure copy). here are the commands (please note that you have to specify the correct identity file path, based on what u see in ssh-config.

scp -P 2222 -i ./.vagrant/machines/default/virtualbox/private_key index.js vagrant@127.0.0.1:~
index.jsscp -P 2222 -i ./.vagrant/machines/default/virtualbox/private_key package.json vagrant@127.0.0.1:~
package.json

Now let's go back to the vagrant shell using vagrant ssh. You should find the files there. in the vagrant home directory. Lets now upload this to the container.

Lets install and run the app. npm install will install the nodejs app, by downloading all the dependent files.

buildah run --net host myNodeJSContainer npm installnpm WARN buildah@1.0.0 No description
npm WARN buildah@1.0.0 No repository field.added 50 packages from 37 contributors and audited 50 packages in 3.581s
found 0 vulnerabilities

Let’s configure the entry point for the container.

buildah config --entrypoint ‘[“node”, “index.js”]’ myNodeJSContainer

Let’s test the application, lets login to the container shell (please note that we are already in vagrant shell, and now we are getting into the container shell. If you don’t run buildah on a vagrant VM, you don’t have worry about the vagrant commands above).

buildah run — tty myNodeJSContainer /bin/bash

The app can be tested by running

node index.js

and a curl to check if the app is running fine.

Once we are happy that the app is running, now we need to commit the container image. So far the image is only being constructed, it will not still show in the local repository. for example if you run podman images

you will not see the new container image that we are building, we only see the node image (which got downloaded as we built our container using buildah from node). Now lets commit the container as an image.

Build Image from Container

buildah commit myNodeJSContainer mynodejsapp

I named my image as mynodejsapp, myNodeJSContainer was just a working container name.

as you can see now the whole image is built and committed to local repository. Now lets see if we can see the image using podman images

As you can see, now we have our image. We can use now use podman to push the image to the dockerhub or any other registries. Lets test the container now using podman

Run and test the container using podman

podman run -dt -p 8080:3000/tcp mynodejsapp

This runs the container and starts the container, with the entry point config that we had created using buildah. Lets now see if the container is running using podman ps

curl http://localhost:8080
Hello World of buildah and podman!

We can see that the container is running and we are able to get the response using curl… Now that we are happy with this, we can use podman push to push it to the desired registry.

That's it!!! the best part of using buildah is that we did not need any deamon, or long Dockerfile (if we have it we can always use it with bud and still run it, but as a developer/devops engineer, buildah provides more command level flexibility and we should be able to run this using a shell script. We have much more control over the commands we want to run, rather than send a batch of commands to docker deamon).

Now let take a quick look at why do we need Skopeo

Skopeo

Skopeo is a powerful tool to inspect, copy, delete images from repositories. Synchronizing the images between repositories (both internal and external) is one of the most powerful feaures of skopeo.

Installing Skopeo is straightforward on our centos (for other OS, refer to https://github.com/containers/skopeo/blob/master/install.md)

sudo dnf -y install skopeo

Once installed, we can inspect the meta data about the image that used to build our container

skopeo inspect docker://docker.io/library/node:10

you will see a long output with all the information about the image.

Skopeo can be used to copy images between repositories using skopeo copy. This includes Quay, Docker Hub, OpenShift, GCR, Artifactory etc. Skopeo also provides a way to synchronize images across two repositories/registries using skopeo sync . Skopeo can also be used to delete the images using skopeo delete .

Skopeo is very powerful for GitOps pipelines where the images need to be managed across the pipeline and for Continuous delivery, where we might be using different repositories/registries.

That’s it for now, You will realize the value of podman, buildah and skopeo more, as we get into managing large amount of containers and images across various repositories. These tools provide a very light weight way to manage the images and containers, which can easily be coded into any GitOps pipeline. I have plans to do a blog on Modern GitOps soon…

See you soon…ttyl :-)