Dockerless container builds using Buildah
This article is a hands-on-try-this-new-thingy-out from a seasoned Docker user, but a complete newb on Buildah. Saw a quick demo of it on KubeCon earlier this year, but never got around to try it myself.
Here is a RedHat blog post introducing Buildah that is worth reading before reading this article.
I will show you exactly how I approached this new thing, and not hide any mistakes along the way.
Kill your darlings
I am a big fan of Docker and what they have done and I have invested a lot of my time the last three years in Docker technologies. However I see a lot of use cases for not having to push the full Docker engine onto peoples laptops, even if Docker for Mac and Windows are great. (If you are on the right version of your OS and have a good enough hardware). But also for CI using Jenkins it can often be a problem for various reasons.
More importantly is actually the promise of reducing the security attack vector in the final image, even if multi-stage builds can do a fairly good job these days.
Reading about Buildah they have seen something similar and actually created something that may solve it in a good way.
Getting started
OK, I admit, I will start this whole thing by cheating.
I do have Docker for Mac on my laptop and I don’t like installing new stuff onto it unless I have to. I do like to have portable stuff that would work on Windows or Linux as well, even if Homebrew is awesome.
RHEL as base image
So, lets grab the latest RHEL 7.5 as the base image since they a supported package for buildah.
docker run — rm -it registry.access.redhat.com/rhel7:7.5Bummer, immediately ran into:
This system is not registered with an entitlement server. You can use subscription-manager to register.
There are no enabled repos.I haven’t really been using RHEL for ages, so took a while to figure out how to proceed, but instead of actually switching to another distribution, I managed to figure it out, not by Google but via the terminal.
subscription-manager register — auto-attachBut I should have figured, need to register, so time to switch to another dist for now. (I later followed up on this one and we do have a nice Red Hat Satellite infrastructure but don’t work from unregistered hosts. It would work if I had tried to built it from within OpenShift.)
Debian as base image
Lets try debian:latest (yes, I normally try to be explicit but the hour is late).
docker run — rm -it debian:latestAnd from within the image:
apt-get update
apt-get install -y gpg
gpg — recv-keys 0x018BA5AD9DF57A4448F0E6CF8BECF1637AD8C79D
gpg — export 0x018BA5AD9DF57A4448F0E6CF8BECF1637AD8C79D >> /usr/share/keyrings/projectatomic-ppa.gpg
echo ‘deb [signed-by=/usr/share/keyrings/projectatomic-ppa.gpg] http://ppa.launchpad.net/projectatomic/ppa/ubuntu zesty main’ > /etc/apt/sources.list.d/projectatomic-ppa.listThis doesn’t work for some reason:
gpg — recv-keys 0x018BA5AD9DF57A4448F0E6CF8BECF1637AD8C79D
gpg: keyserver receive failed: Cannot assign requested addressShoot me, now the bed seems to be calling, but I refuse, lets go for CentOS!
CentOS as base image
docker run — rm -it centos:7.5.1804From within:
yum install -y buildahSo, trying to avoid installing Docker, I instead ended up with LOTS of rpm dependencies instead.
Installed:
buildah.x86_64 0:0.16–3.git532e267.el7Dependency Installed:
..
MANY!!
..
And for all that trouble, I got the 2nd latest version — 0.16
buildah — version
buildah version 0.16 (image-spec 1.0.0, runtime-spec 1.0.0)Looking at the Release notes, it seems like I really want 1.0 instead.
Fedora as base image
docker run — rm -it — entrypoint bash fedoraAnd from within:
yum install -y buildah
buildah -version
buildah version 1.0 (image-spec 1.0.0, runtime-spec 1.0.0)Interlude — How does Fedora, CentOS and RHEL relate to each other?
Yes, how do they RHELate?
I completely forgot the relationship between these three, here is a nice picture and short summary in this article.
I have been using Debian and Alpine mostly for building container base images and CoreOS Container Linux for container hosts.
Success finally, or?
So, the bad thing first, I now feel like a complete n0ob, but the good thing is that I am now on 1.0 and debating if this bad experience was really worth it?
But if the final container image turns out the way I hope, it might be, so lets pause before we continue.
A new day and now it starts to get really fun, not!
containerid=$(buildah from scratch)
ERRO[0000] ‘overlay’ is not supported over overlayfs
ERRO[0000] ‘overlay’ is not supported over overlayfs
‘overlay’ is not supported over overlayfs: backing file system is unsupported for this graph driver
‘overlay’ is not supported over overlayfs: backing file system is unsupported for this graph driverCome on, what now!?
So, there are some really long threads on GitHub for issues related to this, here is one.
Interlude — Podman
Seems to be an alternative to Docker cli that is intended to be used with CRI-O (OCI-based implementation of Kubernetes Container Runtime Interface).
Read about it here.
Since I am on a Mac I don’t think it is for me as of now.
Back in the saddle
Trying to do some workaround:
mkdir -pv var/lib/containers
docker run — rm -it — privileged -v /work/var/lib/containers:/var/lib/containers:rw — entrypoint bash fedora#yum install -y buildah
..
LOTS of packages installed
..containerid=$(buildah from scratch)
scratchmnt=$(buildah mount $containerid)
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/1bfb46968b4e9a6d0964582628b4bfc4983e766559d2b11acb2d6afe4807def2/merged: invalid argument
error mounting “working-container-2” container “working-container-2”: error creating overlay mount to /var/lib/containers/storage/overlay/1bfb46968b4e9a6d0964582628b4bfc4983e766559d2b11acb2d6afe4807def2/merged: invalid argument
So, this is when I give up this time. Buildah may be nice if you are on a RHEL or Fedora host, but trying to run it from within Docker on a Mac is just sheer pain. Also I do miss out the promise of Dockerless if I try to run it this way. I need to reinstall my private laptop that is currently running ElementaryOS, guess it would work using a virtual machine as well or actually run a real RHEL on a public cloud. I am almost tempted to try out OpenShift.io as well but I will see if there are alternatives to Buildah instead.
Enter img
So, turns out that the Atomic project where Buildah resides were nice enough people to write an article about some of the alternative tools that exists.
Since one of them originates from Jessie Frazelle, the choice was easy, since she is makes things work in containers(if you haven’t already, check out her repo on GitHub for desktop Docker images.)
She introduces img in this great blog post, that is so awesome in so many way I say this: “Read it!”.
Interlude — suckless.org
Just read about their philosophy and weep, found it via Genuine Tools which is the organization(?) where img resides.
So lets take img for a spin
docker run — rm -it \
— name img \
— volume $(pwd):/home/user/src:ro \ # for the build context and dockerfile, can be read-only since we won’t modify it
— workdir /home/user/src \ # set the builder working directory
— volume “${HOME}/.docker:/root/.docker:ro” \ # for credentials to push to docker hub or a registry
— cap-add SETGID \ # so we can set groups
— cap-add SETUID \ # so we can map users in a user namespace
— security-opt apparmor=unconfined \ # turn off apparmor so we can mount unprivileged
— security-opt seccomp=unconfined \ # turn off seccomp because it blocks new user namespaces
r.j3ss.co/imgI put it in my ~/bin dir as a shell script instead, and yes, you need to clean up it a bit.
docker run — rm -it \
— name img \
— volume $(pwd):/home/user/src:ro \
— workdir /home/user/src \
— volume “${HOME}/.docker:/root/.docker:ro” \
— cap-add SETGID \
— cap-add SETUID \
— security-opt apparmor=unconfined \
— security-opt seccomp=unconfined \
r.j3ss.co/img $*img -h
Unable to find image ‘r.j3ss.co/img:latest’ locally
latest: Pulling from img
ff3a5c916c92: Already exists
510ec5dca3d7: Pull complete
f9b0e6abab23: Pull complete
741b5229554c: Pull complete
3adb717baa46: Pull complete
Digest: sha256:eba0c1b6020277fe2a519a6d9c9b186fc99f11abc1b299f969fb2bce763dacf5
Status: Downloaded newer image for r.j3ss.co/img:latest
Usage: img <command>Commands:build Build an image from a Dockerfile.
du Show image disk usage.
ls List images and digests.
login Log in to a Docker registry.
prune Prune and clean up the build cache.
pull Pull an image or a repository from a registry.
push Push an image or a repository to a registry.
rm Remove one or more images.
save Save an image to a tar archive (streamed to STDOUT by default).
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE.
unpack Unpack an image to a rootfs directory.
version Show the version information.
Finally, we are in business. Simple and straight forward, even if one of my main reasons for using Buildah was to be able to do things without a Dockerfile and also using the support of being able to mount volumes into the build which is not possible for Docker builds. But any success at this point I celebrate.
Then I realize, should have tested it first before proclaiming victory.
img build -t “myimage” .
newuidmap: write to uid_map failed: Operation not permitted
nsenter: failed to use newuidmap: Invalid argument
nsenter: failed to sync with parent: SYNC_USERMAP_ACK: got 255: Invalid argumentProbably need the changes Jessie made for runc and Docker as mentioned in the blog post. Updated to latest Edge for Docker for Mac didn’t help.
What now?!
Pondering upon Skaffold, vaguely remember hearing something about dockerless builder there, it turns out Jessie has been busy there as well, looking at this issue.
Turns out that Akihiro Suda was not late on picking up on this issue and has made something truly awesome that I did not know of, see details about his ideas for integrating his work with skaffold here.
According to it, it should work with RedHat OpenShift S2I as well.
So after DockerCon we should see a PR from Akihiro, looking forward to it!
Skaffold with a non-Docker builder, that does unprivileged builds in K8s might be part of the answer to this resent warning — Will Kubernetes collapse under its own weight?
Summary
I failed miserably with my original intent I had in mind when creating this article. Since it is an honest go with the flow, you have seen my struggles and how I bounce back and forth like a pinball ball.
There is a lot of talk about failures, and failing fast, but I prefer not to fail actually. I have embraced the NLP saying (at least this is how I have remembered it):
There is no such thing as failure, only feedback.
I will probably check out Buildah sometime soon again, but next on my list now is testing out Skaffold and reading more about CBI that was a new and very interesting finding, totally worth all the trouble finding out about it! (I hope…)
