One of the fundamental features of container implementations today is the ability to run containers of differing linux distros on the same host machine. It’s not uncommon, for example, to install Docker on an Ubuntu host and to then start a bunch of containers on that host using BusyBox, CentOS, or any other distro you like the look of.
In this article we will will take a look at what makes this possible - namely a combination of the Mount namespace and the pivot_root
system call. Let's start by reviewing the Mount namespace implementation in ns-process
as it currently stands. If you’ve not been following along with this series so far, be sure to check out the previous article(s) first.
💁 The following has been tested on Ubuntu 16.04 Xenial with Go 1.7.1
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 3.0$ go build
$ ./ns-process>> namespace setup code goes here <<-[ns-process]- # cat /proc/mounts
/dev/sda1 / ext4 rw,relatime,data=ordered 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
# ...
There are a number of mounts already listed in the /proc/mounts
file. This may be little surprising given that we’re requesting a new Mount namespace (via the CLONE_NEWNS
flag) and have yet to do any explicit Mount namespace setup.
This doesn’t feel very container-like. Our namespaced process should know as little as possible about the host it’s running on, and certainly shouldn’t be able to see a list of all the host’s mounts. So why’s this happening? Fortunately, an explanation can be found within the mount_namespaces(7) man page.
“When a process creates a new mount namespace using clone(2) or unshare(2) with the CLONE_NEWNS flag, the mount point list for the new namespace is a copy of the caller’s mount point list.”
It seems that this is actually intended behaviour, and it explains why /proc/mounts
is already populated as soon as our namespaced process starts. With this in mind the question now becomes, “What do we do about it?”. We need some way of clearing the host’s mounts from the new Mount namespace in order to keep them secure and away from prying eyes - we need to pivot_root
.
🔄 pivot_root
pivot_root
allows you to set a new root filesystem for the calling process. I.e. it allows you to change what /
is. It does this by mounting the current root filesystem somewhere else while simultaneously mounting some new root filesystem on /
. Once the previous root has been moved, it is then possible to umount it. Thus we have a mechanism for 'clearing' the hosts's mounts from inside a new Mount namespace - we simply pivot away and then umount them!
This is what allows the aforementioned Ubuntu host machine to run a CentOS container. As long as the Ubuntu host has a copy of a CentOS filesystem on disk, we can create a new Mount namespace, call pivot_root
pointing to the CentOS filesystem and then run whatever processes we want to inside the 'pivoted' namespace. The processes will believe they’re running on CentOS the entire time.
Incidentally this is where the reexec from the previous article comes in handy. pivot_root
must be called from within the new Mount namespace, otherwise we'll end up changing the host's /
which is not the intention! And we want all this to happen before the namespaced shell starts so that the requested root filesystem is ready for when it does.
👉 Let’s Go
In Go, pivot_root
is implemented via the PivotRoot
func found in the syscall
package.
func PivotRoot(newroot string, putold string) (err error)
newroot
is the path to the desired new root filesystem and putold
is a path to a directory in which to move the current root. There are a few restrictions imposed on newroot
and putold
by the underlying pivot_root
sys call that we need to be aware of:
- They must both be directories
- They must not be on the same filesystem as the current root
putold
must be underneathnewroot
- No other filesystem may be mounted on
putold
Most of these are fine but the second point there will require a small workaround, as we’ll see in a moment. We’re also going to need a suitable newroot
in which to pivot to.
The process of preparing a newroot
filesystem can be quite a detailed and complex one. Take for example Docker’s layered filesystem approach in which many filesystem “layers” are joined together to present a single coherent root. We’re going to do something much simpler, which is to to assume that a suitable root filesystem has already been prepared for use.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.0$ mkdir -p /tmp/ns-process/rootfs
$ tar -C /tmp/ns-process/rootfs -xf assets/busybox.tar
From now on, ns-process
will expect a root filesystem to exist at this path and will raise an error if one can’t be found. Note that although we’re using BusyBox for this particular example, you could just as easily use any other distro.
Now that we have our newroot
, let’s write some code to make use of it.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.0
# Filename: rootfs.gofunc pivotRoot(newroot string) error {
putold := filepath.Join(newroot, "/.pivot_root")
// bind mount newroot to itself - this is a slight hack
// needed to work around a pivot_root requirement
if err := syscall.Mount(
newroot,
newroot,
"",
syscall.MS_BIND|syscall.MS_REC,
"",
); err != nil {
return err
}
// create putold directory
if err := os.MkdirAll(putold, 0700); err != nil {
return err
}
// call pivot_root
if err := syscall.PivotRoot(newroot, putold); err != nil {
return err
}
// ensure current working directory is set to new root
if err := os.Chdir("/"); err != nil {
return err
}
// umount putold, which now lives at /.pivot_root
putold = "/.pivot_root"
if err := syscall.Unmount(
putold,
syscall.MNT_DETACH,
); err != nil {
return err
}
// remove putold
if err := os.RemoveAll(putold); err != nil {
return err
}
return nil
}
With the pivotRoot
func in place, it’s time to put nsInitialisation
to good use.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.0
# Filename: ns_process.gofunc nsInitialisation() {
newrootPath := os.Args[1]
if err := pivotRoot(newrootPath); err != nil {
fmt.Printf("Error running pivot_root - %s\n", err)
os.Exit(1)
}
nsRun()
}
func main() {
var rootfsPath string
// ... cmd := reexec.Command("nsInitialisation", rootfsPath)
}
Notice that we’re now passing an argument, rootfsPath
, to nsInitialisation
. Once reexec
ed, this argument can be picked up by reading from os.Args[1]
. Also notice how the call to pivotRoot
comes before nsRun
. By doing this, we're ensuring that the new root filesystem will already have been pivoted to before the /bin/sh
process starts.
With all that in place, let's run the updated Go program and check to see which mounts, if any, are available to us now.
💁 The following has been tested on Ubuntu 16.04 Xenial with Go 1.7.1
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.0$ go build
$ ./ns-process
-[ns-process]- # cat /proc/mounts
cat: can't open '/proc/mounts': No such file or directory
Ah … now that we’ve pivoted to a new /
, we no longer have a /proc
! This is actually a good thing as it means we definitely can’t see the host’s mounts anymore, which is one of the main reasons for doing all this work in the first place. But, there’s probably only so far we can get without a working /proc
, so let’s add one to our new root.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1
# Filename: rootfs.gofunc mountProc(newroot string) error {
source := "proc"
target := filepath.Join(newroot, "/proc")
fstype := "proc"
flags := 0
data := ""
os.MkdirAll(target, 0755)
if err := syscall.Mount(
source,
target,
fstype,
uintptr(flags),
data,
); err != nil {
return err
}
return nil
}
And just as with pivotRoot
, mountProc
should be called from nsInitialisation
.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1
# Filename: ns_process.gofunc nsInitialisation() {
newrootPath := os.Args[1]
if err := mountProc(newrootPath); err != nil {
fmt.Printf("Error mounting /proc - %s\n", err)
os.Exit(1)
}
if err := pivotRoot(newrootPath); err != nil {
fmt.Printf("Error running pivot_root - %s\n", err)
os.Exit(1)
}
nsRun()
}
Ok, that should now be everything. Let’s try it out.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1$ go build
$ ./ns-process
-[ns-process]- # cat /proc/mounts
/dev/sda1 / ext4 rw,relatime,data=ordered 0 0
proc /proc proc rw,nodev,relatime 0 0
That’s looking much better - the host’s mounts are no longer visible to us and we have a new /proc
mounted and ready for action. But wait … there is one more thing …
🤔 PID namespace
The changes implemented above have had an unintentional side effect on the PID namespace setup. Prior to mounting the new /proc
, running ps
inside the namespaced shell would’ve resulted in all the host’s processes being listed. This is because ps
relies on /proc
to detect running processes and we were still referencing the host’s /proc
.
This is obviously a pretty terrible thing to happen from a container perspective! But fortunately now that we have our own /proc
(and are requesting a new PID namespace via the CLONE_NEWPID
flag), running ps
shows only processes that are relevant to us.
# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1$ go build
$ ./ns-process
-[ns-process]- # ps
PID USER TIME COMMAND
1 root 0:00 {exe} nsInitialisation /tmp/ns-process/rootfs
5 root 0:00 /bin/sh
8 root 0:00 ps
📺 On the next…
We’re nearing the season finale of “Namespaces in Go”, but we’re still missing one key piece of configuration - networking. What needs to be done to allow our namespaced shell to talk to the Internets? The answer to this and plenty more coming up, stay tuned…
Update: Part 6, “Namespaces in Go - Network” has been published and is available here.