Containers: Part 3 — Understanding chroot

From previous article where we explained history of containers (Part 2) we can see that containers are built primarily to be run on Linux.

Today we have support for MacOS and Windows and we will discuss those separately, but since this started on Linux lets understand first how it started on Linux, and from our previous article we can see that everything started with chroot so lets explore how it works.

In order to keep things safe, and since I use MacOS, I used actual Virtual Machine (VM) that runs Debian 11 (or “bullseye”) that is managed by VirtualBox.

Running Debian 11 in Virtual Box

So imagine that we have following problem: we want to start bash process from which you can use only ls, cat, bash and you will have access to also directory that will be called /secret and will contain secret files that are passed from parent/host machine.

For this to work we need to create directory where we will chroot and we can create it inside tmp directory.

mkdir /tmp/container # create directory where we will chroot
mkdir /tmp/container/secret # storage for secrets
mkdir /tmp/container/bin # storage for binaries 
mkdir /tmp/container/lib # storage for libraries 
mkdir /tmp/container/lib64 # storage for 64bit libraries 

# lets write secret files
echo "SECRET_1='foo'" >> /tmp/container/secret/secrets.txt
echo "SECRET_2='bar'" >> /tmp/container/secret/secrets.txt

Now we can copy binaries that we want to use in chrooted process from our host machine

cp /bin/{bash,ls,cat} /tmp/container/bin

Binaries will not work unless we have all the libraries that they depend on, to find out dependencies about specific binary, like bash , you can use ldd

ldd /bin/bash

But since we have multiple binaries I will use a bit different command that checks this for multiple binaries, and creates list of libraries that we need to copy

ls /bin/{bash,ls,cat} | \
   xargs ldd | \
   grep '/lib' | \
   egrep -o '(\/lib[^ ]+)' | \
   sort | uniq

So we need to copy all of these to /tmp/container/lib and to /tmp/container/lib64 that we created before. But, also you will see that inside this directories libraries have subdirectories, so you can tweak command from before to generate commands to create those subdirectories.

ls /bin/{bash,ls,cat} | \
   xargs ldd | \
   grep '/lib' | \
   egrep -o '(\/lib[^ ]+)' | \
   xargs dirname | sort | uniq | \
   awk '{ print "mkdir -p /tmp/container" $1 }'

This will output commands that you need to run in order to create this directories

Create subdirectories for libraries that you need

Now you can use command from before that helps you list all libraries that your binaries depend on, to also generate copy commands for those libraries

ls /bin/{bash,ls,cat} | \
   xargs ldd | \
   grep '/lib' | \
   egrep -o '(\/lib[^ ]+)' | \
   sort | uniq | \
   awk '{ print "cp " $1 " /tmp/container" $1 }'

So output of this command will create list of commands that you need to execute in order to copy this libraries

Command to generate copy statements for libraries

Now that you executed all of this commands, you can run chroot

chroot /tmp/container /bin/bash

This will start bash process where / is actually /tmp/container , and only 3 binaries are available ls , cat , bash anything else that you try to run will not work, for example whoami.

Full output from my test machine

root@TestVM:~# mkdir /tmp/container 
root@TestVM:~# cd /tmp/container 
root@TestVM:/tmp/container# mkdir ./{bin,secret,lib,lib64}
root@TestVM:/tmp/container# echo "SECRET_1='foo'" > /tmp/container/secret/secrets.txt
root@TestVM:/tmp/container# echo "SECRET_2='bar'" >> /tmp/container/secret/secrets.txt
root@TestVM:/tmp/container# cp /bin/{bash,ls,cat} /tmp/container/bin
root@TestVM:/tmp/container# ls /bin/{bash,ls,cat} | xargs ldd | grep '/lib' | egrep -o '(\/lib[^ ]+)' | xargs dirname | sort | uniq | awk '{ print "mkdir -p /tmp/container" $1 }'
mkdir -p /tmp/container/lib64
mkdir -p /tmp/container/lib/x86_64-linux-gnu
root@TestVM:/tmp/container# mkdir -p /tmp/container/lib64
mkdir -p /tmp/container/lib/x86_64-linux-gnu
root@TestVM:/tmp/container# ls /bin/{bash,ls,cat} | xargs ldd | grep '/lib' | egrep -o '(\/lib[^ ]+)' | sort | uniq | awk '{ print "cp " $1 " /tmp/container" $1 }'
cp /lib64/ld-linux-x86-64.so.2 /tmp/container/lib64/ld-linux-x86-64.so.2
cp /lib/x86_64-linux-gnu/libc.so.6 /tmp/container/lib/x86_64-linux-gnu/libc.so.6
cp /lib/x86_64-linux-gnu/libdl.so.2 /tmp/container/lib/x86_64-linux-gnu/libdl.so.2
cp /lib/x86_64-linux-gnu/libpcre2-8.so.0 /tmp/container/lib/x86_64-linux-gnu/libpcre2-8.so.0
cp /lib/x86_64-linux-gnu/libpthread.so.0 /tmp/container/lib/x86_64-linux-gnu/libpthread.so.0
cp /lib/x86_64-linux-gnu/libselinux.so.1 /tmp/container/lib/x86_64-linux-gnu/libselinux.so.1
cp /lib/x86_64-linux-gnu/libtinfo.so.6 /tmp/container/lib/x86_64-linux-gnu/libtinfo.so.6
root@TestVM:/tmp/container# cp /lib64/ld-linux-x86-64.so.2 /tmp/container/lib64/ld-linux-x86-64.so.2
cp /lib/x86_64-linux-gnu/libc.so.6 /tmp/container/lib/x86_64-linux-gnu/libc.so.6 
cp /lib/x86_64-linux-gnu/libdl.so.2 /tmp/container/lib/x86_64-linux-gnu/libdl.so.2
cp /lib/x86_64-linux-gnu/libpcre2-8.so.0 /tmp/container/lib/x86_64-linux-gnu/libpcre2-8.so.0
cp /lib/x86_64-linux-gnu/libpthread.so.0 /tmp/container/lib/x86_64-linux-gnu/libpthread.so.0
cp /lib/x86_64-linux-gnu/libselinux.so.1 /tmp/container/lib/x86_64-linux-gnu/libselinux.so.1
cp /lib/x86_64-linux-gnu/libtinfo.so.6 /tmp/container/lib/x86_64-linux-gnu/libtinfo.so.6
root@TestVM:/tmp/container# apt-get install -y tree
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
tree is already the newest version (1.8.0-1+b1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@TestVM:/tmp/container# tree
.
├── bin
│   ├── bash
│   ├── cat
│   └── ls
├── lib
│   └── x86_64-linux-gnu
│       ├── libc.so.6
│       ├── libdl.so.2
│       ├── libpcre2-8.so.0
│       ├── libpthread.so.0
│       ├── libselinux.so.1
│       └── libtinfo.so.6
├── lib64
│   └── ld-linux-x86-64.so.2
└── secret
    └── secrets.txt

5 directories, 11 files
root@TestVM:/tmp/container# chroot /tmp/container /bin/bash
bash-5.1# tree
bash: tree: command not found
bash-5.1# whoami
bash: whoami: command not found
bash-5.1# cat /secret/secrets.txt 
SECRET_1='foo'
SECRET_2='bar'
bash-5.1# ls -al /
total 24
drwxr-xr-x 6 0 0 4096 Jan 16 15:07 .
drwxr-xr-x 6 0 0 4096 Jan 16 15:07 ..
drwxr-xr-x 2 0 0 4096 Jan 16 15:08 bin
drwxr-xr-x 3 0 0 4096 Jan 16 15:08 lib
drwxr-xr-x 2 0 0 4096 Jan 16 15:09 lib64
drwxr-xr-x 2 0 0 4096 Jan 16 15:08 secret
bash-5.1# exit 
exit

You have to be really careful with chroot if you are developing your own software that is using this system call. Child processes that use chroot need to be handled properly, for example you need to close all open resources from parent. Example with some of the exploits .