Spawning Container with runc

5 min readJan 27, 2024

The container technology which started with few Linux primitives is continuously evolving — placing layers of abstractions over the basic idea of container. These abstractions makes our life easier — by helping us manage complex environment in efficient manner.

Personally I love to go deep and learn the first principles driving an idea. While I learned layers and layers of abstractions created on a container, I never lost the yearning to understand core principle of containerization. This and subsequent article is my attempt to organize my learning of these core principles through hands-on example.

In this article we will see how we can generate an OCI bundle and spawn a container with runc .

runc

runc is a container runtime which is Open Container Initiative(OCI) complaint. runc is used to spawn a container and is widely used by container technology like Podman and Kubernetes as an underlying layer.

runc requires OCI bundle to start a container. In next few section I will share an example of generating OCI bundle out of a container image and spawning a container with the generated OCI bundle.

OCI Image Layout Specification

In this section we will generate OCI Image Layout Specification using skopeo command line utility. skopeo helps us to copy a specific Image to OCI Image Layout Specification like below:

$ skopeo copy docker://registry.redhat.io/rhbk/keycloak-rhel9:22-7  oci:keycloak-rhel9:22-7 --remove-signatures
Copying blob f72461870632 done  
Copying blob 4a07e32e5b90 done  
Copying config 355cf15f97 done  
Writing manifest to image destination
$

We can see there were 2 blobs and 1 config copied while image was pulled from an OCI registry— lets see the same in directory structure below, where we can see same 2 blobs and 1 config :

$ tree  keycloak-rhel9/
keycloak-rhel9/
├── blobs
│   └── sha256
│       ├── 355cf15f973ac0dd192200deb60dfe71f3886f7f613f2034def26102eff29767
│       ├── 4a07e32e5b90d5efdf68adcd722229e75ee66db782edcd51c70201f6a47a3ff3
│       ├── 9b161cd842de59b117726ca8508070c6190964bf1ad4abcd139012b400e85a8d
│       └── f72461870632592db36cc0addefccfe6805504402b9f0d2dfba63348cef50829
├── index.json
└── oci-layout

3 directories, 6 files

Also there is an additional entry which is just an index to specific image. If we open the index file we can see it is pointing to a blob 9b161cd842de59b117726ca8508070c6190964bf1ad4abcd139012b400e85a8d and this blob contains the digest of 1 config and 2 tar+gzip file copied during image pull.

# Index to specific image
keycloak-rhel9]$ cat index.json  | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:9b161cd842de59b117726ca8508070c6190964bf1ad4abcd139012b400e85a8d",
      "size": 566,
      "annotations": {
        "org.opencontainers.image.ref.name": "22-7"
      }
    }
  ]
}
keycloak-rhel9]$ cd blobs/sha256
#If we open the above mentioned digest - we could see the config file and 2 layers (the same as 2 blobs that was downloaded as part of image) 
sha256]$ cat 9b161cd842de59b117726ca8508070c6190964bf1ad4abcd139012b400e85a8d | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:355cf15f973ac0dd192200deb60dfe71f3886f7f613f2034def26102eff29767",
    "size": 9072
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f72461870632592db36cc0addefccfe6805504402b9f0d2dfba63348cef50829",
      "size": 7193288
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4a07e32e5b90d5efdf68adcd722229e75ee66db782edcd51c70201f6a47a3ff3",
      "size": 240366885
    }
  ]
}

So what are these blobs:

File with type application/vnd.oci.image.config.v1+json contains the configuration like ExposedPorts , Env , Entrypoint , rootfs ,Labels , history of changes made.
File with type application/vnd.oci.image.layer.v1.tar+gzip contains the different layer of filesystem. We can extract these files to see the filesystem. Through these application/vnd.oci.image.layer.v1.tar+gzip the final root fileystem is created.

sha256]$ cat 355cf15f973ac0dd192200deb60dfe71f3886f7f613f2034def26102eff29767 | jq
{
  "created": "2024-01-04T18:14:25.145904275Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "User": "1000",
    "ExposedPorts": {
      "8080/tcp": {},
      "8443/tcp": {}
    },
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "LANG=en_US.UTF-8"
    ],
    "Entrypoint": [
      "/opt/keycloak/bin/kc.sh"
..........

sha256]$ tar -xvf 4a07e32e5b90d5efdf68adcd722229e75ee66db782edcd51c70201f6a47a3ff3
afs/
bin
boot/
dev/
dev/null
etc/
etc/.java/
...........
opt/
opt/keycloak/
opt/keycloak/LICENSE.txt
opt/keycloak/README.md
opt/keycloak/bin/
opt/keycloak/bin/client/
opt/keycloak/bin/client/keycloak-admin-cli-22.0.8.redhat-00001.jar
opt/keycloak/bin/client/keycloak-client-registration-cli-22.0.8.redhat-00001.jar
opt/keycloak/bin/client/lib/
opt/keycloak/bin/client/lib/bcprov-jdk18on-1.74.0.redhat-00002.jar
opt/keycloak/bin/client/lib/keycloak-crypto-default-22.0.8.redhat-00001.jar
opt/keycloak/bin/client/lib/keycloak-crypto-fips1402-22.0.8.redhat-00001.jar
opt/keycloak/bin/federation-sssd-setup.sh
opt/keycloak/bin/kc.bat
opt/keycloak/bin/kc.sh
opt/keycloak/bin/kcadm.bat
opt/keycloak/bin/kcadm.sh
opt/keycloak/bin/kcreg.bat
opt/keycloak/bin/kcreg.sh
opt/keycloak/conf/
opt/keycloak/conf/README.md
opt/keycloak/conf/cache-ispn.xml
opt/keycloak/conf/keycloak.conf
opt/keycloak/data/
opt/keycloak/lib/
opt/keycloak/lib/app/
opt/keycloak/lib/app/keycloak.jar
.......

OCI Bundle

OCI bundle should have a config.json file and container root filesystem.

OCI bundle can be generated using umoci which will require the OCI Image Layout which was created in previous section of the article. umoci will generate the config.json and container root filesystem out of the OCI image layout.

keycloak]$ umoci unpack --rootless --image keycloak-rhel9:22-7 bundle
   • rootless{root} ignoring (usually) harmless EPERM on setxattr "user.overlay.impure"
   • rootless{usr/lib} ignoring (usually) harmless EPERM on setxattr "user.overlay.impure"
   • rootless{usr/lib64} ignoring (usually) harmless EPERM on setxattr "user.overlay.impure"
   • rootless{usr/lib64/pm-utils} ignoring (usually) harmless EPERM on setxattr "user.overlay.impure"
keycloak]$ cd bundle
bundle]$ ls 
config.json  rootfs  sha256_9b161cd842de59b117726ca8508070c6190964bf1ad4abcd139012b400e85a8d.mtree  umoci.json
[rissingh@rissingh-thinkpadp1gen4i bundle]$

Spawn the container using runc

We can now spawn the container by providing the generated OCI bundle to runc.

To be able to start the container — we will have to set terminal to false and remove the user section from config.json.

bundle]$ cat config.json 
{
 "ociVersion": "1.0.0",
 "process": {
  "terminal": true,   ----> set to false
  "user": {           ----> remove this complete section
   "uid": 1000,
   "gid": 0
  },
  "args": [
   "/opt/keycloak/bin/kc.sh"
  ],
# If not removed we will get error like following
bundle]$ runc create test-keycloak
ERRO[0000] runc create failed: cannot allocate tty if runc will detach without setting console socket
bundle]$

Lets create a blank container with name test-keycloak and start Keycloak server within the container like below:

bundle]$ runc create test-keycloak
bundle]$
bundle]$ runc list
ID              PID         STATUS      BUNDLE                                                        CREATED                         OWNER
test-keycloak   221979      created     /path/to/bundle   2024-01-27T09:39:47.15449274Z   #109626
bundle]$
# Starting the keycloak instance within the container - it will fail to start because we have not done the basic plumbing like creating network namespaces etc
bundle]$ runc exec test-keycloak /opt/keycloak/bin/kc.sh start-dev
Updating the configuration and installing your custom providers, if any. Please wait.
2024-01-27 09:40:53,473 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 2838ms
2024-01-27 09:40:54,215 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
2024-01-27 09:40:55,091 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2024-01-27 09:40:55,315 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2024-01-27 09:40:55,339 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': IO Exception: "java.net.UnknownHostException: umoci-default: umoci-default: System error" [90028-224]
2024-01-27 09:40:55,340 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.Nu

Conclusion

This was a short article which talks about the OCI Image Layout, OCI bundle and spawning a container using runc. Understanding these basic idea can be helpful in understanding Container Runtime Interface of Kubernetes and why it is called by Kubelet to spawn a container.