CRI-O

Over the last few years my team and I have been working on tools and libraries related to linux containers. This includes the tools to start them, called container runtimes. Until recently, the main container runtime has been the upstream docker project, but now we are seeing a proliferation of tools in the ecosystem.

Today, the OpenShift and Kubernetes orchestration projects support upstream docker as an underlying container runtime. It was a natural fit, and this has worked OK, but there have been some very real problems.

First, with every new release of the upstream docker container runtime, there has been the cascading effect of things breaking in Kubernetes workflows. This is because upstream Docker is an independent project that has its own expanding goals, and is not required to ensure that Kubernetes workloads continue to work on each release. There were no tests in upstream docker to prevent this.

Second, we saw more and more features being added to the upstream docker project that were growing its complexity. Most of these features were not necessary for, nor focused on Kubernetes. This is because the upstream docker project was focused on their own orchestration tools. There were conflicting goals with the upstream docker project.

The upstream docker project understands these problems and has started to break code out into modules like containerd and Moby. While these projects are interesting, perhaps even promising, it seems a long term solution is a way off. These projects are in early days, and changes are happening very quickly. Furthermore, there is no open governance structure in place. So, we are back where we started — more breakages in Kubernetes and the potential for conflicting goals like the upstream docker project.

We decided in the fall to start looking into a simpler alternative to serve as the underlying container runtime for OpenShift and Kubernetes workflows.

We wanted something that would be explicitly compatible with, and designed for Kubernetes systems — something that wouldn’t break when we did updates. We also wanted something that would incorporate the features that Kubernetes users care about (read only containers, no builds in production, better image caching/distribution, etc). We wanted a container runtime that is designed with a laser focus on industry standards — built to the Kubernetes Container Runtime Interface (CRI) and Open Containers Initiative (OCI) specifications. Finally, we wanted a project with completely open governance and transparency.

With the OCI and CRI specifications in place, all of the components were available to build a stable container runtime using open industry standard components. We called the project CRI-O, and are happy to announce we have an alpha release and on track to 1.0.

What is CRI-O?

CRI-O is an implementation of the Kubernetes Container Runtime Interface (CRI) which enables using Open Container Initiative (OCI) compatible runtimes. It allows Kubernetes to use any OCI compliant runtime as the container runtime for running pods. Today it supports runc and Clear Containers as container runtimes but, in principle any OCI conformant runtime can be plugged in and we look forward to expanding in the future.

CRI-O supports OCI container images and can pull from any compliant container registry. It is a lightweight alternative to using Docker as the runtime for kubernetes. CRI-O is developed by maintainers and contributors from Red Hat, Intel, SUSE, Hyper, IBM and others. We are a community driven, open source project — feedback, users, and of course contributors are always welcome: https://github.com/kubernetes-incubator/cri-o

Scope

The scope of the project is tied to that of the CRI. Currently the only supported user of CRI-O is Kubernetes. Given this we strive to ensure that CRI-O always works with Kubernetes by providing a stringent and comprehensive test suite. These Kubernetes end-to-end tests are run on every pull request to ensure that it doesn’t break Kubernetes. These tests are constantly evolving to keep pace with changes in Kubernetes.

Components

CRI-O is made up of several components that are found in different GitHub repositories. Let’s take a look at the components.

OCI compatible runtimes

CRI-O supports any OCI compatible runtime. We test with runc and Clear Containers today.

https://github.com/opencontainers/runtime-tools contains the library that is used to generate OCI configurations for these runtimes.

Storage

The containers/storage is used for managing layers and creating root file-systems for the containers in a pod: Overlayfs, devicemapper, AUFS and btrfs are implemented, with Overlay as the default driver. There is also support for network based file system images (NFS, Gluster, Cefs) on the way.

Image

The containers/image library is used for pulling images from registries. It supports Docker schema 2 version 1 and version 2. It also passes all Docker and Kubernetes tests.

Networking

Container Network Interface (CNI) (https://github.com/containernetworking/cni) is used for setting up networking for the pods. Various CNI plugins such as Flannel, Weave and OpenShift-SDN have been tested with CRI-O and are working as expected.

Monitoring

conmon is a utility within CRI-O that is used to monitor the containers, handle logging from the container process, serve attach clients and detects Out Of Memory(OOM) situations.

Security

Container security separation policies are provided by a series of tools including SELinux, Capabilities, seccomp, and other security separation policies as specified in the OCI Specification.

Pod Architecture

With CRI-O you’ve got the following setup:

A detailed breakdown of the architectural components is as follows:

  • Pods live in a cgroups slice and hold shared IPC, net and PID namespaces.
  • The root filesystem for a container is generated by the containers/storage library when CRI CreateContainer/RunPodSandbox APIs are called.
  • Each container has a monitoring process (conmon) that receives the master pseudo-terminal (pty), copies data back and forth between master/slave pty pair, handles logging for the container and records the exit code for the container process.
  • The CRI Image API is implemented using the containers/image library.
  • Networking for the pod is setup through use of CNI, so any CNI plugin can be used with CRI-O.

Status

CRI-O version 1.0.0-alpha was released on June 20, 2017. This release is passing all of the kubernetes end-to-end tests.The features implemented so far:

  1. Pod and container lifecycle
  2. Image lifecycle
  3. CNI networking integration
  4. Logging
  5. Exec (sync/streaming)
  6. Attach/Detach
  7. Port forwarding
  8. OOM detection and reporting
  9. Support for restarting the daemon
  10. Support for multiple storage backend drivers (overlay, devicemapper, aufs, btrfs)
  11. SELinux support
  12. Seccomp support
  13. Clear Containers support
  14. Supports latest runc v1.0.0-rc3 and runtime-spec v1.0.0-rc5
  15. Gpg check on image pull
  16. Mixed runtimes (runc and Clear Containers)

Demos

You can see CRI-O in action in these demos:

Try it yourself

  • CRI-O can be setup using kubeadm. You can try it out by following the instructions at https://github.com/cri-o/cri-o-ansible
  • Minikube integration is in progress and should be available soon.
  • Feel free to join #cri-o on IRC (freenode)

Next Steps

Releasing CRI-O 1.0 final is the next goal for the project. We are concentrating on fixing bugs and improving stability to reach that goal. We are also looking to graduate from the Kubernetes incubator and to track and support upcoming kubernetes releases. The maintainers and contributors of CRI-O are very excited to have made this release that allows Kubernetes to use any OCI compliant runtime. We are looking forward to growing this project and would welcome your contributions in future release.

How can you contribute?

CRI-O is developed at https://github.com/kubernetes-incubator/cri-o. You can take a look at the issues and make PRs to contribute fixes and features to the project. Also, testing it out and opening issues for any bugs would be very helpful. One way to do that would be to follow the README and test various kubernetes features using CRI-O as the runtime. https://kubernetes.io/docs/tutorials/ is a good starting point to test out various kubernetes features. We are introducing a command line interface to allow users to play/debug the back end of cri-o, and we need lots of help building out the CLI. Anyone wanting to do some GOLANG programming is welcome to take a stab. Help with packaging and documentation is also always needed.

Communication happens at #cri-o on IRC (freenode) and on GitHub issues and PRs.