Today, someone asked whether it was possible to use Podman to copy files into a CRI-O container using podman mount instead of using crictl. However, they could not see CRI-O containers using podman ps (and if they had used crictl ps to list CRI-O’s containers, they would not see containers created by podman run and podman create). However, they were able to see containers created by Podman and CRI-O with buildah containers --all, as well as containers what were created by Buildah itself. This poses a question: why can Buildah see containers from Podman and CRI-O, but Podman and CRI-O can only see containers they themselves created?
Let’s start by answering why Buildah can see containers from all three tools. The answer is the containers/storage library. CRI-O, Podman, and Buildah all use this library to manage storage for the containers and images they create, and all share a single store by default. Because of this, images will be shared by the three tools by default (if an image is pulled down with podman pull, CRI-O does not need to pull it again if it wants to create a container using the image). The same is true for containers. All tools using the storage library can see and mount all containers created by using the library.
This answers why Buildah can see all containers on the system, but not why CRI-O and Podman cannot. The answer there is more complicated, but the core is this: a container is more than just storage. The storage library handles creating container root filesystems from images, committing changes made into new images, and similar things. But to run a container, you need to know many more things than just what image a container was made from (look at the manpage for podman run for an example — there are dozens of options and flags, and all of them must be accounted for when running the container). CRI-O and Podman store this extra information as persistent state, CRI-O in a single file per-container, Podman in a shared database. Both of them only recognize containers present in their state, instead of querying the storage library for a list of all containers. Without the full description of a container from the state, we cannot do many things (starting, stopping, inspecting, and even listing many basic details in ps would be impossible). Buildah does a far more limited set of things to containers, and as such does not require an extensive state.
This explains why CRI-O and Podman cannot see Buildah containers, but not why they cannot see each other’s containers. The answer here is that they previously were able to, and soon they will again. Podman was originally built off of CRI-O code, and could interact with and debug CRI-O containers. However, the podman run command proved difficult to implement, as CRI-O was designed specifically for Kubernetes and much of it was too single-purpose to do what we required. The solution was to temporarily diverge, allowing Podman to develop a library for managing containers (libpod) without making major changes to CRI-O. That library would then be reintegrated with CRI-O to again allow the two to share containers. This reintegration hasn’t happened yet, but we’re hard at work on the last blockers preventing it. Soon, CRI-O and Podman will again be sharing containers. There are also plans to change Buildah to use libpod for container management as well, unifying all three tools and letting all three share containers.