Run a MongoDB Docker Image as a Singularity Container Service

Photo by Venti Views on Unsplash

This past month one of my tasks has been to take a hardened mongo image created with Docker and convert it to a Singularity container for use in a high performance computing environment. This was my first time working with Singularity (aka Apptainer), which means I learned so many fun things 😂. I just want to share a few of those lessons and challenges to bring you up to speed faster.

Challenge 1: Convert the Image

As I mentioned above, the image I started with was created by someone else via Docker. Thankfully I have access to the Dockerfile, but how do I convert this image without having to re-write the entire image as a Singularity Recipe (think Dockerfile)?

Well Singularity (Apptainer) actually has great support for converting Docker or OCI images into the Singularity format (SIF). You can pull, run, and build public or private docker images / archives and Singularity will automatically handle the conversion (see https://docs.sylabs.io/guides/3.0/user-guide/singularity_and_docker.html).

In my case, pulling from the private registry inside the deployment environment was not an option. So I needed to upload a tar’d (tarred?🤷) archive of the image and convert that. So something like:

singularity build mongo.sif docker-archive://mongo.tar

*Note: I didn’t use Docker to create the tar. I actually used Podman to pull the image and then tar it.

Challenge 2: No Docker Installed

The environment this container is supposed to be deployed to does not have Docker installed by design. So my next question was how can I convert the archive without docker. Doesn’t Singularity need Docker to be able to use the “docker-archive” directive? Turns out you don’t need docker installed! My guess is that Singularity is not calling the docker daemon at all, it just needs to know where and in what format your looking for the image. You can pull or build from public and private docker registries without Docker.

Challenge 3: Running with Environment Variables

So after running:

singularity build mongo.sif docker-archive://mongo.tar

my next goal was to run my SIF image with runtime environment variables for authentication. There are a couple ways to do this.

singularity run --env VAR=val mongo.sif

singularity run --env-file envfile mongo.sif

SINGULARITYENV_VAR=val singularity run mongo.sif

You might get a warning from Singularity that SINGULARITYENV_ will be deprecated and you should use APPTAINERENV_ instead. Either works.

Checkout https://docs.sylabs.io/guides/latest/user-guide/environment_and_metadata.html? for more info on these options and others.

Challenge 4: Mount a Writable Directory For Data

Singularity containers have read-only filesystems. Making them writable is possible through some different CLI options (--writable, --writable-tmpfs), but it’s typically a better practice to write your container data to a more persistent option than inside the container anyway. In a K8s cluster, I’d do this with a persistent volume / claim, but in this environment (at least for now) I’m wrote the data to a locked down directory on the host. The method for this is really similar to Docker.

singularity run mongo.sif --bind /path/to/datadir:/data/db

So my final imperative command was this:

APPTAINERENV_MONGO_USERNAME=testuser APPTAINERENV_MONGO_PASSWORD=userpass singularity run mongo.sif --bind $PWD/data:/data/db --auth

And it worked! Except for the fact that it runs in the foreground …

Challenge 5: Run as a Service

Perhaps I missed it but I never found a flag you can use with the Singularity run command that works like “-d” to detach the container. The way Singularity handles background running containers is through the “instance” interface. You can read through the docs here: https://docs.sylabs.io/guides/3.0/user-guide/running_services.html.

Easy enough, I just re-wrote my command like so:

APPTAINERENV_MONGO_USERNAME=testuser APPTAINERENV_MONGO_PASSWORD=userpass singularity instance start --bind $PWD/data:/data/db mongo.sif mongoContainer --auth

This certainly started an instance of the mongo container, but when I tried connecting the backend to it I got “ECONNREFUSED 127.0.0.1:27017” ….

Is it a firewall issue? Not likely. I checked anyway and that wasn’t it.

Challenge 6: Troubleshooting

Ok, so troubleshooting 101, where are the logs? Logs for Singularity instances can be found at one of the following paths (depends on the version of Singularity you’ve installed):

“~/.singularity/instances/logs/<host>/<user>/” || “~/.apptainer/instances/logs/<host>/<user>/”

Under my log output I found a mongo1.out file. The file was empty. What? The service was running when I launched the container via the run command but now nothing.

Let’s take a look inside the container:

singularity shell instance://mongoContainer

You’ll get something like:

Apptainer>

I did a “ps aux” for the running processes and did not find mongod. Bummer. So something about starting the instance is not executing the entrypoint / CMD of the mongo image.

I tried starting the service manually inside the container:

Apptainer> mongod --auth

The service started up but then immediately shut down due to permissions on the /data/db read-only file. Since Singularity executes the container as the user that runs the command, I thought there might be some issue with my file permissions.

The data folder was good to go, so maybe the container has to be run as a specific user. Thankfully I had the Dockerfile so I perused a bit and noticed that the mongodb user has ownership. So I started running the instance as the mongodb user. This didn’t work either so I tried as “sudo” and checked the logs and the process was running! Definitely not ideal though…. The backend still failed, however, with an authentication error.

I double checked the creds I’d passed the command. All correct. So now what?

singularity exec instance://mongoContainer env | grep MONGO_

This outputs all the environment variables in the container and then greps for the specific ones starting with MONGO_. The USERNAME and PASSWORD variables were not there. 🤔

Challenge 7: Creating a Recipe For Running as Instance

Reading through the “Running Services” docs, I realized they were making heavy use of Recipe files, which got me thinking that perhaps you HAVE to have a recipe for running containers as instances. And as far as I can tell that is correct.

Why can’t you start an instance entirely imperatively like you can “run” a container? I don’t know all the gritty behind-the-scenes details, but the major difference is in how the entrypoint gets executed. The Singularity “run” command, like Docker, calls the entrypoint script at startup, along with any args passed via CMD. When you start an instance, though, singularity is looking for a “%startscript”, which is a piece of the recipe file, as the entrypoint.

Thus I needed a recipe file (mongo.def) like this:

Bootstrap: docker-archive
From: mongo.tar

%environment
    MONGO_USERNAME=testuser 
    MONGO_PASSWORD=userpass
    PORT=27017
    export MONGO_USERNAME MONGO_PASSWORD PORT


%startscript
    docker-entrypoint.sh mongod

*Make sure to “export” your env vars!
*Also pass any CMD args directly as part of the “%startscript”.

Next is to build an image from the recipe file:

singularity build mongo.sif mongo.def

then:

singularity instance start --bind $PWD/data:/data/db mongo.sif mongoContainer

and it worked! The mongod process started, no write issues (even without sudo) and all env variables are present for authentication. The backend connected flawlessly.

I should note that you don’t have to place the environment variables in the recipe. You can still declare them imperatively at runtime with the --env flag. They didn’t seem to work in the APPTAINERENV_ format though.

Well that’s it. I won’t say that Singularity is exactly an easy transition but overall it’s not horrible. This is of course a small and specific use-case. There’s a whole lot more to explore in the docs, but hopefully this rundown will point you in the right direction.
Cheers!