High-speed listing of container labels

At Wehkamp, we’re making good (and heavy) use of metadata. A great way to support our vision of self-describing services, where we store the information inside a Dockerfile. Yes, very convenient indeed.

Actually, our deployment pipelines depend heavily on these labels so we do the following in order to prepare deployments:

docker pull wehkamp/blaze-dodo-service:87–2b51c14
docker inspect wehkamp/blaze-dodo-service:87–2b51c14

This is what Jenkins would do:

13:06:19 2017-09-20 13:06:30,608 - util.get_metadata - INFO - Pulling image wehkamp/blaze-dodo-service:87–2b51c14 from docker hub
13:07:55 2017-09-20 13:08:06,665 - util.get_metadata - INFO - Inspecting local image wehkamp/blaze-dodo-service:87–2b51c14

This will show all data regarding this container image, and in our Python tooling, we’ll just grab the LABELS. There is just one tiny little problem with this approach. It’s slow. Really, really slow. As in, the docker pull can easily take several minutes; all the while stalling the deployment and thus preventing code from reaching production.

Luckily, the nice folks at the AtomicProject have created a great tool to perform remote inspections (leveraging Dockerhub API improvements that simply were not there when we took off). Check it out at https://github.com/projectatomic/skopeo.

Now, all we needed was change our call to docker inspect into a docker run. Awesome!

time docker run \
  -v ${HOME}/.docker/config.json:/root/.docker/config.json \
  --rm wehkamp/skopeo \
  inspect docker://docker.io/wehkamp/blaze-dodo-service:87-2b51c14 | jq .Labels
{
  "blaze.service.deployment.cpu": "0.1",
  "blaze.service.deployment.internal-port": "5000",
  "blaze.service.deployment.memory": "300",
  "blaze.service.deployment.minimum-instances": "1",
  "blaze.service.features.health-check.enabled": "true",
  "blaze.service.features.health-check.endpoint": "/status",
  "blaze.service.features.metrics.enabled": "true",
  "blaze.service.id": "dodo",
  "blaze.service.main-language": "dotnet",
  "blaze.service.name": "blaze-dodo-service",
  "blaze.service.team": "Ebony",
  "blaze.service.version": "87-2b51c14"
}

real 0m4.001s
user 0m0.022s
sys 0m0.016s

And in our tooling, we ended up doing this:

-    if pull:
-        log.info("Pulling image %s from docker hub" % full_name)
-        result = run_command(["docker", "pull", full_name])
-        if result.status:
-            raise BlazeImageNotFound("Image %s not found on docker hub" % full_name)
-
-    log.info("Inspecting local image %s" % full_name)
-    inspection_raw, err, _ = run_command(["docker", "inspect", full_name])
-    if "No such" in err:
-        raise BlazeImageNotFound("Image %s not found locally" % full_name)
-
-    if pull:  # pragma: no cover
-        log.info("Removing local image %s" % full_name)
-        run_command(["docker", "rmi", full_name])
-
-    inspection_json = json.loads(inspection_raw)
+    log.info("Inspecting image %s" % full_name)
+    inspection_raw, err, status = run_command([
+        "docker",
+        "run",
+        "--rm",
+        "-v",
+        "/var/run/docker.sock:/var/run/docker.sock",
+        "wehkamp/skopeo:latest",
+        "inspect",
+        "--creds=%s:%s" % (os.environ["U"], os.environ["P"]),
+        "docker://docker.io/%s" % full_name
+        ])
+    if status != 0:
+        raise BlazeImageNotFound("Image %s not found" % full_name)
+
+    config = json.loads(inspection_raw)
-    config = inspection_json[0]["Config"]
     labels = dict(filter(lambda e: e[0].startswith("blaze.service"), config["Labels"].iteritems()))
     env = config["Env"]

Eat that, Jenkins! And thank you RedHat :)

07:30:23 2017-09-25 07:30:23,881 - metadata.fetch - INFO - Fetching metadata for image 'wehkamp/blaze-dodo-service:87–2b51c14'
07:30:23 2017-09-25 07:30:23,900 - util.get_metadata - INFO - Inspecting image wehkamp/blaze-dodo-service:87–2b51c14
07:29:14 2017-09-25 07:30:25,168 - metadata.fetch - INFO - Fetching metadata overrides

This is obviously quite specific to our Wehkamp tools, but you can easily tell the inspection now only took a second.

And that was it, warp speed! Plus, we no longer had to run the docker-gc container to remove (clean-up) all those pulled containers to keep disks from stressing.

Thank you for taking the time to read this story! If you enjoyed reading it, clap for me by clicking the 👏🏻 below so other people will see this here on Medium.