Your ops team with an unlabelled container?

Kelsey & Anne’s Holiday Webinar- How Metadata Stole Christmas

Kelsey Hightower and I held a webinar on Monday to talk about containers, labelling, annotations, Docker, Kubernetes and generally all things holiday-themed. Please bring yourselves, a glass of something Christmassy and a Santa hat :-)

SNEAK PREVIEW — Containers, Naughty or Nice?

To Kelsey, from Anne

It’s the holidays and we all love presents. A present is supposed to be a surprise but we still usually label them with a “from” and a “to”.

To ?, from ?

Unlike Christmas presents however, containers are not supposed to be a surprise for your ops team. But, less than 4% of public containers are labelled and an unlabelled container can be a complete surprise to ops: they don’t definitely know who built it, where the source code is or about any constraints on the application like “this needs loads of memory” or “you can only run this in Lapland”. But, it doesn’t have to be that way. You can label a container just like you can label a present. In fact, you can do a whole lot more.

Because containers all look the same (i.e. a common abstraction of your application) container metadata could be provided in a standard format and then read and understood by any tool or any person or even by your application itself. All of this is surprisingly powerful.

In this post we’re going to talk about providing metadata for Docker Containers running on a Kubernetes orchestrator. Both Docker and Kubernetes are doing interesting things with metadata. However, they’re confusingly using the same words for different concepts and there’s a huge interoperability issue! But before we discuss that, let’s step back and review what we’re actually talking about…

What in Santa’s Name is a Label?

Docker and Kubernetes mean different things by labels (!) So, let’s start at the beginning. I think we have 4 different, complementary, metadata types, which are all potentially useful — even vital.

The 4 Wise Metadata Types

• Docker build-time “labels”
• Kubernetes “labels” for pods (or any k8s object but let’s concentrate on pods for now)
• Kubernetes “annotations” for pods
• Docker run-time “labels”

As a reminder, Kubernetes doesn’t expose the concept of containers on their own, it deals exclusively with “pods”, which are grouped sets of containers that always deploy and run together. There is a 1:N relationship between pods and containers — that can be 1:1.

Step 1: Build-Time Docker Labels

Your first container operation is usually building an image using “docker build”. In your Dockerfile you can define a set of immutable build-time labels for your container image. Build-time labels should provide unchanging information about your image like what code it was built from or when it was built. Build-time labels can also specify important information that you as a developer know but an ops person won’t necessarily know like “this application requires lots of memory”. For a recommended set of build-time labels see label-schema.org

FROM alpine:3.3
MAINTAINER Ross Fairbanks "ross@microscaling.com"

ENV BUILD_PACKAGES ca-certificates

RUN apk update && \
    apk upgrade && \
    apk add $BUILD_PACKAGES && \
    rm -rf /var/cache/apk/*

# Add binary and Dockerfile
COPY microscaling Dockerfile /

RUN chmod +x /microscaling

# Metadata params
ARG VERSION
ARG VCS_URL
ARG VCS_REF
ARG BUILD_DATE

# Metadata
LABEL org.label-schema.vendor="Microscaling Systems" \
      org.label-schema.url="https://microscaling.com" \
      org.label-schema.name="Microscaling Engine" \
      org.label-schema.description="Optimal resource util" \    
      org.label-schema.version="1.2.3" \
      org.label-schema.vcs-url=$VCS_URL \
      org.label-schema.vcs-ref=$VCS_REF \
      org.label-schema.build-date=$BUILD_DATE \
      org.label-schema.docker.schema-version="1.0"

The Docker build-time labels can be read for an container image using the “docker inspect” command.

Step 2: Kubernetes Labels and Annotations

Kubernetes pulls pre-built container images out of your registry and deploys them to your infrastructure using the rules specified in your resources file (yaml). There are 2 different types of metadata you can specify in this resources file.

• Kubernetes Labels are metadata for internal setup and configuration. Kubernetes labels let you name pods for usability (like “web” or “back-office” or “canary1”) and then, when paired with Selectors, let you define which pods can talk to one another and which services they can use (e.g. you can say that “web” objects can only talk to other “web” objects). So, Kubernetes labels are for defining your what-talks-to-what config and, amongst other things, they let you safely share physical resources between deployments that historically would’ve been physically separated and separately provisioned. This what-talks-to-what purpose for labels is the key difference between labels and annotations (we think).
• Kubernetes Annotations are where you provide your more externally-facing pod metadata, the kind of stuff that a third party tool like a monitoring app or a load balancer might read, for example a name and description or a health check url.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
  labels:
    app: microbadger-web
    name: microbadger-web
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 3000
    name: http
  - port: 443
    protocol: TCP
    targetPort: 3000
    name: https
  selector:
    app: microbadger-web
  sessionAffinity: None
  type: LoadBalancer

Step 3: Run-time Docker Labels

Docker have the concept of run-time labels that you can specify as part of a “docker run” command. These apply to a container as long as it is running and can be queried with a “docker inspect” command on the running container. They cannot be changed for a running container.

$ docker run --label com.example.foo=bar ubuntu bash

Step 4: Finally, Kubernetes Annotations at Run-time

Finally, Kubernetes annotations don’t only have to be set at deploy-time. They can be added or changed for a pod even while the pod is running. They could, for example, provide time-dependent metadata like “Right now, I’m the most critical application here” to third party operational tools like application-aware schedulers.

Docker and Kubernetes Are Not Playing Nicely

Here’s where we reveal the big inter-operability issue — Kubernetes and Docker don’t co-operate that well wrt Docker build-time or run-time labels. The Docker labels are available to the pod itself using the Kubernetes downward API but they don’t seem to be exposed by the external Kubernetes APIs, which means those potentially useful Docker labels are effectively invisible to third party tools in a Kubernetes deployment.

That’s a shame. Although, arguably, Docker run-time labels don’t add anything over K8s run-time annotations, Docker build-time labels play a vital role: they allow developers to communicate information to the ops team that the ops team would otherwise lack —for example what code the container was built from.

There need to be tools to bridge this gap — more on that soon…

Please hit the Recommend button below if you found this article interesting or helpful, so that others might be more likely to find it.

Check out MicroBadger to explore image metadata, and follow Microscaling Systems on Twitter.

Read next → Inspect your private Docker images