Timezone in Kubernetes With k8tz

Kubernetes has established itself as the de-facto standard for container orchestration. It allows us to execute our containers over clusters of nodes and control all the configurations around them such as mounting storage volumes, secrets, network management, and much more.

Containers run on the host’s kernel and from there they get their clock, but the time zone does not come from the kernel but the user-space. Therefore in most cases they use Coordinated Universal Time (UTC) by default.

Photo of a clock, by Jiyeon Park on Unsplash

Even when the code is time zone agnostic, even things like correlating container logs with system logs become a headache. Some applications use the machine’s timezone as the default timezone and expect user to set the timezone. And the problem becomes even larger when container’s timezones are inconsistent in the cluster. There is just no standard. Luckily, there are several ways how to solve this problem and determine the time zone of the containers we are running. But before we get to the solution, let’s understand the problem first.

In most UNIX systems, different time zones are defined by Time Zone Information Format (TZif). It is a binary file format introduced in the 1980s. A reliable and accurate source for these files can be found in IANA Timezone Database. The files are usually located in /usr/share/zoneinfo. In most distributions, these files are part of the distribution and are installed by default. When we talk about docker base images, most of them do not contain this package by default and they need to be installed manually using a package manager or compiled from source, for example:

# debian/ubuntu
apt-get install tzdata
# alpine
apk add tzdata

Some base images such as centos and fedora come with the tzdata package installed out of the box.

Photo of terminal with bash by Gabriel Heinzer on Unsplash

But having these files is not enough, we still need to define, at the level of the machine (or container), what is the desired time zone. The /etc/localtime file configures the system-wide timezone of the local system. It is usually a symbolic link pointing to /usr/share/zoneinfo, followed by a timezone database name such as “Europe/Amsterdam” (i.e. /usr/share/zoneinfo/Europe/Amsterdam). You can find the list of available time zones on Wikipedia.

Another way of specifying the timezone is using the TZ environment variable. It can be set to the timezone identifier (i.e. TZ=Europe/Amsterdam) but it is customary to set it only in cases where the required time zone is different from the time zone of the machine. When set, it is prior to /etc/localtime.

Now that we understand where timezones coming from and how to set the timezone of the machine (or container), let’s dive deeper and try to demonstrate the problem with docker/podman. For that, we will execute the command date inside an empty ubuntu container:

$ docker run -i --rm ubuntu:21.04 date
Tue Sep 29 00:00:00 UTC 2021

From the output, we can see that the timezone is UTC. Ubuntu base image does not contain /usr/share/zoneinfo nor /etc/localtime by default and as we learned before, these are required to set the containers timezone. If we have the option, we can create Dockerfile that will install tzdata package and set a link from /etc/localtime to our desired timezone (Europe/Amsterdam), for example:

FROM ubuntu:21.04
RUN \
    apt-get update && \
    apt-get install -y tzdata && \
    ln -sf /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime && \
    rm -rf /var/lib/apt/lists/*

If we build it with docker build -t ubuntu:amsterdam and run the date command inside it:

$ docker run -i --rm ubuntu-amsterdam:0 date
Tue Sep 29 02:00:00 CEST 2021

The output now is different, the time offset is +0200, and the time zone abbreviation which in this case is CEST (Central European Summer Time).

But this is not always an option to create this layer above the image, there are few reasons why this should not be considered as the best practice:

• Not all images have package managers, some images are just FROM scratch
• It will be a struggle to maintain a Dockerfile for any of our dependencies images
• We will have to upload these images to somewhere our clusters can reach
• We may now know or decide what the timezone should be in the build time

There are few more problems that we will discuss later, but let’s first address these.

Instead of installing tzdata package in build time, we can mount the required files from our host machine, for example:

docker run -v /usr/share/zoneinfo/Europe/Amsterdam:/etc/localtime -i --rm ubuntu:21.04 date
Tue Sep 29 02:00:00 CEST 2021

So that’s was a lot easier, using -v we mounted the TZif file from our host machine directly to the container’s /etc/localtime and solved all the problems above, and we can now run almost any image with our desired timezone.

It is recommended, in addition, to mount the entire /usr/share/zoneinfo directory since the application may need information about another timezones at runtime, i.e -v /usr/share/zoneinfo:/usr/share/zoneinfo:ro.

Photo of ship with containers by Ian Taylor on Unsplash

So now we know how to solve this problem, let’s move on to Kubernetes. For that we will create a file date-pod.yaml that running the command date on ubuntu image, similar to our docker run command:

# date-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: date-pod
spec:
  containers:
  - image: ubuntu:21.04
    name: ubuntu
    args: ["date"]
  restartPolicy: OnFailure

Let’s execute this pod with kubectl apply -f date-pod.yaml and print the logs with kubectl logs date-pod and we will see that the date printed will be similar to:

Tue Sep 29 00:00:00 UTC 2021

So again, the timezone is UTC. To fix that we will use hostPath volume and volumeMounts to mount the timezone, which is similar to the -v argument from the docker run command. Create date-pod-amsterdam.yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: date-pod-amsterdam
spec:
  containers:
  - image: ubuntu:21.04
    name: ubuntu
    args:
    - date
    volumeMounts:
    - name: zoneinfo
      mountPath: /etc/localtime
      subPath: Europe/Amsterdam
      readOnly: true
  volumes:
  - name: zoneinfo
    hostPath:
      path: /usr/share/zoneinfo
  restartPolicy: OnFailure

apply it with kubectl apply -f date-pod-amsterdam.yaml and watch the logs with kubectl logs date-pod-amsterdam:

Tue Sep 29 02:00:00 CEST 2021

It is recommended, in addition, to mount the entire /usr/share/zoneinfo directory since the application may need information about another timezones at runtime.

So, problem solved right? Not exactly…

• If the pod already defined the TZ environment variable, our /etc/localtime mount will be ignored
• This is very manual work, for each pod we need to add hostPath volume and have 1 or 2 volumeMounts
• There is no guarantee that all the nodes in the cluster have tzdata installed
• We don’t always have permissions to use hostPath nor access to the nodes at all
• If some pods will have timezone and others not — it will just make more mess
• Most of Helm packages doesn’t support adding these patches by values.yaml

So its time to meet k8tz …

Cleanup created resources with: kubectl delete pod date-pod date-pod-amsterdam

Kubernetes Timezone Controller Logo, k8tz

k8tz is a Kubernetes admission controller and a CLI tool to inject timezones into Pods. It can be used as a manual tool to automate the transformation of our Deployments and Pods locally or it can be installed as an admission controller and use annotations to automate the process completely for any Pod created. But, it is more than that. To solve all the problems mentioned earlier, k8tz uses a slightly different strategy.

Instead of hostPath, it allocates emptyDir and injects bootstrap initContainer to fill the volume with the TZif files. The emptyDir is then used to mount both /etc/localtime and /usr/share/zoneinfo to every container in the Pod. And to make sure the desired timezone is effective, it adds TZ environment variable to all containers.

k8tz admission controller can be installed with Helm, it is easy as:

helm repo add k8tz https://k8tz.github.io/k8tz/
helm install k8tz k8tz/k8tz --set timezone=Europe/Amsterdam

Pods can be created now without the need for more configurations:

$ kubectl run -i -t ubuntu --image=ubuntu:21.04 --restart=OnFailure --rm=true --command date
Tue Sep 29 02:00:00 CEST 2021

You can also use annotations such as k8tz.io/timezone to select the timezone for a pod:

$ kubectl run -i -t ubuntu --image=ubuntu:21.04 --restart=OnFailure --rm=true --command date --annotations k8tz.io/timezone=Europe/London
Tue Sep 29 01:00:00 BST 2021

Or to disable timezone injection for a Pod with k8tz.io/inject:

$ kubectl run -i -t ubuntu --image=ubuntu:21.04 --restart=OnFailure --rm=true --command date --annotations k8tz.io/inject=false
Tue Sep 29 00:00:00 UTC 2021

If for some reason you want to use hostPath and not initContainer, you can use k8tz.io/strategy annotation:

$ kubectl run -i -t ubuntu --image=ubuntu:21.04 --restart=OnFailure --rm=true --command date --annotations k8tz.io/strategy=hostPath
Tue Sep 29 02:00:00 CEST 2021

These annotations can be specified in the Namespace as well, and affect all pods created in the namespace.

Conclusion

There are several solutions to the problem of time zones in Kubernetes, these solutions can be implemented manually but with several challenges and limitations along the way. Using k8tz you can automate the process and in a way that works without the need for additional settings or changes to existing resources, even when the required files are not available on the nodes. Ensuring that the time zone of all components in the system is consistent and all components have access to information about different time zones.

k8tz is free and open-source project, check it out: github.com/k8tz/k8tz.

Disclaimer: I’m the author of k8tz.