Looks like bitnami/elasticsearch-curator is gone

Maybe you have realized, since last week (mid of June 2023) the old bitnami/elasticsearch-curator has been removed from DockerHub, which causes several issues with our elasticsearch installation.

Since I haven’t seen any announcement somewhere (which was kind of a surprise) I just want to shortly summarize what we did to overcome this, maybe it helps others as well.

We had quite a hard time with our benchmarks and clusters, since elasticsearch was filling up and caused several issues. Normally we track the throughput and latency of several clusters in one dashboard, which doesn’t look healthy end of last week.

We realized quickly that curator cronjobs were no longer running and crash loop because the images were no longer available.

You can also reproduce this via:

$ bashdocker pull bitnami/elasticsearch-curator:5.8.4
Error response from daemon: pull access denied for bitnami/elasticsearch-curator, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Turned out that the image has been renamed to https://hub.docker.com/r/bitnami/elasticsearch-curator-archived

$ docker pull bitnami/elasticsearch-curator-archived:5.8.4
5.8.4: Pulling from bitnami/elasticsearch-curator-archived
Digest: sha256:46c98206dfaef81705d9397bd3d962d1505c8cfe9437f86ea0258d5cbef89e7f
Status: Downloaded newer image for bitnami/elasticsearch-curator-archived:5.8.4
docker.io/bitnami/elasticsearch-curator-archived:5.8.4

If you use helm charts (like we do) and have a cronjob defined in the helm charts it will not help to upgrade the charts. The reason is that jobs, cronjobs, etc. are immutable.

You have to delete the job/cronjob and do a helm upgrade with--reuse-valuesand setting the right curator image.

If you use our charts (camunda-platform-helm) you have to set -set camunda-platform.retentionPolicy.image.repository=bitnami/elasticsearch-curator-archived to use the new curator image.

helm upgrade \
 "$releaseName" <YOUR-CHART> \
 --reuse-values \
 --set camunda-platform.retentionPolicy.image.repository=bitnami/elasticsearch-curator-archived

As an alternative, you can recreate the installed helm releases. To do so you can use the following script:

# we had several clusters to fix so this was part of a loop
ns="YOUR NAMESPACE" 
# release name is in our case the namespace name
releaseName="$ns"
# Get the values for the installed chart release (to reuse them)
values=$(helm get values “$releaseName” --namespace “$ns” -o yaml); 

# You can store the values into a separate file to be on the safe side
echo "$values" > "$ns-values.yaml"
# ...
# You could either set the curator image now here in the values
# or set it directly on installation
# ...
# Uninstall the chart
helm uninstall "$releaseName" --namespace "$ns"
# Install the chart inject values via stdin
echo "$values" | helm install "$releaseName" <YOUR-CHART> --namespace "$ns" --values -

Another thing we run into was that elasticsearch was in on some cases soo full that it didn’t even recover and was not able to free up space (with a curator). Later we found out that you need to increase the disk a bit, such that elasticsearch can recover and curator can free up space.

If you ask why this change happened (renaming of the docker image), I haven’t found any resources for it. The current assumption is that the curator is deprecated with elasticsearch 8 and likely will not work anymore with 8.x.

But if you still use a lower version, you might still want or need to use the curator so I hope this will help someone.