Fixing Overlaps in Thanos

As the Thanos documentation tells you overlapped blocks are:

blocks with exactly the same external labels in meta.json and for the same time or overlapping time period.

Their docs also tell you that your compactor won’t work if you have overlapped blocks in your store. This can be quite painful if you rely on its compaction and retention in your monitoring setup.

However what the docs don’t tell you, at least not explicitly, is how to find their source and remove them. Albeit at the cost of losing the metrics in the overlapped blocks.

How to Find the Source and Delete the Blocks

Initial Setup

Install the thanos CLI on your machine or wherever you intend to run the next steps from. This can be done with brew, docker, a download from their releases page or in many other ways.

brew install thanos

Then create a config.yaml containing the configuration for your back-end. For example, the following configuration is for an AWS S3 bucket accessed with AWS SDK credentials.

type: s3
config:
  aws_sdk_auth: true
  bucket: thanos
  endpoint: s3.amazonaws.com
  sse_config:
    type: SSE-S3

Find the Source

First, create a report of all the overlapped blocks.

thanos tools bucket verify --issues=overlapped_blocks \
    --objstore.config-file=config.yaml &> report.txt

Then, if you don’t know the source yet, find the blocks by their ulid in the web UI and infer the problematic instances from their labels.

thanos tools bucket web --objstore.config-file=config.yaml

Delete the Blocks

Warning: After this step you will lose the metrics that were in the overlapped blocks, on the bright side, you’ll get your compaction and retention back.

After fixing the source, mark all the overlapped blocks for deletion. For example, with the following bash script.

grep -Eo 'ulid: ([0-9A-Z]+)' < report.txt | while IFS= read -r id
do 
    thanos tools bucket mark --id="$(cut -d' ' -f2 <<< $id)" \
        --marker=deletion-mark.json \
        --details="delete overlap" \
        --objstore.config-file=config.yaml
        
done

And finally delete all the blocks marked for deletion.

thanos tools bucket cleanup --delete-delay=0 --objstore.config-file=config.yaml

Final Thoughts

I hope you found this post helpful and informational. Overlapped blocks can be very painful in some Thanos setups.

Interested in this type of work? We’re looking for talented people to join our team!