Percona backup for MongoDB caveats

Percona backup for MongoDB (pbm) is a well known backup solution

TL;DR

While scheduling backups with point in time recovery (pitr) you must be aware, that rotating backups with

$ pbm delete-backup [<flags>] [<name>]

will delete only full backups while the pitr slices will stay along with a their base full backup

Problem

pitr slices are not deleted with

$ pbm delete-backup [<flags>] [<name>]

Solution

If you've already met this problem, the solution will be to

Plan A

1. change the storage settings to another prefix or even a bucket
2. turn off pitr
3. resync the storage settings
4. turn on pitr
5. take another backup
6. adjust backup rotating script
7. after the rotating period delete backups manually from s3 storage

Plan B

If you don'care about previous backups, you can

1. delete all pitr slices
2. delete base for pitr slices
3. adjust backup rotating script

Let's cover in details both plans A and B

Plan A

1. change the storage settings to another prefix or even a bucket

$ cat /etc/pbm-storage.yml 
pitr:
  enabled: true
  oplogSpanMin: 60
storage:
  type: s3
  s3:
    region: us-west-2
    bucket: mongo-backup
    prefix: mongo-prod
    endpointUrl: https://s3.example.com
    credentials:
      access-key-id: access_id
      secret-access-key: key_id
$ vi /etc/pbm-storage.yml
$ cat /etc/pbm-storage.yml 
pitr:
  enabled: true
  oplogSpanMin: 60
storage:
  type: s3
  s3:
    region: us-west-2
    bucket: mongo-backup
    prefix: mongo-prod-fixed
    endpointUrl: https://s3.example.com
    credentials:
      access-key-id: access_id
      secret-access-key: key_id

2. turn off pitr

# get the list of backups, you will be able to see pitr turned on and a base backup
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# turn off pitr
$/usr/bin/pbm config --set pitr.enabled=false --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"

3. resync the storage settings

$ /usr/bin/pbm config --force-resync --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"

4. turn on pitr

# get the list of backups, you will be able to see pitr turned off and a base backup
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# turn on pitr
$/usr/bin/pbm config --set pitr.enabled=true --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# get the list of backups, you will be able to see pitr turned on and a base backup
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"

5. take another backup

# get the list of backups
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# turn on pitr
$/usr/bin/pbm backup --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# backup will start shortly, track the backup flow with the following command at every single node of the cluster, because backup can be triggered from any of them
$ journalctl -xefu pbm-agent
# or
$ /usr/bin/pbm logs --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# get the list of backups, you will be able to see the new backup
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# track if pitr is started with the following command at every single node of the cluster, because backup can be triggered from any of them
$ journalctl -xefu pbm-agent

6. adjust backup rotating script

# adjust at every single node of the cluster
$ cat /usr/local/bin/rotate_backup.sh
#!/bin/bash
code=$(mongo admin --port 27017 -u pbm_user -p pbm_password --quiet --eval "rs.isMaster().ismaster";)
if [[ $code -ne true ]]; then
exit 1
fi
/usr/bin/pbm delete-backup -f --older-than $(date -d '-14 days' +\%Y-\%m-\%d) --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
exit 0
$ vi cat /usr/local/bin/rotate_backup.sh
$ cat /usr/local/bin/rotate_backup.sh
#!/bin/bash
code=$(mongo admin --port 27017 -u pbm_user -p pbm_password --quiet --eval "rs.isMaster().ismaster";)
if [[ $code -ne true ]]; then
exit 1
fi
/usr/bin/pbm delete-pitr --force --older-than $(date -d '-14 days' +\%Y-\%m-\%d) --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
/usr/bin/pbm delete-backup -f --older-than $(date -d '-14 days' +\%Y-\%m-\%d) --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
exit 0

7. after the rotating period delete backups manually from s3 storage

for the command details refer to the your favourite s3 client tool documentation

Plan B

1. delete all pitr slices

# get the list of backups, you will be able to see pitr slices at the second part of the output
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# remove all pitr slices
$/usr/bin/pbm delete-pitr -a --force --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
# track pitr deletion with the following command at every single node of the cluster, because the command can be triggered from any node of the cluster
$ journalctl -xefu pbm-agent
# get the list of backups, you will see that slices will disappear
$ /usr/bin/pbm list --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"

3. adjust backup rotating script

# adjust at every single node of the cluster
$ cat /usr/local/bin/rotate_backup.sh
#!/bin/bash
code=$(mongo admin --port 27017 -u pbm_user -p pbm_password --quiet --eval "rs.isMaster().ismaster";)
if [[ $code -ne true ]]; then
exit 1
fi
/usr/bin/pbm delete-backup -f --older-than $(date -d '-14 days' +\%Y-\%m-\%d) --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
exit 0
$ vi cat /usr/local/bin/rotate_backup.sh
$ cat /usr/local/bin/rotate_backup.sh
#!/bin/bash
code=$(mongo admin --port 27017 -u pbm_user -p pbm_password --quiet --eval "rs.isMaster().ismaster";)
if [[ $code -ne true ]]; then
exit 1
fi
/usr/bin/pbm delete-pitr --force --older-than $(date -d '-14 days' +\%Y-\%m-\%d) --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
/usr/bin/pbm delete-backup -f --older-than $(date -d '-14 days' +\%Y-\%m-\%d) --mongodb-uri="mongodb://pbm_user:pbm_password@localhost:27017/?authSource=admin&replicaSet=examplers"
exit 0

Documentation

Percona backup for MongoDB documentation page

PBM conmmand reference