How to do ransomware resistant backups properly with Restic and Backblaze B2
I have been using Backblaze combined with Restic for years to do my backups and archiving for my personal systems. After an employer of mine got hit by a ransomware attack earlier this year it got me thinking. Is the backup scheme I am currently using going to resist a ransomware attack? Won’t an attacker be able to delete my existing backups?
This is a story about building immutable backups on the cheap. The example given here is using my setup, but I already rolled out the results of the threat model presented here to some Kubernetes-based infrastructure using the K8up.io, a backup utility for Kubernetes, that is using Restic under the hood.
Where I started
In the beginning, I was using restic with a Backblaze B2 Bucket with an Access Key generated through the Backblaze Web UI. I configured a systemd service on a timer to take daily snapshots of my relevant data. Since I wanted to create Backups using that key, it needed read and write permissions. Backblaze allows scoping keys to Buckets, which I did. Sounds like a pretty simple setup.
This left me with a problem: This backup aims to protect against problems with my machine. It should work fine for hardware failure, house fires, and user error, but if my machine gets compromised the attacker will find a high-privilege access key for the Bucket containing the backups. The attacker could just delete my backups! So, what can be done about this?
Object Lock?
Backblaze has a feature called Object Lock, which allows setting a holding period for objects in Buckets, that prevents deletion until a time specified while locking. This lock can not be removed before expiry. The only option is to delete your whole Backblaze account.
So, why can we not use this with Restic? Let’s start with a short intro to the Restic Repository Format. This is a simplification, but I think it should make it clear why Object Lock is hard to use in this case.
Pack and Blob reuse in restic
Restic uses a concept called content-addressable-storage to store de-duplicated data. When creating a backup of a file Restic splits it into chunks, hashes them using SHA256, and stores the chunk with some metadata using the SHA256 as the path. When encountering the same chunk again, e.g. in a subsequent backup, it will not store the chunk again. These so-called Blobs are assembled into Packs, which assemble Blobs into their files again, and these Packs are referred to by Snapshots, which are created every time you trigger a Backup. There is no explicit ownership between Snapshots, Packs, and Blobs, which allows Restic to reuse Packs and Blobs across Backups. But that also means, that when it is time to remove/prune old backups, there is no guarantee, that Blobs created by a Snapshot, that is about to be deleted can be deleted as well, since these Blobs could have been used by subsequent backups.
It would be possible to lock any object associated with a snapshot, which could be done with some custom scripts. But that sounds like a lot of work and could lead to issues with restic, which expects objects to be writeable and deletable. There are also much better options:
“Deleting” objects in B2/S3
In Object stores, like B2 and S3, there are two ways to get rid of objects. You can overwrite them with a new version or you can delete them. But there is also the option to overwrite an existing object with a “hidden marker”. This is a “non-destructive delete” since the old version is kept by default. Deleting files is destructive and there is no recovering data after it has been called on an Object. After hiding a file you can still access it as a previous version of the Object.
These are two distinct operations, which require two different permissions to perform. (On B2: writeFilesand deleteFiles )
Overwritten objects, also called file versions, will be kept in the bucket and can still be accessed and will not be removed from the bucket by default. But you can set Lifecycle Rules on B2 to delete hidden objects after a certain time.
This gives us an interesting option to implement backup protection on B2’s side without having to handle anything on the client side. Since deleteFiles is a destructive operation you should not use any keys with that capability on machines for automatic backup, since an attacker could acquire these keys and delete your backups. But, Restic does not need the deleteFiles capability, so you can use keys without it. Restic only needs listBuckets, listFiles, readFiles, writeFiles. But not deleting files would mean that your Bucket would grow due to the old backups not getting deleted. That can be fixed with a lifecycle rule.
Lifecycle Rules
OK, so we can “delete” data from our repository. But that data containing old data will continue to accumulate and cost money. S3/B2 has a solution for this: Lifecycle Rules
Lifecycle Rules are operations that allow you to run operations on your stored objects. On B2 you can do two operations: hide uploaded files after x days and delete hidden objects/old object versions after y days.
We are not interested in hiding uploaded objects, but using a lifecycle rule to delete hidden objects a certain time after they have been deleted by our backup tool. This allows for timed-immutable backups, which is not as secure as truly using an Object Lock but is much more serviceable in practice. It is secure as long as an attacker gets access to your B2 Master Key or a Key that can create new keys.
Putting it together (The TLDR)
Application Keys in Backblaze B2 created with the Backblaze CLI or the Backblaze API can have much more granular permissions than those created via the Web UI. So we have to use the CLI
This CLI command creates a Bucket that deletes all hidden files after 30 days:
$ backblaze-b2 create-bucket --defaultServerSideEncryption=SSE-B2 important-backup --lifecycleRule '{"daysFromHidingToDeleting": 30, "daysFromUploadingToHiding": null, "fileNamePrefix": ""}' allPrivatewriteThis command creates a restricted key for use by automated backup systems:
$ backblaze-b2 create-key --bucket important-backup restic-key listBuckets,listFiles,readFiles,writeFilesIf someone deletes your backups you can restore them within 30 days with tools like https://github.com/viltgroup/bucket-restore.
For this to work properly you need to make sure your Backblaze account is protected from unauthorized access. This means not storing high privilege keys (e.g. Keys that can provision new keys and keys with delete access on the backup bucket) on the target system and having Two-Factor-Authentication enabled.
And setting this up is very cost-effective. You only pay for deleted files a maximum of x days after they have been “deleted” and if you accidentally uploaded a large amount of data you want to have gone right now you can force delete it with a higher privilege key.
What more can be done?
Restricting the Key deployed on the target system, that is used for backups should protect you against data loss caused by that key. But there is more that can be done
Backblaze Cloud Replication
To take it a bit further you could create another seperate “cold-storage” Backblaze account and set up a cloud replication rule. Hide Markers are not replicated, but deleted objects are deleted on the target. I tested this method using my backup repository and it seems to work just fine. Checking the repository by running restic check on the replicated bucket did return some warnings about duplicate packs, but all the data was readable.
The only thing you will have to do is to use the --no-lock flag on all restic commands, since the replica contains stale operation locks, that have been hidden on the source bucket but due to a limitation on how B2 Cloud Replication works, are still existent on the destination bucket.
restic -r b2:important-backup-replica:/ check --no-lock --read-data-subset 1/10 --no-cache --jsonMounting the repository also lets me access the data just fine.
Cloud Replication seems like a good additional layer of security, if you decide that the overhead of a second B2 Account is worth it. Probably overkill for individuals, but appropriate if you use B2 on a larger scale to get another layer of redundancy.
