QNAP Glacier IAM User
I prefer my functional accounts to be restricted to the minimum set of permissions required to do the job. Not only is it a recommended best practice but it also limits the blast radius in case something goes wrong. As I couldn’t find any documentation on the minimum set of IAM permission required by the QNAP Glacier app, I started my own investigation. This article summarises my findings that are also available as a Terraform module on https://github.com/dumrauf/qnap_glacier_iam_user.
The Importance of Pictures
I love taking pictures. When on holidays or in everyday life. Pictures transport me back to the place they were taken. They bring back memories of that one picturesque sunset over some of the most spectacular scenery on earth. Plus, all the anxiety on the steep three hour uphill slog in the middle of the night that followed shortly after. I’ll never forget those seven river crossings that were a mere side note in the hiking guide. But then, look at that view!
Pictures are my everyday teleporter. My time machine. My short escape. Naturally, they mean a lot to me.
The Blessing of High Resolution Cameras
I’ve owned a camera as long as I can remember and I was there when the transition from analogue to digital happened. Digital photography has really come a long way in the last decades. From grainy pictures best viewed at fingernail size in the dark to modern 35+ megapixels (MP in short) miracles that paint a stunning high resolution image of the world with silicone and photones.
And these are just your standard full frame cameras available on the high street. If it has to be the cutting edge of what’s possible, then there are always medium format megapixel monsters from the likes of Hasselblad and Co. which easily crack the 50+ MP barrier and go up to 100 MP.
The Curse of High Resolution Cameras
However, this increased resolution comes at a price — storage. While you can play with formats and settings to reduce the resulting file size, given that you’ve spent all that hard earned money on a high resolution camera in the first place, you probably end up shooting in something like JPEG Fine mode. RAW if don’t fancy compression. TIFF if storage hardly means anything to you.
A Lot Of MBs for a Snap
Not too long ago (and still available if you can get hold of one), a 12 MP camera used to produce on average 6 MB images on half way decently lit subjects. A modern 35+ MP camera easily churns out images of an average size of around 30 MB. You could significantly lower the size of the images by just shooting in pitch dark environments but what’s the point of 35+ MP series of pitch dark images? Oh, and if you really want to burn through storage, start filming in 4K with a high bitrate.
36 Snaps on a Roll
Back in the film days, an image had a real instant cost. You had to buy the film before you could start and those things cost real money. Plus, take up space in your luggage. Oh, and there was no delete option in case you didn’t like the picture (there was no option to check, remember?!). Now, all of that has changed with digital photography. No more need for films. Delete, if you don’t like the snap. Pop in a new storage medium to continue, most likely an SD card these days. Get them by the dozen (or rather not, as speed and reliability can matter and drive up the price, depending on your use case). It’s not uncommon to see people set off on extended trips with half a Terabyte of available storage. That’s 500 GB. Imagine that as a stack of printed sheets of paper…
Securely Storing Those Precious Photos
While it’s perfectly reasonable to store the precious pictures on an SD card inside or outside the camera while on the road, it’s not really an ideal long term archiving solution in my opinion. Plus, it obviously limits the number of pictures you can take in the future, unless you’re invested in SD cards. So, they have to come off. But where to?
A Bunch of Drives on My Local
A local machine might be a good option. But it’s hard drive could fail. So, we need a backup to another drive. This means backing up the main drive regularly which can be a bit tedious and more often than enough I ended up forgetting about it. Brilliant. The backup was out of sync. Not really ideal in a disaster situation. So, is there a good tool for this job?
A Bunch of Drives in my Network
It turns out that a half-way decent NAS with at least two bays is well suited for the job. Put it into some RAID mode, depending on the number of bays and personal preference, and you’re good. As usual, buy drives of identical size from different manufacturers to lower the risk of outages due to manufacturing errors.
A Bunch of Drives in the Cloud
However, at this point, all your precious picutures are most likely stored in a single physical location. A power surge or a natural disaster could destroy your precious photos in an instance. We need more replication. So, buy another NAS? This could be an option but it’s only a tiny bit better than what we have so far.
If we securely store the pictures in the cloud, then they are backed up off-premise and can be retrieved in the event of a disaster. Plus, we get an SLA around how securely our precious pictures are stored and how long it takes to retrieve them again. All we need is an affordable cloud storage provider and an application that performs regular backups to the long term cloud storage of choice.
An answer to the first problem can be AWS Glacier. It’s AWS’s long term archivig solution trusted by numerous organisations around the world. It comes with a rich suite of features, including access controls via IAM policies and vault access policies as well as immutable data retention policies. The latter can be used to enforce regulatory requirements via their apply, test within 24 hours, and apply for good mechanism. Note that once applied, vault lock policies cannot be changed. It does what it says on the tin. Locks the vault policy.
And all of that at a pretty competitive price point. Storing about 1.5 TB of data in AWS Glacier sets me back around USD 6 per month. On top of that goes a one-off cost of around USD 20 for the initial upload. The only catch is that retrieving the data can take up to several hours. Oh, and cost around USD 150. However, if my NAS just went up in flames, then USD 150 seem to be quite reasonable in comparison to kissing all my precious pictures goodbye forever.
The QNAP Glacier App
The answer to the second question depends on your NAS. On my QNAP, there is an app called QNAP Glacier which pretty much does what it says on the tin. It expects a pair of access and secret access keys in the case of AWS. From there on, you just point it to the AWS Glacier vault you want the app to use, select the folders to be synchronised, a (periodic) execution time, and lean back and see the bits’ digital clones wander off to AWS Glacier.
Creating an IAM User for the QNAP Glacier App
In order to arrive at a set of access and secret access keys, we need
- An IAM user
- An access key and a secret access keys for the IAM user
- A policy granting minimum permissions to the IAM user
While we’re at it, we may as well create
- An SNS topic
- The AWS Glacier vault and have it send all
InventoryRetrievalCompletedevents to the SNS topic created in 4.
A First Attempt at a Policy
While the first two resources are fairly straight forward to create, the policy turned out to be a tricky one. The original idea was to allow the IAM user to perform all CRUD operations on archives but limit the operations on the underlying vault to ready only. Following the excellent Amazon Glacier API Permissions: Actions, Resources, and Conditions Reference documentation, the first attempt at a policy turned out to be
Here, the value of
Resource is tailored to the specific region, account, and vault being used.
A Second Attempt at a Policy
The infrastructure required for the QNAP Glacier app is a nice example of what can be built in AWS in a short span on time. The parts work really well inside the QNAP app as well, as it happily accepted my pair of access key and secret access key. However, when testing access to the actual AWS Glacier vault, the QNAP Glacier complained that
Because the vault inventory is updated once a day, you cannot use the exist vault for a backup job. Please create a new vault.
This makes sense when considering that AWS Glacier vault inventories are generated approximately once a day. From the QNAP Glacier app’s perspective, it’s simply the safest way to create a brand new AWS Glacier vault and use that as that’s more likely to be empty.
The above limitation implies that the
glacier:CreateVault permission is required by the QNAP Glacier IAM user. That's not really ideal, as the Amazon Glacier API Permissions: Actions, Resources, and Conditions Reference documentation suggests that the
glacier:CreateVault permission can only be granted on a global scope. This then implies that the QNAP Glacier app could potentially create lots of AWS Glacier vaults.
With a little experimentation, it turns out that the
glacier:CreateVault permission can be restricted to a specific AWS Glacier vault. The final IAM policy being used for the QNAP Glacier IAM User is hence
Again, the value of
Resource is tailored to the specific region, account, and vault being used.
The above policy eventually gets the job done and allows to supply access and secret access keys to the QNAP glacier app. However, the assumptions made by the app somehow limit the ability to leverage other functionalities of AWS Glacier vaults. I have yet to find the option that allows me to configure notifications, set vault access policies, set vault Locks, or tag vaults that are being created via the QNAP Glacier app.
Creating a QNAP Glacier IAM User
There are many ways to create access and secret access keys for a QNAP Glacier IAM User. The AWS console provides an option to create and configure a corresponding IAM user and CloudFormation can be used for the job as well. In general, Terraform seems to be well suited for the job of programmatic infrastructure deployment. This is what we are going to use here.
qnap_glacier_iam_user Terraform Module
Before you can use the Terraform module in this repository out of the box, you need
After applying the Terraform module in the repository, you get an IAM user with corresponding access key and secret access key that can be used in the QNAP Glacier app. Here, the IAM user is only permitted to work with archives in an AWS Glacier vault specified in the input variables. Modifying or deleting the vault is not permitted.
The input variables for the module are defined in https://github.com/dumrauf/qnap_glacier_iam_user/settings/example.tfvars as
region = "<your-region>"
shared_credentials_file = "/path/to/.aws/credentials"
profile = "<your-profile>"
qnap_vault_name = "<your-QNAP-Glacier-vault>"
Here, you need to replace the example values with your settings. Note that you also need to update the
qnap_vault_name to the name of an AWS Glacier vault that does not exist. Moreover, the current value is not a valid input
Make sure that the
qnap_vault_name refers to an AWS Glacier vault which does not exist. The QNAP Glacier app assumes that it can create the given AWS Glacier vault.
Initialise Terraform by running
As a best practice, create a new workspace by running
terraform workspace new example
The QNAP Glacier IAM user can be planned by running
terraform plan -var-file=settings/example.tfvars
and created by running
terraform apply -var-file=settings/example.tfvars
The module has two outputs, namely
qnap_glacier_user_secret_access_key which are the access key and secret access key of the newly created IAM user.
The access key and secret access key are intended to be used in the QNAP Glacier app as is.
The QNAP Glacier IAM user can be deleted by running
terraform destroy -var-file=settings/example.tfvars
Note that this also removes all access to the AWS Glacier vault for the QNAP Glacier app.
Using Terraform to Tweak the Vault Configuration
While Terraform has the option to pull resources back under its control, it seems to be widely accepted that it’s better to start out with Terraform creating them in the first place. So, it’s possible to tweak the configuration of the AWS Glacier vault created by the QNAP Glacier app but, at least for me, the side effects have yet to be unearthed.
qnap_glacier_iam_user Terraform module introduced above makes it easy for me to create IAM users tailored to the QNAP Glacier app. This allows me to back up my precious pictures to AWS Glacier with relative peace of mind. While this works for me, it may be different in your case. Feel free to leave a comment or contribute to the Git repository, so we can make things better for everyone. Thanks!