As a search tool that handles your data from other SaaS tools, clearly CTX has a strong focus on security.
So, for the first CTX technical blog post, I thought I’d follow on from the previous (non-technical) post about our attitude to the security of your data.
One of the layers in our security approach is encryption at rest. This means that your data is stored on disk in an encrypted form (though it may be available in RAM in an unencrypted form during use.)
I needed a solution that met these basic requirements:
- strong encryption — AES or something similar
- automated — no boot-time password entry, as there isn’t necessarily an operator on hand at boot time)
- simple to implement — solo founder, time constraints apply
- performance — minimal impact on indexing or search speed
- compatibility with our backup strategy (encrypted snapshots stored in S3)
After a period of analysis, reading documentation and trialling a couple of options I settled on EncFS. EncFS has some limitations, and there are some corner cases (not required for CTX) where I wouldn’t use it, but right now, for CTX’s current implementation, it fits the bill nicely.
Now, how to plumb it in?
Configuration management
All of the CTX infrastructure is managed using Ansible. Ansible is a configuration management tool that allows me to write code to control the creation and management of servers, instead of logging on and doing things manually.
Ansible uses “roles” and “playbooks” to organise units of work. Most of the important stuff goes in Roles (go read the Ansible documentation, it’s excellent).
Naturally, the configuration of CTX is already split up into playbooks, so the missing piece to employ EncFS was to find, or write, an Ansible Role that I could use to install and manage EncFS.
Of the limited set of existing options, some were overkill and the most useful looking one was specific to RPM based distributions, while CTX is a .deb setup.
So I wrote one.
If you want to try it, go have a look at the GitHub repository: https://github.com/rorygibson/ansible-encfs
Basic usage
It’s dead easy. Simply tell the role where you want your sensitive data mounted, and give it a passphrase (see below for more details on the passphrase_fetch_cmd)
- hosts: servers roles: - rorygibson.encfs vars: encfs: /var/sensitive-data passphrase_fetch_cmd: "echo 'MY PASSPHRASE'"The role will ensure that the encfs directory exists - in this case /var/sensitive-data - and will create the parallel storage area for the encrypted data in. Here, this would default to /var/sensitive-data_encrypted.
You then wire your application up to expect its data in plain-text in /var/sensitive-data.
When the volume is mounted and EncFS is running, your app can read the plaintext. When the volume isn’t mounted, that directory is empty and all that exists is the AES-256-encrypted data in the backing area.
A note on passphrases
Typically in an application deployment you have some secret data — database passwords, that kind of thing — that need to be present on the servers at runtime, but that are sensitive enough that you don’t want to have them lying around in plaintext for anyone to see.
Oftentimes it’s OK to have them accessible to the user that runs your processes — so in a common case, the tomcat user runs the JVM, and you have a configuration file on disk, readable only by tomcat with permissions 0600 (after all, if someone breaks in and they can see that file, they also have full control of the process, so this level of security is typically appropriate).
Passphrases for encryption at rest are a bit more secret again, and it didn’t feel right to have them on disk at all, if it could be avoided.
So how to do it?
Many modern hosting services (AWS, Digital Ocean, Google Cloud all included) have established the pattern of providing a Metadata Service. This is a Web Service, running in the hosting environment / DC, accessible to your servers, which you can make requests to for arbitrary key-value data (which you control either through an admin panel or a separate API).
This data is scoped to the server asking for it, so it’s usually called “instance metadata”.
Concretely, this means that there’s a service that’s very tightly locked down where you can say “give me the piece of data” and it will return it to you in real time, in memory. If we put the passphrase in there, then we can retrieve it on-the-fly.
This is the reason for the perhaps unusual variable declaration above.
If you want to retrieve your passphrase from instance metadata instead of hardcoding it (or putting it in an Ansible vault file, or whatever), then you can do it like this (DigitalOcean example, the metadata server IP is documented):
- hosts: servers roles: - rorygibson.encfs vars: encfs: /var/sensitive-data metadata_server_ip: 169.254.169.254 passphrase_fetch_cmd: "curl --silent http://{{ metadata_server_ip }}/metadata/v1/user-data "On DigitalOcean, the instance metadata User Data above comes back as plain text. The command above assumes that this is the passphrase. If you’ve got other data in there, you’ll need to parse it or whatever.
If you were on Google Cloud, you’d get a JSON doc back. AWS is different again. YMMV.
Note: if you store your passphrase in instance metadata, you should probably encrypt the passphrase itself. This is out of the scope of this blog post though, naturally, the Ansible role we use in CTX contains this feature — it’s just not present in the lightweight version released alongside this post.
(This is the nature of security — it’s turtles all the way down)
Originally published at getctx.io on July 20, 2017.
