Docker Notary: Very TUF, but devil is in the detail!
Docker Content Trust
Software distribution systems need a mechanism to prove the trustworthiness of the software they are distributing, like providing a signature to certify the image and its publisher, prove its integrity, freshness, etc. Traditionally software image distribution systems sign images using GPG keys. Signing an image does prove the identity of the publisher and the authenticity of the image being published. However, only signing does not provide several security mechanisms like the ability to survive a compromise of the signing keys.
Docker fixes this problem by providing an underlying framework called Docker Content Trust which verified images before running the container. Not just that, it constantly polls the Docker registry for any updates in the image and if there is, it is fetched before launching the container. The engine behind enforcing and managing this trust is Docker Notary, a Docker service which is implemented based on The Update Framework (TUF).
TUF is a specification for secure software distribution. It establishes trust using a hierarchy of roles represented by asymmetric keys and metadata signed using these asymmetric keys. A key hierarchy of root key, snapshot key, timestamp key and target keys together provide several security guarantees like freshness guarantees, survivable key compromise, etc. The root key is the root of all trust that signs the root metadata. The snapshot metadata is a collection of hashes of all the files for a given snapshot of the image.
Every image that is published is signed using a combination of server side and publisher’s end keys to ensure that only a breach of server keys is not enough for attackers to tamper an image and publish a malicious image. Notary also suggests storing the root key offline to prevent disclosure.
A small gotcha, though!
Among the above metadata, timestamp metadata is meant to ensure freshness of content and has a very short expiry time. It is signed using a timestamp key which is always online for convenience because it is fetched frequently by the client for freshness check by checking timestamp. If nothing has changed, the image on the client side can be used as is.
In the implementation of Notary, the server side private keys are stored in a keystore database as cipher text encrypted using AES symmetric keys. However, this AES symmetric key is stored in clear text in an environment variables on the notary signer service. Environment variables are proven to be a very unsafe location for storing secrets in clear text as they do not come with any access controls. This can be the weakest link in the chain of keys, making the whole Docker registry as safe as this one secret key.
The threat
The threat discussed here is a freeze attack as a result of disclosure of weakly stored symmetric key:
· The attacker gets access to the weakly protected AES symmetric keys.
· The attacker fetches the cipher text of timestamp key from the keystore database and unwraps the AES wrapped private key.
· Re-signs the timestamp metadata with a very long expiry.
· The client polls the timestamp metadata expiry and confirms it has not expired and is still valid. So, there is no need to re-fetch the image.
· The client will never get updated till the timestamp expires, thus leave it stuck with an old and potentially vulnerable version of the image.
The solution
For a service like Notary which has to be available all the time and should come up automatically without any human intervention, storing this final secret securely is not a very easy problem to solve as there will always be one final secret to maintain.
The proposed solution here is to ensure that the symmetric AES keys, the final secret in the whole chain, are stored using secure storage mechanisms like HSM/Vault to prevent disclosure but still be available on demand at runtime.
However, it is beyond the scope of Docker Notary to support all/any possible means of external secure storage. Also, if Notary code base has knowledge of external mechanisms, it will have to be re-built and released for any new secure storage mechanism support.
Notary will need to provide a plugin interface for external Vault/HSM integration at runtime using external plugins that confirm to this plugin interface. This also makes Notary future proof to any future methods of secure storage.
Below is a diagram that describes the proposed solution:
In the diagram above the proposed change is only to introduce a plugin interface. All plugin development is outside the scope of Docker Notary.
Summary
Docker Notary goes a long way in ensuring secrets are secured, but misses that one last secret. This solution helps store the secrets in hardware, thus solving the last secret conundrum. Here is an issue opened by me with Docker Notary to create a plugin interface for external secure storage integration: https://github.com/docker/notary/issues/1091
Here is my pull request on Docker Notary’s github repository to bring in an interface to accommodate plugin interface for dynamically loading external storage plugins at runtime: https://github.com/docker/notary/pull/1177
