Securely storing secrets in Git

Secrets management and secure storage of secrets as code is always an interesting topic. There are various tools that exist for storing secrets in a Git repository. We will look at several of them that exist and which ones are better or the preferred approach. We will look at several different tools including transcrypt, git-crypt, git-secret, and SOPS. I currently use or have used several of these in the past and have contributed to some, so I will try to share some of my experiences with them.

Before we dig into each of them, I will say the most flexible and preferred one is SOPS. I will dig into more details of why it’s the best option in my opinion and some recommended configurations for using it. You can jump directly to that SOPS section

Sealed-secrets is also a common option but it is not covered here because it is restricted to only Kubernetes. Sealed-secrets functions by decryption only by the Kubernetes cluster. The user encrypts a secret that only the cluster can decrypt using a public/private key pair.

Repository disclaimer

When storing any secrets in a repository even when the files are encrypted, other protections should still be taken. Any repo storing secrets should be a private repo and only grant access to those who need to know. While most of the ciphers in use today are secure when used properly, that does not mean that they will always be. There have been attacks and weaknesses found in ciphers in the past and current ciphers are no exception.

All the tools here are using well-known and tested tools or libraries for encryption. All of them are generally secure when pushed to the remote repository in their encrypted format given that the passwords or private keys are not compromised.

Transcrypt

Transcrypt is transparent encryption for git using OpenSSL as the encryption tool. The default cipher is AES-256-CBC but can use any cipher supported by OpenSSL. It uses .gitattributes for determining which files to encrypt/decrypt so you can only encrypt/decrypt a subset of folders or files based on name, path, or regex.

$ cat .gitattributes
sensitive_file filter=crypt diff=crypt merge=crypt
$ cat sensitive_file
super sensitive secret

But once pushed to the repository and viewed by someone that does not have access, the contexts are encrypted.

$ cat sensitive_file
U2FsdGVkX19KGqMWib3gssm5rnP5LnujJ/G4AvVmvbq2ms5LJ9YmBb772kD85GM+

This means it decrypts files on clone/pull and encrypts files on push automatically and requires no additional commands by the user. Transcrypt also supports showing the diff when doing a git diff to see changes between the files.

$ diff --git a/sensitive_file b/sensitive_file
index f0ff5de..0211c3b 100644
--- a/sensitive_file
+++ b/sensitive_file
@@ -1 +1 @@
-super sensitive secret
+new super sensitive secret

Pros:

• Seamless to the user and does not require additional commands to update files
• A bash script that uses commonly available system tools so it can be included in the repo for all users to easily access.
• Encrypts the entire file contents

Cons:

• Transcrypt stores the password in plain text in the .git/config file
• The files are stored in plain text on each computer where the repo is checked out and transcrypt has been configured.
• If the .gitattributes is incorrectly changed it can result in sensitive files being pushed to the remote repo in plain text. This would not be as common with single file entries but in a more complex regex configuration.

Git-crypt

Git-crypt is similar to transcrypt and transparent encryption and depends on OpenSSL as well but uses AWS-256-CTR as the encryption algorithm. git-crypt also uses .gitattributes for determining which files to encrypt/decrypt like transcrypt.

$ cat .gitattributes
sensitive_file  filter=git-crypt diff=git-crypt
$ cat sensitive_file
super sensitive secret

Git crypt stores GITCRYPT at the beginning of each file followed by the binary encrypted data when pushed to the repo.

$ cat sensitive_file
GITCRYPT�b"َp:_��g��lQ0\�_�(��:1�
                                 ���rT�

This also decrypts on clone/pull and encrypts files on push so the user does not need to run additional commands. Git-crypt both support showing the changes locally with git diff as well.

Pros:

• Seamless to the user and does not require additional commands to update files
• Uses a better cipher than AES-256-CBC
• Encrypts the entire file contents

Cons:

• Stores the password in text files in .git/git-crypt/keys/ folder
• Uses a fixed key length when generating the encryption key
• The files are stored in plain text on each computer where the repo is checked out and git-crypt has unlocked the repository.
• If the .gitattributes is incorrectly changed it can result in sensitive files being pushed to the remote repo in plain text. This is the same as transcrypt

Git-secret

Git-secret uses GPG instead of OpenSSL for encryption and is not a transparent solution like the others. It uses AES-256-OCB Git-secret stores the plain text and encrypted versions as 2 different files. The plain text is stored as the original file name and the encrypted version is stored with the extension of .secret. You must also add the plaintext filename to .gitignore to prevent it from being checked into the repo. To add someone with

$ cat .gitignore
.gitsecret/keys/random_seed
!*.secret
sensitive_file
$ cat sensitive_file
my secret file

Since git-secret stores the encrypted version in a different file, we need to check that file to see the encrypted contents. It stored the .secret version as an encrypted binary file.

$ cat sensitive_file.secret
��PZ�_yɳ�!�0�K�t���D�1'o|��1-��H��m�ȆT������-�3�` ����v���&d�KA�ͽ΋
<'4����sI�8<MO���V�MiB����Gi�C9��ίr2~R��1��_@�PWF7�n=�
                                                       �z<]���v�Р

Pros:

• Uses AES-256-OCB which is similar to CTR but is an authenticated encryption (this is dependent on the GPG version you are using)
• Uses GPG to share to multiple people natively so a password to protect the private key can (and should) be used.

Cons:

• Maintains a 2nd copy of each file so there is an easier chance of accidentally checking in a secret file.
• Must add each file to .gitignore so that the source file is not tracked
• Must manually run git secret add $filename (wildcards are supported) for each file you want to encrypt

SOPS

SOPS stands for Secrets OPerationS and was developed as an open-source project by Mozilla. SOPS is different than several of the others as it does not encrypt the entire file and only a subset of the contents which are normally considered the sensitive part. This is also the only solution that easily supports storing secrets with different permissions per secret. The encrypted secret contains metadata about who can decrypt the secret so you can have thousands of secrets each with different permissions if needed.

There are really a few features that make SOPS stand out above all the others.

• All secrets are kept encrypted on disk all the time and you only decrypt the file when you need to e.g sops -d $file.yaml
• SOPS supports Shamir’s Secret Sharing Scheme (covered in more detail below)
• Integrations with Terragrunt, helm-secrets, helmfile, Flux, and others.

We can take an existing plain text file and convert it to a sops encrypted file with sops -e -i $filename. SOPS also has built-in editor support, so you can edit a sops encrypted file by just doing sops $filename and it will open the editor (like vi) with the decrypted contents.

$ cat my_secret.yaml
username: my_username
password: my_secret_password

Once converted to an encrypted file, it encrypts the values of each YAML field in the file, while leaving the keys in plaintext.

$ cat my_secret.yaml
username: ENC[AES256_GCM,data:sli5qzAq3tQGGjk=,iv:e3+riN+PjCwJErUDyMqah+QYmLqXOnBzBiraZAakjJA=,tag:lYE/6fVhS5WbzqzQY1rkCw==,type:str]
password: ENC[AES256_GCM,data:vGcOkuV/mx3GCLk+heqAUp3e,iv:/77z12Dm6XxAkJ6jbo5wkGWMvRfHECTUEOZgYN3gFZo=,tag:qDkaHadGNjDxLoUuY8lhXQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2022-07-18T21:38:38Z"
    mac: ENC[AES256_GCM,data:9IVSULD9o4qeF3fTgXGkJ/Oxm9bmZC08VfuAy41NIXhFtOEuTqKBDNEtQeO27RYKxWZWBBlKnblTdY0Uet9ranQWW586O/uB/Gxnrm9I39p21TGHRv6KVJDRx37lR3IxVdq402N9YkrlHZPjX4WfwsnUCoHqdU6h8+yL48/8EO0=,iv:oZGeCU6zdjq2GQ5tkIdleq0YuoE5QTUTi4q+PnkRZeo=,tag:/kTdQ7czhYyOw4HZ+XvdnA==,type:str]
    pgp:
        - created_at: "2022-07-18T21:38:38Z"
          enc: |
            -----BEGIN PGP MESSAGE-----
hF4D7Kht36Nmso4SAQdADtCBr7qHsLWUcj3JIW6cPGNrzrx0oVV7u9qKuQI21jUw
            bdMJwE46oJIVzZ3uBYP3vClOvIiOOgVYm7IZERuyzz35uMYYEIgA4DTk6zj0bdoh
            1GgBCQIQ+LbJAENuPZa6m8vzoPA4P++uScf2dybKUMDHSTEwGR+bgo4xXKOyinxh
            ZlgKXZfoQw5AtK6t7l7hMBt9a5T4lP9rEfB6z3j2s8xwmDbSk1gEJCm7SWWBd6J7
            0m5viZUZP9YaUg==
            =aaAK
            -----END PGP MESSAGE-----
          fp: F69F7C85BCBEE72C769BE27DC73DF6C795316F83
    unencrypted_suffix: _unencrypted
    version: 3.7.3

The file stays encrypted until you are ready to actually use the file. Then running a decrypt on the file outputs the contents.

$ sops -d my_secret.yaml
username: my_username
password: my_secret_password

Pros:

• Uses AES-256-GCM is an industry-preferred cipher by due to being authenticated encryption and is part of TLS1.2/TLS1.3. Intel has also worked to optimize GCM on intel processors making it faster than several of the other ciphers
• Supports Multi-key operations via Shamir’s Secret Sharing Scheme (See below)
• Files stay encrypted on every machine
• Supports different users/keys per secret
• Supports multiple different types of keys including HSMs like AWS, GCP, and Azure KMS solutions as well as age and gpg encryption.
• Third-party integrations

Cons:

• SOPS works great with JSON and YAML files but can have problems from time to time with binary data
• Since SOPS only encrypts the value and not the key, it can leak data about the structure or contents of the file. As long as this is accounted for this shouldn’t be a major issue.
• Workflows must be updated to handle runtime decryption instead of just reading plaintext files

Shamir’s Secret Sharing in SOPS

One of the major benefits of SOPS is the usage of Shamir’s Secret Sharing Scheme. PassGuardian has a great online tool that you use to can see it in action. In layman’s terms, it takes a secret value and splits it into a desired number of shares and you configure a threshold, which is the minimum number of shares that must be present in order to reconstruct the secret.

If we take a secret ofmypasswordand split it into 5 shares with a threshold value of 3, we get

801740b2092d05dd9a2b4bbc2e9de1e9cf9a8a0bde090
802ac91f42d3d35bdd2a7884179a2e68fb135548560cb
803d99ab0bf9f680b706433f0900ff87248edf4418036
8044719390ab46c60affba63e10ae9e841a7d8c250e25
80532127d981631d60d381d8ff9038079e3a52ce1eed8

With these values, it would require ANY 3 (or more) values in order to properly reconstruct the original secret. Any less than the configured threshold and reconstruction fails.

This is implemented in SOPS natively and allows a mix and match of keys from different sources and split the secrets between them. So in the event of a compromise of the single key, the secrets are not compromised.

creation_rules:
- path_regex: /super_secret/**
  shamir_threshold: 3
  key_groups:
  # 1st key group
  - pgp:
    - F69F7C85BCBEE72C769BE27DC73DF6C795316F83
  # 2nd key group
  - kms:
    - arn: arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
  # 3rd key group
  - age:
    - age1l39ptl7udyfqavym4z6hst977xrftxexk9wuzu3jhfyssm0scssssjkdcu

This example, uses a PGP key, an AWS KMS key, and an age encryption key. The number of Shares is the number of key groups listed and the Threshold is determined by the value set for shamir_threshold. Since there are 3 key groups and 3 shares, we know ALL keys must be present in order to decrypt this value. So 2 keys locally on the machine AND valid credentials to talk to AWS KMS service.

Because of this setup, there are multiple keys that would need to be compromised before the secrets could be decrypted.

Summary

Storing secrets in Git repositories is becoming more and more common but people must ensure appropriate controls must be implemented to protect secret data. With employee turnover and things like BYOD (Bring Your Own Device) policies, ensuring that secrets are kept safe is becoming more difficult.

Using ANY of the options listed here is certainly better than storing secrets in plain text but when it comes to protecting secrets, SOPS just shines above the rest with the choice of ciphers, many different key options, Shamir’s Secret Sharing Scheme, and never storing in unencrypted form on disk.