The way to handle CI-owned ssh keys in Gitlab

I’d like to present the alternative approach to solving endless struggle with ‘deliver CI credentials in a safe and ergonomic way’.

The problem

We have a dedicated account for our CI (gitlab) to access staging servers. We need to provide it with credentials for accessing them. We don’t want to expose those credentials to non-administrators of the project (that means we can’t store them in plain text in git).

The old (bad) solutions

Solution 1: use passwords and save password as a variable in a project. Password access to servers sucks.

Solution 2: Save private key as a file variable in a project settings, use it as ssh-key to access servers. This one was fine to run jobs, but:

1. You need to handle ssh key in a very special way (LN/LF problems).
2. If you want to access server to fix something under CI account, you need to copypaste key from very uncomfortable window.

Alternative (mine) solution

1. Create CI key with password protection. Save key inside git (it’s safe to save the key because it’s encrypted). As bonus, you have public key for CI available to everyone (including any ansible plays in this repo). For this example the key is stored at .ci/ci
2. Save a passphrase for this ssh key as a gitlab variable (f.e. SSH_PASSPHRASE).

The next thing is hacky hack as we’re going to fight with ssh-add.

3. Create a simple script .ci/ci_pass:

#!/bin/sh
echo -n $SSH_PASSPHRASE

4. Add this into job in .gitlab-ci.yaml (in script) section:

script:
    - chmod 0600 .ci/ci
    - |
      eval $(ssh-agent -s)
      SSH_ASKPASS=.ci/ci_pass ssh-add .ci/ci

5. Add this into variables section

variables:
  DISPLAY: '1'  # we need this for ssh-add to use SSH_ASKPASS. 

That’s all.

The story behind ‘ssh-add’ and DISPLAY=1 is absolutely bizarre, and I really upset that I need to use hacks here to fight against ssh-add. Ssh-add won’t use SSH_ASKPASS variable until there is a DISPLAY variable, so we use value ‘1’ just to trick ssh-add to use our custom ssh-askpass program.

Nevertheless this trick works great. CI is happy, and the user can add the ssh key to own keyring if needed (provided it has access to passphrase).