Automate SSH Key Rotation With Ansible part 1

Thomas Nyambati
Sep 6, 2018 · 9 min read

This article covers a brief introduction on ssh, ssh keys and the manual process of rotating ssh keys. It serves as a pre-requisite for the second part of this article. If you want to jump straight to implementation with ansible and bash scripts visit the second part of this article.

If you have worked with virtual machines, you’ve probably heard of the term “SSH”. So what does it actually mean? SSH — Secure Shell is a network protocol that provides system administrators a secure way of accessing remote servers. It provides strong authentication and encrypts communication between computers connecting over insecure networks such as the internet.

System administrators rely on this protocol to manage systems and applications remotely. Through SSH, you can log into another computer over the network, without being in the actual computer location. This means that as a system admin, you are not limited to doing maintenance at the actual server location and, you can manage your servers from any location as long as you have a connection to the network.

How does SSH work?

The SSH protocol uses the client-server model to authenticate servers and encrypt the data shared between them. For a connection to be established, the remote machine must be running a SSH daemon which listens for connections on a specific network port, authenticates connection requests and spawns the appropriate environment when provided with correct credentials.

The SSH clients can authenticate using passwords or SSH Keys. As much as passwords are easier to use, they can be easily compromised by bots trying gain access to remote servers. For this reason, it is recommended to use SSH key-based authentication at all times.

SSH Keys are a set of matching cryptographic keys used for authentication. They entail both private and public keys. Public keys can be shared freely while private keys must be vigilantly guarded and never exposed.
To successfully connect to a remote server, a user must have the private key on the local computer and the public key on a remote computer. Public keys are stored in the ~/ssh/authorized_keys. This file contains all the public keys that are authorized to log into the server.

Why does rotating ssh keys matter?

If you are working with a large team, it’s reasonable to have each user have a key they can use to access the servers. With team members working on different projects, comes the possibility of constant changes in personnel and access privileges. This constant change exposes your infrastructure to vulnerabilities such as compromised keys from current users, off boarded ones or idle active keys that can be used to exploit the system.

To minimize such risks, it is recommended to always rotate the SSH keys based on a defined cycle. The cycle is usually arbitrary based on how your organisation handles security and audit policies. During rotation, new keys are generated and switched with the old ones, inactive keys are also removed. This constant change of keys decreases any chances of compromised keys being used to get access to the remote servers.
It is important to note that as much as rotation of ssh keys reduces the chances of your system being compromised, it is important to put in place security policies on how ssh keys will be shared and secured while in possession of the users.

Rotating ssh keys

A SSH key rotation process involves three simple steps,

  • Create a new ssh key
  • Add the public key to an authorised keys file
  • Test the new keys and replace the old ones

In this article, we shall be looking at how to implement these steps. We’ll also go a step further and automate this process with Ansible and bash.

Pre-requisites

It is my assumption that you are familiar with the Linux command line and Ansible as well. If you have not done either of the above I will recommend you brush up before proceeding. I also recommend reading the official Ansible documentation and this Getting started with Ansible article on scotch.io

In this article, we shall be using the following technologies

To install the above tools on MacOs, run the commands listed below, for other operating systems, refer installation instruction from the respective websites.

Setting up vagrant

The instructions in this tutorial can be used on any hosted server, on the cloud or on a data center. On for this tutorial, we’ll be using Vagrant machines to simulate remote servers since it’s simple to work with and free. To confirm your vagrant installation was successful, run vagrant version.

Now that we have our vagrant installation working, we can go ahead and set up our working directory.

This command will create a ssh-key-rotation directory and initialize vagrant, by creating a Vagrantfile. Vagrant uses this file to provision and configure its virtual machines. The initial vagrant file comes with defaults, modify the file with the code snippet below.

This configuration defines a private network accessible on IP address 192.168.33.12 and the hostname as octo. Boot the VM by running vagrant up.

Note: If this is your first time using Vagrant and Ubuntu/trusty64 box, it might take some time downloading the box depending on your internet connection. This might the best time to get a cup of coffee.

Once the command is done executing, your VMs should be ready to go. You can check the status by running vagrant status.

Now that our VM is up and running, we can ssh into it using vagrant ssh <hostname>. in our case octo.


Generating SSH Keys

When vagrant boots up VMs by default it generates ssh keys for each each one of them. These generated keys enable us to ssh into the VM without use of a password . The first step of rotating keys, is generating new keys that would replace the old ones. We can generate new keys through ssh-keygen, the command below demonstrates.

This command will generate a new key named octo-ssh-key and octo-ssh-key.pub. Move these two files into the ssh folder to keep things tidy.

If this is your first time using the ssh-keygen command, you might be wondering what the options passed to the command actually do. Let’s jump right into it, shall we?

  • -t Specifies the type of key to generate, this can be DSA or RSA
  • -b Specifies the type of bit encryption to use, I recommend using 4096 as the minimum.
  • -C This option allows us to add a comment to accompany out a public key for easy identification.
  • -N Specifies the passphrase to encrypt the keys with.
  • -f Specifies the directory or path the keys will be stored.

You can learn more about the options that can be passed to this command on the Linux Man Page at die.net.

Add the Public key to the authorized keys file.

After generating key pair, the next step is to add the public key to our Virtual Machine. One of the obvious options would be to ssh into the Machine and edit the authorized keys file. As much as that would work, it wouldn't be very scalable when working with alot of VMs

Lucky there is a there is a command that we can use to copy our public key to the host machine. ssh-copy-id logs into the remote server and modifies the remote users’ .ssh folder including the authorized_keys file. When the -i flag is specified, it will copy the public key and append it into the authorized_keys file.

When this command is done executing, ssh into the VM, and open the authorized_keys file in the .ssh folder. You should be able to see the public key we just generated appended to it. Excellent.

Test the new key and replace the old ones

Having added the public key to our server, let’s test it and ensure that we can use it to connect to the server. First thing first, let’s add our private key to the ssh-agent.

When prompted with passphrase type the passkey we used to generate the key. The passphrase we used is `the-pass` which is very weak, insecure and therefore should not be used in a production setting. Since this is a demo, we can get away with it. At all times ensure you are using a strong and secure passphrase to generate your keys.

Now that we have added the private key to the ssh-agent let’s attempt to ssh into the machine, run ssh user@[hostname/ip] in our case, ssh vagrant@192.168.33.12.

Woohoo! yeah, it worked.

We have successfully generated new ssh keys, added the newly generated public key to our VM and also confirmed that we can actually access the VM using the new ssh key. At this point the old key is still in use and its about time to remove it from the system.

We could ssh into the VM and edit the authorized key file, but we ascertained that it’s not a scalable option. sed to the rescue! If you have used sed before you can attest that it’s a powerful editor. Below is a template of how we can use sed to remove the old key from the system.

For this command to work, it requires a unique identifier to match our key. Remember when creating the ssh keys we specified a comment, using this comment we can uniquely pick the key we want to remove from our server. All vagrant keys have a comment of vagrant, thus qualifying as our unique identifier. We can now substitute the command with actual values and it will be as below.

If you are wondering what .bak means well when we use .bak sed will create a backup for our keys, if you don’t need them you can do away with it later. Let’s stick with it for now. To run this command against our VM, combine our commands as below.

When it’s done ssh into your VM and check the authorized_keys file. You should notice that the old public key has been removed. Awesome right?

If you try to ssh into the VM via vagrant ssh octo, it will prompt you for the password since it could not find the default public key on the server.

Conclusion

So far we have talked about SSH , SSH keys, SSH keys rotation and explored the steps to take when rotating keys. However, its important to note that the steps we have taken can also be used when issuing new keys to new members and removing old unused keys, application depends on you.

In part 2 of this article we shall be taking a more hands-on approach where we’ll automate these steps using Ansible and bash scripts. See you then.

Rack Brains Africa

Our mission is to provide training and awareness about DevOps, its ecosystem and, the importance of designing and building effective products.

Thomas Nyambati

Written by

Full stack developer | DevOps Engineer |Author |Tech Evangelist | Founder @rackbrainz

Rack Brains Africa

Our mission is to provide training and awareness about DevOps, its ecosystem and, the importance of designing and building effective products.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade