Automate SSH Key Rotation With Ansible part 2

In the previous article we looked into SSH, SSH keys and the steps to follow when you are doing a key rotation. We also did a manual key rotation based on the steps. In this article we will be looking at how we can automate this process via Ansible and Bash scripts.
So far we have managed to successfully rotate our keys manually. It has not been a pain yet and that’s because we are using one server. It might be fun now but when your servers grow to hundreds or thousands, you will not be smiling at all.
So what do we do? Automate. Bash scripts have been around for ages and since it comes bundled with most linux machines, it will be our first approach to automate this process.
Automating the process with Bash Scripts
We will be using the previous article folder structure. If you jumped straight to this article, you can create a new workspace by running the command below. Full codebase for this article can be found on our GitHub repository.
Feel free to think tinker with it as you see fit.

Update the vagrant file with the snippet below.

Boot up the VM,

Now that we have that taken care-of, lets get started.
In our working directory create a folder and call it bin and in it a file named rotate-keys.
Keep in mind that all bash scripts file will be located in the
binfolder for the remainder of this article.

Update rotate-keys.sh with the code snippet below.

The approach we are taking here is to be able to run our file as a CLI. This will give us the flexibility to pass arguments making our script dynamic. With getopts program we can create options based arguments that we can pass to our scripts. This argument values are assigned to variables and used later in the automation process. You can get more details on getopts on linux man page.
To ensure that we always the correct values at all times. We have to do a sanity check. The sanityCheckensures that all the required variables have been assigned, if not, it assigns default values in their place. These variables are passed to three functions generateNewSshKeyPair which generates new key pair, addNewSshPublicKey which adds a new public key to the server and removeOldPublicKey which removes the old public from the server.
These three functions abstracts all the commands we did manually earlier into a bash script. when the script is run with the right arguments the rotation process will be done for us.
Before running this script ensure that you have added the private key to the ssh agent and your VM is in running state. If you are using the VM from the last article, add the new generated keys in the ssh folder. If you are using a new VM use the private key located in
.vagrant/machines/default/virtualbox/private_key.

To keep our workspace tidy, move the generate key pairs from bin folder to the ssh folder . We can then access the VM using the new key as shown on the snippet below.

Woohoo! Worked like a charm.
Even though we have been able to successfully automate this process, it is still hard to scale this script to the large fleet of servers. We can add a loop and more argument but the script will get more complex and hard to maintain. At this point, we need to look for a new alternative. Ansible.
Automating the process with Ansible Scripts
Manual process and bash script might have worked but when you are working with a large fleet of servers they may not be able to scale. At this point we need to find a tool that that is easy to work with and can scale to as many servers as possible and Ansible fits the bill. So let’s see how we can convert the above steps into Ansible scripts.
If you are not familiar with Ansible, I will encourage to read Getting started with Ansible on scotch.io, and the official documentation as well.
Setup Ansible configuration
Working with Ansible scripts in pretty easy, it enables you to structure your scripts in an easily readable and maintainable script using YAML syntax. Before we proceed to translate our bash script to Ansible scripts, we need to setup Ansible configuration.
At this point, I will assume you have already installed Ansible on your machine and you have a basic understanding of how Ansible works.
Run the command below to get started.

All the ansible related files will be located in the ansible folder for the remainder of this article.
Before we start writing and testing our script we have to ensure that Ansible understand the hosts it will be provisioning and what credentials to use. This is achieved by using ansible.cfg, ssh.config, and inventory.ini. Since we already have the inventory file, lets create other files.

In the files created above, modify them with the code below.

This file contains the information on how to connect to the host machine via ssh.

This file provides additional information to the ssh client on how to connect to the host machine.
The last thing we need to do is to tell ansible which hosts to provision. This is done via ansible inventory file. In this file we can place all the hosts that ansible will be provisioning.
Modify the inventory file with the code snippet below.

If you are using a different host you may need to change the IP address to match yours. At this point your folder structure should look as shown below.

Having setup ansible, let’s test if our setup actually works. Run the command below to ping the server.

Note: I recommend you use a new instance of vagrant for this section to avoid unpredictable outcomes due to changes made in previous article. To do this run vagrant halt && vagrant destroy && vagrant up. This will destroy the current VM and create a new instance. You might need to delete `.vagrant` folder before you run the above commands.
If you don’t get the same output, make sure your configuration is correct and you have registered your private key with ssh-agent.
Generate ssh key
Now that we have tested that our connection works, it’s time to generate the ssh keys. Since we have done this in the previous section I will not explain how it works but rather convert the process to an Ansible script. Ansible has a module called command which enables you to run bash commands. Since ssh does not have its own module we will need to run the command itself.
Modify ssh-key-rotation.yml with the code snippet below.

In the above snippet, we have used Ansible command module to run ssh-keygen command. We have used variable substitution in the command to make more dynamic. For now, I have passed the variables directly using vars directive but in you might want to abstract these into a different file for security reasons.
Since we need this command to be executed on our local machine it is important for us to use the delegate_to directive to inform Ansible of our intention.
And its show time !

One of the appealing attribute of Ansible is how it beautifully display real-time feedback as the script runs. When its done executing it also provides a summary of any changes that have been introduced by the script. And for us we should have our keys generated.

The good thing about Ansible is even if you run this script again, it will not generate new key since we already have one. This makes it easy to have consistent key even if the script is used several times.
Add the Generated SSH public key to the authorized_keys file
Now that we have the SSH key pair has been generated, we need to add it to the authorized keys file. As compared to the examples above, Ansible has a module called authorized_key. This module abstracts all the complexities of changing authorized keys.
In our main playbook file , let’s add another task. Modify the file with the code snippet below.

In the code snippet above we have added two tasks. One to store the value of public ssh key. In this task we are using lookup(‘file, path/to/file) method to fetch the value of the public key.
In the next task, we are setting the value to the authorized key file. It’s important to note that this task has to be done as root since we are modifying files only accessed by the root user. The authorized_key modules do require more parameters than we have specified for now we can do with the default as long as exclusive, user and key are specified.
Let’s run the script again.

Awesome, our script works. If you ssh into you vm vagrant ssh, and cat the authorised_keys file you should see a new key with suffix of domain@example.com.
Remove old ssh Public Keys
Now that we have added the key to our VM the next step is to remove the old key. How do we go about this? In the previous example, we had to ssh into the VM and edit the authorized_keys file using sed. The beauty of the authorized_key module is that we can add the key and remove the old one at the same time.
If you refer back to our script, you will see we added a variable called is_exclusive. This variable enables us to specify if we want our new key to be exclusive or it should be appended to the already existing keys. To remove the old key all we have to do is change the variable to yes from no.

Now, let’s test our script again.

If you try accessing the VM via vagrant ssh command, you will be prompted to enter a password as compared to the default seamless access. This is due to the fact that the script has replaced the default public key with the new generated ssh key. The only way to access this VM is via the new ssh key or password auth with vagrant as the password.
Let’s test if our new ssh key works.

Woohoo! it works
You can find the final version of this code on our Github account. Feel free to tinker with it as much as you want
Conclusion
In this article, We have looked into how we can simplify rotating SSH keys with different approaches, as you have noticed Ansible makes the process much simpler and easy. What we have done here is a simple example you can check my Ansible role on Ansible Galaxy with a little more features. What you can achieve with Ansible is limitless.
It’s time to get your tinker hat and simplify your processes with Ansible. If you have a question or suggestion feel free to leave a comment, happy tinkering.

