SSH Agent Forwarding Configuration with Ansible and Tower

It was a little tough to scrape together all of the details from the various sources I could find online, so I thought I’d bring together the end-to-end process for how to set up SSH and Ansible for using bastion hosts to access instances that don’t have a public address. The scenario is fairly simple: I want to be able to use Ansible Tower to configure instances in a private subnet of a VPC deployed to AWS via a bastion host in a public subnet.

To be clear, we’re looking to take a deployment in an arbitrary network or environment without direct connectivity to the AWS Environment. Tower and Ansible will use the Bastion Instance as a jump box to access instances in private subnets within a VPC deployment. The only requirements in this scenario are:

  • Tower is to be used to manage the process of configuring instances in a private subnet in a non-local network
  • The bastion instance in this configuration allows public ingress via SSH with a specific user and key
  • Instances deployed in the private subnet(s) are deployed with a different key than the bastion instance

There are two steps to setting up this process — first we need to create an SSH configuration for Ansible to use. Check out the following example:

Host 172.16.*.* ServerAliveInterval 60 TCPKeepAlive yes ProxyCommand ssh -q -A ubuntu@ec2–54–148–35–101.us-west-2.compute.amazonaws.com -i /etc/ansible/bastionInstanceKey.pem nc %h %p ControlMaster auto ControlPath ~/.ssh/mux-%r@%h:%p ControlPersist 8h User ubuntu

There are a few things to note here:

  • Using the Host section header, we’re scoping this configuration to only affect attempts to connect to 172.16.0.0/16 — you can modify this to fit the size of your VPC, but note that the syntax is wildcard-formatted and not CIDR notation
  • The SSH Key used to access the bastion itself was saved to a path that’s referenced in the ProxyCommand section of this configuration (/etc/ansible/bastionInstanceKey.pem). This file needs to be readable by the awx user (this is created by the Tower install process) and group as Tower will be executing the Ansible binaries to perform configuration updates:
sudo chown awx:awx /etc/ansible/bastionInstanceKey.pem && sudo chmod 400 /etc/ansible/bastionInstanceKey.pem
  • The ProxyCommand statement references the user and DNS or IP address for accessing the bastion which uses SSH Agent Forwarding (-A)
  • Note that we’re also using netcat (nc) to connect to the destination host (%h) and port (%p) as well
  • The User configuration specifies the user to be utilized when connecting to the instance behind the bastion (in this case, we’re using Ubuntu on both the bastion and the instances, so the user is simply ‘ubuntu’)

Next, you need to update the ansible.cfg file that’s located at /etc/ansible:

[defaults]
system_errors = false
hostfile = /etc/ansible/hosts
library = /usr/share/ansible
host_key_checking = False
ask_sudo_pass = False
pattern = *
timeout = 10
poll_interval = 15
[ssh_connection]
ssh_args = -o ControlPersist=15m -F /etc/ansible/ssh.config -q
scp_if_ssh = True
control_path = ~/.ssh/mux-%%r@%%h:%%p

To be clear, the main configuration necessary here is the addition of the [ssh_connection] configuration:

  • The ssh_args configuration includes the -F switch which references the ssh.config file we created earlier on
  • For more information on SSH arguments, check out the man page
  • Given that these instances are all potentially fairly vanilla, enabling scp as a replacement to sftp is a good idea (scp_if_ssh)
  • With the jump through the bastion, defining the control_path makes life a little easier in managing connection socket.

Alright! We should be all set. From here, you’ll need to save the private key that you’ll be using to access those private instance inside of Tower as a credential, and then when instantiating a given job template, you just select the key that you want it to use to access that instance (the SSH Agent Forward is nearly transparent at this point!).

On the ‘Credentials’ page, hit the ‘+’ menu bar item to pop up the credential add dialog:

In the credential add dialog, add your machine credential — more information on the deep-details of what’s here and how to use credentials in Tower can be found in the latest Tower users guide.

Finally, when you’re firing off job templates, you can use the credential that you created to connect to the instance itself! To get more of a walkthrough of this, check out the webinar Dave Johnson and I did on Ansible configuring the ELK stack on AWS using this technique.

Finally, a huge thank-you to Alex Bilbie (for getting me enough info to get this stood up the first time!) and Dave Johnson (for his help in getting our ELK demo nailed down).