Ansible External Secure Access & Dynamic Inventory on Google Cloud Platform (GCP)
As a Cloud engineer, I frequently orchestrate Proof of Concept (POC) deployments on Google Cloud for a diverse range of clients. A pivotal part of these POCs is setting up private Google Kubernetes Engine (GKE) clusters. Ideally, direct access to these resources from within the same network would simplify the process, but real-world scenarios aren’t always that accommodating.
Many times, the stringent security protocols of my clients or the unique network configurations prevent me from having that coveted private access. Google’s Identity-Aware Proxy (IAP) helped me to create a secure tunnels to VMs over Https and used GCP public APIs. The solution worked like a charm, offering both security and accessibility.
Inspired by the success with IAP tunnels, I sought to integrate this technology with Ansible. The vision was clear: Seamlessly deploy the GKE cluster and all associated Kubernetes objects without any hitches, so what i need to determine is a method to execute my scripts directly from the bastion host using Ansible.
In my quest, I stumbled upon a configuration in an existing article that seemed promising. However, it faltered when applied across multiple projects. So i decided to write this article and share with you the fix. But first let’s start with some definitions :)
Identity-Aware Proxy (IAP)
Google’s Identity-Aware Proxy (IAP) is a service that allows you to establish secure, context-aware access to applications running in Google Cloud. For exemple, instead of using a traditional SSH, IAP verifies user identity and the context of the request to determine if a user should be allowed access to the VM instance.
Ansible
Ansible is an open-source automation tool that facilitates tasks such as configuration management, application deployment, and task automation. It uses a simple, human-readable language (YAML) to define automation jobs and doesn’t require an agent to be installed on the target system, making it lightweight and easy to use.
Ansible Configuration
To achieve the desired setup, you’ll need a trio of configuration files. Here’s a breakdown:
Step 1 : Update Ansible configuration file
ansible.cfg file is the cornerstone of our configuration. While its placement is not strictly sequential, its role is paramount. Here an example, a full configuration is provided here
#ansible.cfg
[inventory]
enable_plugins = google.cloud.gcp_compute
[defaults]
.....
[ssh_connection]
# Enabling pipelining reduces the number of SSH operations required
# to execute a module on the remote server.
# This can result in a significant performance improvement
# when enabled.
pipelining = true
ssh_executable = scripts/gcp-ssh-wrapper.sh
ssh_args = None
# Tell ansible to use SCP for file transfers when connection is set to SSH
scp_if_ssh = true
scp_executable = scripts/gcp-scp-wrapper.shNote that `google.cloud.gcp_compute` plugin is activated for inventory purposes and we’ve made modifications to the default behavior of the `ssh` and `scp` commands. Now, when Ansible attempts to access a VM using the `ssh` command or to transfer files via `scp`, it will invoke two specific SSH scripts.
We’ve chosen to harness the capabilities of gcloud compute ssh and gcloud compute scp for building the IAP tunnels within our wrapper scripts. A key aspect to highlight is the extraction of the project name. It's derived directly from the hostname, which Ansible forwards as its primary parameter. As a result, it's imperative that the hostname aligns perfectly with the self-link of the VM instance for seamless integration. Below is a sample from one of our scripts to give you an idea of its structure and functionality. The complete code is readily available in this repository.
#scripts/gcp-ssh-wrapper.sh
#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP SSH option to connect
# to our servers.
# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
selfLink="${@: -2: 1}"
cmd="${@: -1: 1}"
# Unfortunately ansible has hardcoded ssh options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for ssh_arg in "${@: 1: $# -3}" ; do
if [[ "${ssh_arg}" == --* ]] ; then
opts+="${ssh_arg} "
fi
done
# Parse self link to get the project and the selfLink
project=$(echo "${selfLink}" | cut -d '/' -f7)
host=$(echo "${selfLink}" | cut -d '/' -f11)
exec gcloud --project "${project}" compute ssh $opts "${host}" -- -C "${cmd}"Step 2 : Configure inventory
You need to create an inventory file inventory.gcp.yml witch will contain your plugin settings, this file is pivotal for ensuring dynamic configuration.
plugin: google.cloud.gcp_compute
projects:
- [[ Add prjects here ]]
filters:
- status = RUNNING
- scheduling.automaticRestart = true AND status = RUNNING
auth_kind: application
keyed_groups:
- key: labels
prefix: label
- key: zone
prefix: zone
- key: (tags.items|list)
prefix: tag
- key: project
prefix: project
groups:
bastion : "'bastion' in name"
hostnames:
# List host by name instead of the default public ip
- name
compose:
# Set an inventory parameter to use the Public IP address to connect to the host
ansible_host: selfLinkKey Notes :
- We’ve activated the
google.cloud.gcp_computeplugin to streamline our interactions with Google Cloud. - In our host filtering process, we specifically target hosts bearing names that include the term “bastion.”
- Crucially, for a seamless connection, we’ve designated the
ansible_hostparameter to align with the "selfLink" value.
It’s essential to be mindful of these configurations when working within this environment to ensure optimal operation.
Step 3: Optimizing SSH and SCP Arguments via group-vars
group_vars/all.yml: global SSH arguments are added via this file, ensuring that there’s a smooth communication line established with the target servers.
ansible_ssh_args: --tunnel-through-iap --zone={{ zone }} --no-user-output-enabled
ansible_scp_extra_args: --tunnel-through-iap --zone={{ zone }} --quietBy channeling the ssh and scp arguments through the group-vars, we're able to dynamically assign the zone based on the VM's pre-determined zone in Ansible's inventory. This specificity is crucial. If one were to omit specifying the zone, they would be met with an unmistakable error from gcloud:
ERROR: (gcloud.compute.ssh) Resource [xxx] lacks adequate specification. Please include the [--zone] flag.Test and validate this configuration ?
To effectively test and validate this configuration, initiate a virtual machine (VM) with IAP enabled. It’s essential that the VM’s name includes the term ‘bastion’ for compatibility. For your convenience and to eliminate the tediousness of manual copying, I’ve compiled all the necessary configurations into a GitHub repository. This ensures a seamless transition and setup on your end.
1- Clone the git repo
git clone https://github.com/belgacem-io/ansible-iap.git2- No SSH keys are used for access control, you just need to be authenticated to your GCP environment
gcloud auth login3- Update your inventory file inventory.gcp.yml by adding the list of your target projects where your VMs has been created
4- Finally, you can run your ping command using ansible
cd ansible
ansible bastion -i inventory.gcp.yml -m pingThe output of this command must be like this :
xxxx-bastion-xxxx | SUCCESS => {
"changed": false,
"ping": "pong"
}Conclusion
I share this insight in the hopes that it offers clarity and guidance for anyone else who may have encountered this hiccup. Understanding the nuances of these configurations can make all the difference in smooth operations.
