Managing Permissions on mounted SSH keys in Cloud Run

Overview:

CloudRun is a google’s managed compute platform that lets you run and scale containers directly on top of Google’s scalable infrastructure. This particular document may prove useful for people trying to connect applications running on CloudRun using SSH keys(uploaded via Google Secrets Manager(GSM)) to other services. ForExample — ansible scripts connecting to a VM, application connecting to a database etc. I was trying to execute a use-case to execute ansible scripts on a GCE and faced this particular issue of changing permissions of mounted SSH keys. This document contains detailed steps performed to connect ansible to an GCE and provide workaround to a major blocker that occurs with mounted ssh keys to used with ansible.

Ansible by default expects a SSH connection from the master to the remote nodes before executing the scripts. The ssh connection can be performed by a variety of ways like manually adding private key to the master node, using ssh-agent, or saving the private key on the machine and referencing it from the script using ansible_ssh_private_key_file tag. In GCP following best security practices, it is always recommended to use Secrets Manager to store and mount the keys to the container running in Cloud Run.

Steps:

This section contains detailed info on all steps performed to arrive and overcome this issue. This use-case can be divided at a high-level into the following steps,

• Creating the Image with scripts.
• Create network connection from CloudRun and VM.
• Manage keys in CloudRun.
• Manage the mounted keys to match permissions of ansible.

Creating the Image with scripts:

A ubuntu image with ansible installed was used from docker-hub. For this POC, a simple ansible script to create a sample text file was used. Following are the snippets of all files used for this use-case,

inventory.ini file:

{ 
[targets]
{internal-ip of VM}     ansible_connection=ssh        ansible_user={user-name}   ansible_ssh_common_args='-o StrictHostKeyChecking=no' ansible_ssh_private_key_file={secret-key-path}
}

ansible-playbook.yaml:(to create an empty textfile in target VM)

- name: To change the permissions of key file in container
  hosts: localhost
  tasks:
    - name: execute the local script to change permissions
      script: /home/before_script.sh
      register: log

    - debug: var=log  

- name: This is a hello-world example
  hosts: {internal-ip of VM}
  tasks:
    - name: Create a file called '/tmp/testfile.txt' with the content 'hello world'.
      copy:
        content: hello world
        dest: /tmp/testfile.txt

before_script.sh

mkdir -p /root/secrets
chown -R {username}:{username} /root/secrets
chmod 755 -R /root/secrets
cp /mount/{secret-key mounted} /root/secrets/cr-ansible-key
cat /root/secrets/cr-ansible-key
chmod 600 /root/secrets/*

The docker image once created is pushed into google container registry from which CloudRun can use this image to execute the job. To push the container image into gcr, Create and download service-account.json file with permissions to connect to gcr and set the docker to use this service account to push/pull images. Tag the new image according to GCR standards and push the image to gcr repository.(push/pull images from gcr). Following commands are used to connect to gcr and push/pull images.

** to save changes made to the running container ** 
Docker commit $containername (this will create a new image)

** connect to gcr using docker **
docker login -u _json_key --password-stdin https://gcr.io < account.json

** push/pull images into gcr ** 
docker tag ${imagename according to gcr std}
docker push ${imagename}

Creating Connection & CloudRun job execution:

To connect the CloudRun with the VM, create a serverless VPC access connector on the same VPC and region as the VM. This connector is used by the CloudRun job to establish a network connection from the job to the VM.

Create private and public key by running the ssh-keygen command. Save the private key as a secret key under Google Secrets Manager and add the public key to the VM under SSH metadata section. This metadata automatically adds the public-key under the authorized_keys file inside the VM for ssh-connection. Under Cloud Run service, open Cloud Run Job and create a new job. (NOTE: Ansible containers can be deployed only as a job because a Cloud Run service requires the application to listen on some port — mostly used for web applications.)

Select the container image, region (must be same as the VPC connector), under General tab, pass the argument to run the ansible-playbook inside(screenshots provided below), reference the ssh-key added as the secret under Variables & Secrets tab and add the connector created to the connect to the VM under Connections tab.

Arguments for entry points in CloudRun job

Manage the mounted keys to match permissions of ansible:

On executing the job, it throws the below error,

“fatal: [#.#.#.#]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '#.#.#.#' (ED25519) to the list of known hosts.\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: UNPROTECTED PRIVATE KEY FILE! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0444 for '/mount/cr-ansible-key' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\nLoad key \"/mount/cr-ansible-key\": bad permissions\r\n{username}@#.#.#.#: Permission denied (publickey).", "unreachable": true}
”

When we try to reference the mounted key from the script it does not work from ansible as intended because Cloud Run mounts the key as a read-only file with 0444 permissions and ansible expects the key to have 0600 and throws an error saying the private key has too many open permissions.

To overcome this error, a folder with necessary permissions — 0600 is created for the user and the mounted ssh-key is copied into this folder. The key copied into this folder is now referenced in the ansible-script and the ansible-script executes without the above error. This is executed by the before_script.sh which is run before the main ansible task in ansible-playbook.yaml. Snippet of the attached script is added in the first section of this document.

This use-case is written for mounting the keys via secret manager. However the keys can be passed by passing it as an environment variable(can cause security issues), directly into the script or accessing the key directly by hitting the Google Security Manager api.