AppService Meets CI/CD

Have you ever faced the challenge of running a shell command during the deployment of an Azure AppService (referred to as “app” or “appservice” from now on)? Or, in a broader context, how do you handle pre- and post-deployment commands when updating your web app, be it on Azure or another platform (such as Elastic Beanstalk on AWS)? If you’re seeking a potential solution, continue reading as I describe a smart workaround that might be just what you’re looking for.

Throughout the deployment process of an AppService, you might encounter scenarios where you need to run particular commands on your production app, like executing migrations or clearing the cache.

Furthermore, if your database resides within a private network, this introduces an additional challenge. Although there are methods to work around this limitation, they compromise the objective of maintaining your database within Azure’s secure private network, posing a security risk.

Initially, this might appear to be a straightforward task if the application were running on a virtual machine. However, given that it operates within a Dockerized environment, the situation becomes more intricate.

As per this documentation for AppService, it is feasible to SSH into a running app and manually execute shell commands. Regrettably, this approach doesn’t align with our objective of fully automating the process. Consequently, we had to craft a step-by-step solution to overcome this challenge.

I assume that you already have your GitHub Action set up to handle automated deployment to AppService.

Azure AppService (AAD) logo. Image by Microsoft from the Microsoft Architecture icons collection. Usage scope: technical documentation blog in the public domain.

Why this?

When transitioning our web deployments from traditional LAMP virtual machines to AppService, a managed web app by Azure, we aimed to retain our existing deployment pipelines. Think of it as a “lift & shift” approach: new infrastructure, but maintaining our old pipelines. Since our pipelines were SSH-enabled, even our deployment tool, Deployer (itself somewhat old-school), we started exploring ways to achieve this within AppService.

Lift & skift! Image credits: Kipras Štreimikis on Unsplash

This endeavor also aimed to address a significant deficiency in the architecture of AppService. To put it formally, it raised the question: “Where should we run those post-deployment commands?” As a solution, we decided to establish an SSH connection to the app after deployment and execute customized commands in the production environment using CI/CD.

As you’ll discover as you read further, we were able to accomplish this. However, the journey came with its fair share of challenges. Notably, the SSH tunnel proved to be quite unstable, causing our GitHub runner to hang. We tried to address that with good old sleep statements, to no avail. Additionally, the security concerns multiplied, making it feel at times like we were "hacking" into our own system, all in the name of reusing some old pipelines.

When complexity and discomfort reach a certain threshold, it becomes necessary to seek a new solution — and that’s precisely what we did in the end. Nonetheless, we find the technical exercise itself quite fascinating, and we’re eager to share it with you.

Step 1: Generate ssh keys

• Type the command ssh-keygen in your terminal or command prompt and press Enter. (if you get an error about command not found try thissudo apt-get install openssh-client)
• You will be prompted to choose a location to save the key pair. Take note of the file name displayed. Press Enter to accept the default location, typically the home directory.
• Next, you’ll be asked to enter a passphrase for the key. In this case, press Enter to leave it blank. it is much better not to have any password for the key, for a very simple reason: if it’s there, it needs to be piped in into the ssh command, and that’s cumbersome
• The SSH key generation process will create two files: id_ed25519 or id_rsa (the private key) and id_ed25519.pub or id_rsa.pub (the public key).

Image credits: A. Ayomide.

Ensure that you keep the private key ( id_ed25519 or id_rsa) secure and do not share it with anyone. Conversely, the public key ( id_ed25519.pub or id_rsa.pub) can be shared with the relevant parties or added to the authorized keys on remote servers as required.

Step 2: Upload ssh keys as Github secret

• Open the SSH secret file (the one without the .pub extension) and copy its content: cat $HOME/.ssh/id_rsa
• We’ll now add it as a Github secret…
• Go to your GitHub repository and navigate to the Settings tab.
• In the left sidebar, click on “Secrets and variables” to access your repository’s secrets.

Image credits: A. Ayomide.

Click on New repository secret to create a new secret.

• Name the secret AZURE_SSH_PRV_KEY and paste the content of your SSH secret file into the Value input field.
• Save the secret by clicking on the “Add secret” button.

Next, follow the same procedure for your .pub key; it can be added as a GitHub environment variable. Once you've completed this, ensure that you have these two secrets in your repository.

Image credits: A. Ayomide.

A big disclaimer

Private keys are intended to remain… private. In fact, they should never venture beyond the confines of your ~/.ssh directory. What we're doing here is secure as long as a few essential assumptions hold, such as:

• using a unique key, specific to the environment
• anyone with access to the secret from a pipeline should also be aware of the secret

We simply require passwordless SSH to make everything function seamlessly — after all, a CI/CD pipeline must be entirely automated. That said, please take your environment into account. As always, the security of a solution is only as strong as its weakest link!

You got this (hopefully!). Image credits: sydney Rae on Unsplash

Step 3: Upload ssh key

Once you have set your SSH keys as secrets in GitHub Actions, the next step is to upload the content of the .pub file to your Azure app service. Follow these instructions:

• Start by clicking the SSH button in the sidebar of your Azure portal to establish an SSH connection to your Azure app service.
• In the SSH session, run the following commands:

SSH_HOME=/root/.ssh PUB_KEY="paste the content of your SSH .pub file here" mkdir -p $SSH_HOME / echo $PUB_KEY > $SSH_HOME/id_rsa.pub / chmod 400 -R $SSH_HOME

Step 4: Az cli login with SP

You must create an Azure Service Principal (SP) and securely store its credentials as GitHub secrets. You should save the SP username, tenant, and password as GitHub secrets. This is essential for authenticating our Azure CLI, which will be responsible for performing write operations on the app, such as deployments.

Step 5: Disable strict mode

We’re nearly finished; we only need to disable SSH strict mode so that our commands from GitHub can successfully pass to the App Service. You should execute this command in your AppService via the web SSH from the portal:

sed -i s// /etc/ssh/sshd_config service ssh restart

Here’s a security consideration, as mentioned earlier: only proceed with this if the overall environment is highly controlled. Additionally, be aware that modifying any configuration file, or any file on ephemeral storage, could potentially disrupt the managed solution, necessitating repetition with each deployment. Moreover, future upgrades of the managed image might inadvertently undo this fix (for instance, if a certain SSH configuration switch is renamed, etc.).

Configuration: good to read, not to edit. Image credits: Ferenc Almasi on Unsplash

Step 6: create remote ssh connection + get in with `sshpass`

To establish an SSH tunnel from your workflow, we must initially configure a connection and employ sshpass for authentication. Keep in mind that we are operating within a deployment pipeline, and manual steps are not permissible. Below is the complete workflow step:

- name: Set up ssh key
  env:
    SSH_HOME: /$HOME/.ssh
  run: |
    
    mkdir -p ${{ env.SSH_HOME }}
    echo "${{ secrets.AZURE_SSH_PRV_KEY }}" > ${{ env.SSH_HOME }}/id_rsa
    echo "${{ secrets.AZURE_SSH_PUB_KEY }}" > ${{ env.SSH_HOME }}/id_rsa.pub
    sudo chmod 400 -R ${{ env.SSH_HOME }}
  shell: bash

- name: Run ssh command
  env:
    SSH_PASSWORD: "Docker!"
    SSH_PORT: 6007
    WAIT_TIME: 5
  run: |
    az login --service-principal --username '${{ secrets.SP_USERNAME }}' --tenant '${{ secrets.SP_TENANT }}' --password '${{ secrets.SP_PASSWORD }}'

    sleep ${{ env.WAIT_TIME }}
  
    az webapp create-remote-connection --subscription '${{ env.APP_SUBSCRIPTION }}' --resource-group ${{ secrets.APP_RS_GROUP }} -n ${{env.APP_NAME }} --port ${{ env.SSH_PORT }} &
  
    sleep ${{ env.WAIT_TIME }}

    sshpass -p ${{env.SSH_PASSWORD}} ssh -o StrictHostKeyChecking=no root@127.0.0.1 -p ${{ env.SSH_PORT }} "ls -a"
  shell: bash

Notice we append & after az webapp create-remote-connection: this is important, it allows us to create a background tunnel where we can pipe our SSH commands, read more here.

Conclusion

In summary, the workflow steps described above offer a solution for setting up an SSH tunnel and executing commands on an Azure app service within your workflow. By utilizing az login for authentication with the Azure service principal, az webapp create-remote-connection to create a remote connection to the app service, and sshpass for authentication and establishing an SSH connection, you can safely execute commands on the target server. However, please note that some custom SSH configuration is required for this process.

Remember to adjust the values of the following environment variables and secrets:

* `APP_NAME`, env
* `APP_SUBSCRIPTION`, env
* `AZURE_PUB_PROFILE`, secrets
* `AZURE_SSH_PRV_KEY`, secrets
* `SP_USERNAME`, secrets
* `SP_TENANT`, secrets
* `SP_PASSWORD`, secrets
* `APP_RS_GROUP`, secrets

according to your specific setup. And again, keep in mind that secrets in a CI/CD pipeline are never that secret: typically, any shell step downhill can just access them in plaintext.

We are done! Image credits: Joshua Hoehne on Unsplash

Here is the full workflow in a workflow file (including helpful sleep statements and some friendly error messages).

name: App service ssh

inputs:
  AP_SSH_PORT:
    description: app service ssh port -P
    required: true
  AP_SLEEP_TIME:
    description: app service ssh command delay time
    required: true
  SCRIPT_PATH:
    description: Shell script
    required: false
  COMMANDS:
    description: Shell COMMANDS
    required: false

runs:
  using: composite
  steps:
    - name: Set up ssh key
      run: |
        mkdir -p ${{ env.SSH_HOME }}
        echo "${{ env.AZURE_SSH_PRV_KEY_DEV }}" > ${{ env.SSH_HOME }}/id_rsa
        echo "${{ env.AZURE_SSH_PUB_KEY_DEV }}" > ${{ env.SSH_HOME }}/id_rsa.pub
        chmod 400 -R ${{ env.SSH_HOME }}
      shell: bash

    - name: Run ssh command
      env:
        SSH_PASSWORD: "Docker!"
      run: |
        FILE=./.github/scripts/${{inputs.SCRIPT_PATH}}

        az login --service-principal --username '${{env.AZURE_AP_USERNAME_DEV}}' --tenant '${{env.AZURE_AP_TENANT_DEV}}' --password '${{env.AZURE_GITHUB_SP_SECRET}}'
        sleep ${{ inputs.AP_SLEEP_TIME }}
        curl -k https://development-api-001.sharesquare.co
        sleep ${{ inputs.AP_SLEEP_TIME }}
        az webapp create-remote-connection --subscription '${{env.AZURE_AP_SUBSCRIPTION_DEV}}' --resource-group ${{env.AZURE_AP_RESOURCE_GROUP_DEV}} -n ${{env.AZURE_AP_APP_NAME_DEV}} --port ${{ inputs.AP_SSH_PORT }} &
        sleep ${{ inputs.AP_SLEEP_TIME }}

        if [ -f "$FILE" ]; then
           sshpass -p ${{env.SSH_PASSWORD}} scp -o StrictHostKeyChecking=no -P ${{ inputs.AP_SSH_PORT }} $FILE root@127.0.0.1:/home/site
           echo copied ssh script
           sshpass -p ${{env.SSH_PASSWORD}} ssh -o StrictHostKeyChecking=no root@127.0.0.1 -p ${{ inputs.AP_SSH_PORT }} "bash /home/site/${{inputs.SCRIPT_PATH}}"
           echo Done executing script
        elif [ ! -z "${{inputs.COMMANDS}}" ]; then
           echo Running ssh command
           sshpass -p ${{env.SSH_PASSWORD}} ssh -o StrictHostKeyChecking=no root@127.0.0.1 -p ${{ inputs.AP_SSH_PORT }} "${{inputs.COMMANDS}}"
        else
           echo script file and ssh command not provided
        fi
      shell: bash

By incorporating these workflow steps into your automation process, you can seamlessly interact with your Azure app service and perform necessary tasks over the established SSH tunnel.

Blog by Riccardo Vincelli and Samuel Segbenu brought to you by the engineering team at Sharesquare.