Reducing VMSS costs at off-hours

For single virtual machines, Azure offers an auto-shutdown functionality. For virtual machine scale sets (VMSS), there is nothing comparable. Here is a little guide on how you can achieve this.

If you look at a VMSS agent pool in Azure DevOps, you can see a setting for standby agents:

If this count is set to 1, there will always be one machine running to swiftly work on queued jobs. This also means, that there will always be one machine that is running and needs to be payed for, even if there are no jobs. There is no UI menu to have this number set according to a schedule.

This setting can be changed by using the ADO REST API.

Using an URL, such as https://dev.azure.com/{organization}/_apis/distributedtask/elasticpools, will return you a list of all available VMSS in your collection. We are interested in pool ids.

Using the REST API and the pool id, we can make a PATCH call to change the agent pool settings:

Each property maps to a property in the agent pool setting UI.

For the above call to work, we need to be authorized. This can be done using a personal access token (PAT). Only the below privilege is needed:

Note: The user that creates the PAT needs to have the privilege to change the agent pool himself.

Save the PAT as a secret in a library:

Now all prerequisites are met. Here is a yaml pipeline that reduces the standby agents at off hours:

# This pipeline is to reduce the azure costs at night

parameters:
# The complete list can be found when you browse to https://dev.azure.com/sartorius-bps/_apis/distributedtask/elasticpools
- name: agent_pool_ids
  type: object
  default: 
    - '14'
    - '15' #Some other pool

variables: 
- group: 'agent-pool-secrets'

schedules:
- cron: 0 17 * * Mon,Tue,Wed,Thu,Fri
  displayName: 'Off hour starts'
  branches:
    include:
    - master
  always: true #run, even if there are no code changes

- cron: 0 9 * * Mon,Tue,Wed,Thu,Fri
  displayName: 'Working hour starts'
  branches:
    include:
    - master
  always: true #run, even if there are no code changes

jobs:
  - job:
    displayName: Decrease agent pools
    condition: eq(variables['Build.CronSchedule.DisplayName'], 'Off hour starts')
    steps:
      - ${{ each poolId in parameters.agent_pool_ids }}:
        - task: PowerShell@2
          displayName: Reduce idle agents for vmss (pool id = ${{ poolId }})
          inputs:
            targetType: 'filePath'
            filePath: '$(System.DefaultWorkingDirectory)/.azuredevops/templates/UpdateIdleAgentCount.ps1'
            arguments: '-PoolId ${{ poolId }} -DesiredIdle 0'
          env:
            MAPPED_PAT: $(AGENT_POOL_TOKEN)
          
  - job:
    displayName: Increase agent pools
    condition: eq(variables['Build.CronSchedule.DisplayName'], 'Working hour starts')
    steps:
      - ${{ each poolId in parameters.agent_pool_ids }}:
        - task: PowerShell@2
          displayName: Increase idle agents for vmss (pool id = ${{ poolId }})
          inputs:
            targetType: 'filePath'
            filePath: '$(System.DefaultWorkingDirectory)/.azuredevops/templates/UpdateIdleAgentCount.ps1'
            arguments: '-PoolId ${{ poolId }} -DesiredIdle 1'
          env:
            MAPPED_PAT: $(AGENT_POOL_TOKEN)

UpdateIdleAgentCount.ps1:

param (
    [int]$DesiredIdle,
    [int]$PoolId
)

# Call the Azure DevOps Services Rest API.
Write-Output "Updating idle agents for pool $($PoolId)"
$url = "$($env:SYSTEM_COLLECTIONURI)_apis/distributedtask/elasticpools/$($PoolId)?api-version=7.0"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$env:MAPPED_PAT"))
$headers = @{Authorization = "Basic $token"}
$body = @{"desiredIdle" = "$DesiredIdle"} | ConvertTo-Json
Invoke-RestMethod -Uri $url -Method 'PATCH' -ContentType 'application/json' -Headers $headers -Body $body

Those fields need to be adopted, according to your needs: