Auto-scale your Sentinel pricing tiers

with Github Actions and Microsoft Sentinel 0ptimiz€r

Introduction

Earlier this year I've published a blog post about how to optimize your Microsoft Sentinel and Azure Log Analytics pricing tiers. I'f you're not yet fully familiar with how the different pricing tiers differ, and how they stack up, I'd suggest you'd give that other article a read first and then come back here.

If you'd rather watch a video, my dear friend and colleague Jeroen Niesen recently uploaded a new video about this topic on his Youtube channel. Go give that a watch instead!

Auto-scaling done differently

The last couple of month I've improved and extended my PowerShell script quite a bit. And I've also created a fully automated solution with Github Actions so that your pricing tiers can automatically scale with you as your data grows or decreases in volume! This way you're always assured that you benefit the most from the discounts available and aren't paying too much at the same time.

Who is this for?

A lot of my customers are migrating their existing SIEM solutions to Microsoft Sentinel. This ensures that the ingest rate increases month after month, and so will the costs associated. The volume of data can grow quite rapidly, especially in the first couple of months. So it's goot practice to regularly pay close attention to both the Sentinel and log analytics pricing tiers to make sure you benefit from discounts as much as possible.

Some customers opt to start off with a more "lift-and-shift" approach when it comes to shipping logs to their new SIEM. This can be due to time constraints or other factors. But you want to make sure that your capacity reservation tiers are also scaled down once the ingest rated is lowered. Otherwise you pay too much for no reason. Optimizing your log ingestion strategies can be for example by applying optimizations such as filtering or real-time ingest transformations. Or because some log sources appeared to be too chatty for analytic rules and you've decided to store them elsewhere.

Because an automated solution can bring you peace of mind, knowing you're not wasting any Azure spend, I think everyone should automate as much as possible!

How does it work?

To implement this auto-scaling solution you'll need a couple of things:

• A Github environment in which you can create a Github Action workflow
• A Github repository containing your ARM templates and parameter files you use to deploy your Azure infrastructure, including Log Analytics and Sentinel
• An Azure Active Directory app registration properly configured to use for OpenID Connect (see instructions below)
• My Microsoft Sentinel 0ptimiz€r PowerShell script

Please note that the Microsoft Sentinel 0ptimiz€r PowerShell script can still be used stand-alone and run interactively. See more information about the script and its available parameters below.

Overview

1. A Github Action workflow, which is scheduled to run every month, will trigger a job in which it will login to Azure with a federated identity with at least read permissions on your workspace.
2. Sentinel 0ptimiz€r PowerShell script will run and will:
   - Perform a KQL query against your workspace(s) to determine the average daily ingest (GB) during the last 30 days.
   - Determine what the optimal pricing tiers are for both Azure Log Analytics and Microsoft Sentinel based on that average ingest.
   - Read your ARM template parameter file(s), used for deploying the workspace(s), and update the parameter values accordingly.
3. Lastly, this change is committed into a new Git branch and a new pull request is initiated.
4. Depending on how your Github environment is setup, people will be notified about this new pull request and they can review the changes proposed by the Github Action before approval
5. Depending on how your infrastructure is deployed, this PR will trigger a new incremental deployment of the workspace(s) to make sure these new changes are applied in the Azure environment.

You can find all files needed to get started on my Github repository

Github Action workflow

Workflow definition

The workflow definition is a YAML file containing all the steps the Github workflow should perform. The file should be saved in your Github repository inside the .github/workflows folder.

name: SentinelPricingTierAutoScale
on:
  schedule:
    - cron:  '0 1 8 * *'    # At 08:00 on every 1st day of the month

permissions:
  id-token: write
  contents: write
  pull-requests: write

jobs:
  ubuntu-latest:
      runs-on: ubuntu-latest
      environment: < Github environment > # Setup in Github --> Settings --> Environments
      steps:

        - name: Login to Azure Cloud
          uses: azure/login@v1
          with:
            client-id: ${{ secrets.CLIENTID }}              # Configure these secrets in Github --> Settings --> Environments --> Environment secrets
            tenant-id: ${{ secrets.TENANTID }}              # Configure these secrets in Github --> Settings --> Environments --> Environment secrets
            subscription-id: ${{ secrets.SUBSCRIPTIONID }}  # Configure these secrets in Github --> Settings --> Environments --> Environment secrets
            enable-AzPSSession: true

        - name: Check out repository code
          uses: actions/checkout@v3
          with:  
            fetch-depth: 100

        - name: Check optimal Log analytics and Sentinel pricing tiers
          id: check-sentinel-pricing
          shell: pwsh 
          run: ./powershell-script/Get-AzSentinelPricingRecommendation.ps1 -subscriptionId ${{ secrets.SUBSCRIPTIONID }} -updateArmParameters $true -parametersFilePath "arm-templates/"

        - name: Get current date
          id: date
          shell: pwsh 
          run: |
            $date = Get-Date -Format dd-MM-yyyy
            "date=$date" >> $env:GITHUB_OUTPUT

        - name: Push changed parameters file(s)
          shell: pwsh
          run: |
            Write-Output 'Variable $fileChanges : ${{ steps.check-sentinel-pricing.outputs.fileChanges }}'
            Write-Output 'Variable $filePath    : ${{ steps.check-sentinel-pricing.outputs.filePath }}'
            git config --global user.email "user@domain.com"
            git config --global user.name "username"
            git checkout -b update/pricing-tier-${{ steps.date.outputs.date }}-${{github.run_number}}
            git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
            git commit --all --message "New optimal pricing tier ${{ steps.date.outputs.date }}"
            git push --set-upstream origin update/pricing-tier-${{ steps.date.outputs.date }}-${{github.run_number}}

        - name: Create pull request
          uses: devops-infra/action-pull-request@v0.5.3
          with:
            github_token: ${{ secrets.GITHUB_TOKEN }}
            source_branch: update/pricing-tier-${{ steps.date.outputs.date }}-${{github.run_number}}
            target_branch: master
            template: .github/PULL_REQUEST_TEMPLATE_PRICING_TIER.md

Workflow variables

Inside the workflow definition there are several references to an environment and environment secrets. The name of the environment is also referenced in the Azure AD app registration you’ve registered for OpenID Connect. See steps below.

Within Github settings you'll be able to register your environment(s) and respective secrets:

Example of app registration details and tenantId being passed as environment secrets

Azure authorization

For logging into Azure, the Github workflow uses OpenID Connect (OIDC). This allows your GitHub workflows to access resources in Azure, without needing to store the Azure credentials as long-lived GitHub secrets.

You'll need to create an Azure Active Directory application and add federated credentials. Follow this guide for step-by-step instructions.

Once the identity is in place you'll need to assign proper permissions to it. Assigning the built-in role of Log Analytics Reader on the workspace should be the least-privilege required to perform the KQL query against the workspace.

For running the script interactively, and let it look at multiple workspaces across subscriptions, you obviously need a little more permissions. Being a Reader on subscription level should then be sufficient.

PowerShell script

As mentioned earlier, the PowerShell script is an improved version of the script I wrote about earlier. It now also pulls in the Sentinel Sku via the API and most importantly It's able to make the proper changes to the resource definition parameter files is necessary.

It can be used with these optional parameters as well:

• subscriptionId — If not specified the script will cycle through all Azure subscriptions you have at least read access to and look for workspaces inside each subscription. For auto-scaling purposes inside a Github workflow it’s best to provide a single subscriptionId.
• updateArmParameters — Default is $false and will only report current Sku and optimal Sku values. Set to $true if you want the script to write changes to ARM template parameter files.
• parametersFilePath — Provide a path to your ARM template parameters files. The script assumes the filenames are the same as the workspace names. If left empty, the script will prompt you for a path if updateArmParameters is set to $true.

./powershell-script/AzSentinelPricingOptimizer.ps1 `
    -subscriptionId 'c2a97e8b-592d-442a-aef5-9e5a6347350d' `
    -updateArmParameters $true `
    -parametersFilePath 'arm-templates/'

Example where 0ptimiz€r loopts through several subscriptions and workspaces

Commit & create pull request

The last step in the workflow is to initiate a pull request. It uses a pull request template provided in markdown. This template can be placed anywhere inside your repository, as long as it's referenced correctly from the workflow.

# Automated Code Pull Request

Before submitting this Pull Request, please make sure and check the list below.

## General

* [x] Change purpose checkboxe(s) are updated
* [x] Change has been described

## Purpose for change

* [x] ~~New/~~ Updated Infrastructure service(s)
* [ ] New/updated Analytic Rule(s)
* [ ] Bug Fixed

## Describe the change

A scheduled Github Workflow (SentinelPricingTierAutoScale) determined that it's better to make changes to the current pricing tier(s) on one or multiple Sentinel workspaces.

Please review the code changes as part of this pull request before approval.

Once a pull request is created this template is also visible in e-mail notifications if this is something you use:

Sample e-mail notifying of a new pull request for approval

Git diff highlighting the changes made by the Github Actions workflow

ARM template and parameters

Unfortunately setting a Sentinel Sku separate from the Log Analytics one is something that is still not documented yet in the Microsoft ARM reference documentation. But it is possible to provide this parameter! For some reason the values differ a bit from those used for Log Analytics. Please refer to a sample template and parameters file provided in my Github repository.

Within the template I’ve used variables that determine if the pricing tier should be either perGb or capacityReservation based on the provided parameter values. This was neccesary because the capacityReservationlevel in only provided if the latter is true.

"variables": {
    "sku": "[if(equals(toLower(parameters('pricingTierLogAnalytics')), 'capacityreservation'), json(concat('{\"name\":\"', parameters('pricingTierLogAnalytics'), '\",\"capacityReservationLevel\":', parameters('capacityReservationLevelLogAnalytics'), '}')), json(concat('{\"name\":\"', parameters('pricingTierLogAnalytics'), '\"}')))]",
    "sentinelSku": "[if(parameters('pricingTierSentinel'), json(concat('{\"name\":\"CapacityReservation\",\"capacityReservationLevel\":', parameters('capacityReservationLevelSentinel'), '}')), json('{\"name\":\"PerGB\"}'))]"
},

Within the parameters file you’ll notice that the parameter for pricingTieLogAnalytics is either PerGB2018 or CapacityReservation. While for Sentinel the pricingTierSentinel is either true or false depending is you want to use perGB or capacityReservation respectively.

"pricingTierLogAnalytics": {
    "value": "CapacityReservation"
},
"pricingTierSentinel": {
    "value": true
},
"capacityReservationLevelLogAnalytics": {
    "value": 400
},
"capacityReservationLevelSentinel": {
    "value": 500
},
"retentionInDays": {
    "value": 180
},

Is your workspace name is for example log-sentinel-westeurope-01 the script assumes the corrosponding ARM template parameters file starts with the same name. It won't be bothered by any additional suffixes and/or extension.

Deploy changes to workspace(s)

Once the new parameters are merged into your main branch you can trigger you incremental deployment(s) or let the PR trigger do its job if configured.

It is not mandatory to use Github Actions for you Azure resource deployments as well when using this auto-scale solution. Below is a screenshot of an Azure DevOps Pipeline performing the incremental deployment of Sentinel:

Even though your code resides in Github, you can still use Azure Pipelines for deploying your resources

Conclusion

I hope this solution will help others to also optimize their Sentinel and Log Analytics pricing, benefit from the optimal discounts available and stop wasting any Azure spent.

Let me know what you think about this solution and if you have any questions. Also let me know if you have realized some major cost saving by leveraging this solution!

— Koos