From soup to nuts: Building a Detection-as-Code pipeline

Part 1 of 2

Image generated by Bing Image Creator

I was inspired to work on a new project a few weeks ago after reading an awesome blog post, Automating Detection-as-Code by John Tuckner of Tines. In his post, John walks us through building a Detection-as-Code (DAC) workflow using Tines and Elastic Security.

I was excited to design and build a new DAC pipeline using Terraform, Sumo Logic, Tines, learn some new skills, and have a good excuse to write some code 🙂. In this two part blog series, I’ll share a methodology for building and implementing a DAC pipeline from start to finish. I’ll be utilizing a practical detection use case to show a process for creating, testing, deploying, and modifying detections using DAC principles.

Part one provides readers with an introduction to the concepts and benefits of Detection-as-Code. I’ve included explanations for the tools used to build and implement the DAC pipeline and how they’re configured. Finally, I walk through a Detection Engineering process using Terraform and Sumo Logic.

Part two includes explanations and code for creating CI/CD workflows to test our DAC pipeline and deploying changes to a dev/prod environment. I’ll walk through a process for handling alert payloads with Tines, testing detections, and validating the alert pipeline. Finally, I’ll demonstrate how the DAC pipeline works end-to-end with a practical use case, detecting & responding to suspicious Okta behavior.

Goals for this blog series

My goals for these posts are as follows:

• Help my fellow defensive security practitioners by sharing knowledge and a methodology to build a DAC pipeline and help them defend their company from threats and reduce/manage risk.
• Keep the barriers to entry as low as possible by demonstrating how to build a DAC pipeline using free, community, or trial editions of software.
• To teach is to learn twice — Teaching and sharing knowledge with others helps to deepen and solidify my understanding of a subject. Plus, I enjoy helping others learn and grow.

This is a way to build and implement a DAC pipeline, not the way. Every security team has a different type of organization to defend, different tools, logs, budget, skill set, and available resources. My goal is to share a way to help people get started with DAC without feeling overwhelmed.

I’ve shared some sample code in the following GitHub repo and will be referring to it throughout this post: threat-punter/detection-as-code-example.

Existing work

As a researcher, I believe that it’s crucial to reference existing work and give credit where it’s due. We’re all “standing on the shoulders of giants” at the end of the day, building upon what others have built before us to make progress.

There’s an “existing work” section at the end of this post that includes links to resources that I think you’ll find useful on your DAC journey. If you feel that I’ve missed any references, please let me know and I’ll be happy to add them.

What is Detection-as-Code?

Detection-as-Code (DAC) is a set of principles that use code and automation to implement and manage threat detection capabilities.

Traditionally, security practitioners would write detection rules or signatures and manually configure them in their security tools such as a SIEM. This process is time consuming, error-prone, and can lead to delays in detecting and responding to threats. You might have also heard a security teammate shout something like this at some point after a threat or Red Team activity was missed. 😆

Who changed this detection rule 6 months ago to filter out command line values containing New-Object System.Net.WebClient).DownloadFile?!”

DAC helps address these challenges by leveraging software development practices and streamlines the process of creating, testing, deploying, and maintaining detections by treating them as code artifacts.

Patrick Bareiss and Jose Hernandez gave a great presentation on Detection as Code: Detection Development Using CI/CD at RSA. I encourage you to watch the recording if you’re interested in learning more on the principles of DAC.

The screenshot below is from their presentation and shows a DAC workflow using CI/CD automation. I’ll be following this workflow throughout this project.

Detection as Code: Detection Development Using CI/CD by Patrick Bareiss and Jose Hernandez

Benefits of Detection-as-Code

I’ve listed some benefits of DAC below, but if you’re interested in a more comprehensive explanation, you should check out this presentation by Kyle Bailey: Detection-as-code: Why it works and where to start.

• Collaboration — DAC makes it easier for team members to discuss and contribute to new detections or modifications to existing ones. A group of practitioners with unique insights working together will result in more accurate and effective threat detection rules. DAC also makes it easier to share detections with other people in the cybersecurity community. There’s also shared blame for false positive explosions — JK 😳.
• Version control — Detection rules are written as code in formats such as YAML or JSON and stored in a version control system (VCS) like GitHub. The security team collaborates on any changes, which are reviewed, tracked, and tested before being deployed to production.
• Automation — CI/CD tools are used to ensure a continuous and iterative process to build, test, and deploy changes to your detections while minimizing bugs and the introduction of failures such as false positives/negatives. New detection capabilities can be rolled out quickly and efficiently.
• Agility — With a DAC CI/CD pipeline, security teams can respond to emerging threats quickly by creating or modifying detection rules and having them tested and deployed with zero (or minimal) manual effort required.

The toolshed

Let’s briefly talk about the tools I’m going to use to build the DAC pipeline.

Sumo Logic

I’ll be using Sumo Logic as my SIEM to ingest logs, run detections, and alert on security events. You can sign up for a free 30-day trial of Sumo Logic here.

GitHub

The following components will be built in GitHub:

• Developing detection rules as code.
• CI/CD — Validating, testing, and deploying changes to detections using GitHub Actions.
• Case management — Alerts from detection rules will be opened as GitHub issues.

Terraform

Terraform allows you to define infrastructure using code, which is version controlled, repeatable, and shareable. This approach enables you to treat your infrastructure like any other software project, making it easier to manage, collaborate, and automate.

I’ll be using the Terraform provider for Sumo Logic to manage resources including detections and webhook connections to send alert payloads to third-party applications.

I also chose Terraform to develop a better understanding of it. Terraform is an in-demand skill in the field of cybersecurity and essential for any developer working with the cloud. Prior to this project, I’d worked with Terraform a bit — I understood the fundamentals, but by building something, making mistakes, and finding solutions, we foster a deeper and more meaningful understanding of the skill being learned.

This presentation by Adam Cole provides a good introduction on using the Terraform provider for Sumo Logic. I referred to his example when struggling to define the correct Terraform syntax to create the webhook connection that sends alert payloads to Tines. Thanks, Adam.

Tines

Tines is used by security teams to automate their workflows without having to write code. I can’t say enough good things about Tines and the people that work there. It’s an incredibly powerful tool and a lot of fun to use.

You can sign up for the free community edition of Tines here.

I’ll be using Tines to automate the handling of alert payloads received from Sumo Logic as well as executing response actions in the end-to-end demonstration once the DAC pipeline is built.

Okta

As I mentioned earlier, I’m going to utilize a practical detection use case while building the DAC pipeline and testing it end-to-end.

My detection rule will rely on Okta system logs. Therefore, we’ll need an Okta organization that we can configure. You can sign up for a free Okta developer account here that comes with an Okta organization for you to administer.

The goal for this project is to build a DAC pipeline that works with a practical detection use case end-to-end. It’ll be straightforward to build additional features and add new detections from there.

Let’s start getting our hands dirty configuring our environment and writing some code to build something useful!

Initial configuration

Next up, I’ll walk through the initial configuration of the environment I used to build the DAC pipeline at a high level. I’ll explain certain concepts and include links to useful references for you along the way.

Creating an Access Key in Sumo Logic

Terraform needs to be able to authenticate with Sumo Logic before it can manage resources. You can create an Access Key in Sumo Logic by following these instructions.

Click to expand image. Managing Access Keys in Sumo Logic

Ingesting Okta system logs into Sumo Logic

You can sign up for a free 30-day trial of Sumo Logic here and a free Okta developer account here.

Follow these instructions to create an Okta API key and configure Sumo Logic to ingest Okta system logs.

Once the above steps are complete, you’ll see Okta system log events flowing into Sumo Logic like what’s shown in the screenshot below.

Ingesting Okta system logs in Sumo Logic

GitHub

Create a new detection-as-code GitHub repository and clone it locally. Create a GitHub personal access token that has read & write permissions for issues in the new repo. This token will be used by Tines and in our CI/CD workflows to create and update alerts (GitHub issues).

Terraform

Install Terraform on your local host using the instructions here and verify the installation by executing terraform -help.

threatpunter ~ % terraform -help
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

Detection Engineering with Terraform and Sumo Logic

Configuring the Terraform provider for Sumo Logic

Terraform relies on plugins called providers to interact with cloud providers, SaaS providers, and other APIs. I’ll be using the Sumo Logic provider to manage resources such as detections, folders, and webhook connections.

The provider for Sumo Logic is configured in main.tf and is shown in the code block below. I’ve included comments to explain each component.

Terraform can read in environment variables that have the prefix, TF_VAR_, which is what I’m doing here for to authenticate with Sumo Logic. For example, SUMOLOGIC_ACCESSID is read from environment variable TF_VAR_SUMOLOGIC_ACCESSID.

terraform {
  required_providers {
    # Sumo Logic Provider docs: https://registry.terraform.io/providers/SumoLogic/sumologic/latest/docs
    sumologic = {
      source  = "sumoLogic/sumologic"
      version = "2.24.0"
    }
  }
  # Required Terraform version.
  required_version = ">= 1.5.2"
}

# Setup authentication variables. Docs: https://registry.terraform.io/providers/SumoLogic/sumologic/latest/docs#authentication
variable "SUMOLOGIC_ACCESSID" {
  type        = string
  description = "Sumo Logic Access ID"
}
variable "SUMOLOGIC_ACCESSKEY" {
  type        = string
  description = "Sumo Logic Access Key"
  sensitive   = true
}

# Configure the Sumo Logic Provider
provider "sumologic" {
  access_id   = var.SUMOLOGIC_ACCESSID
  access_key  = var.SUMOLOGIC_ACCESSKEY
  environment = "us1"
}

The Terraform working directory is initialized by executing the terraform init command, which includes installing provider plugins defined in main.tf.

(venv) threatpunter detection-as-code-example % terraform init

Initializing the backend...

Initializing provider plugins...
- Finding sumologic/sumologic versions matching "2.24.0"...
- Installing sumologic/sumologic v2.24.0...
- Installed sumologic/sumologic v2.24.0 (signed by a HashiCorp partner, key ID 58C1F2777F8E2367)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

Creating a webhook connection in Sumo Logic

What’s a Sumo Logic webhook connection? As per their documentation:

Webhook connections allow you to send Sumo Logic alerts to third-party applications that accept incoming webhooks.

For this project, I’m going to configure a webhook in Sumo Logic to send alert payloads to a Tines story for processing. More on the Tines components later.

I’ve defined a Sumo Logic connection resource in Terraform file sumologic_connections.tf and included it in the code block below.

When a detection is configured to use this connection, Sumo Logic will send alert payloads (JSON objects) to my Tines webhook via an HTTP POST request.

Note, the Tines webhook URL is being read from environment variable TF_VAR_TINES_WEBHOOK_URL_FOR_SUMOLOGIC_ALERTS, because it contains a secret and I don’t want just anyone to send payloads to my Tines story. That’d be bad.

# sumologic_connection docs: https://registry.terraform.io/providers/SumoLogic/sumologic/latest/docs/resources/connection

# Read the Tines webhook URL from the environment variable TF_VAR_TINES_WEBHOOK_URL_FOR_SUMOLOGIC_ALERTS.
variable "TINES_WEBHOOK_URL_FOR_SUMOLOGIC_ALERTS" {
  type        = string
  description = "Tines webhook URL to send Sumo Logic alerts to."
  sensitive   = true
}

# Sumo Logic webhook connection to send alerts to Tines.
resource "sumologic_connection" "tines_webhook" {
  type            = "WebhookConnection"
  name            = "Tines Webhook - Create GitHub Issues from Sumo Logic Alerts."
  description     = "Connection to send alert payloads to Tines webhook."
  url             = var.TINES_WEBHOOK_URL_FOR_SUMOLOGIC_ALERTS
  custom_headers  = { "Content-Type" : "application/json" }
  # The default payload (JSON string) from Sumo Logic to send to Tines webhook.
  default_payload = <<JSON
{
  "rule.name": "{{Name}}",
  "rule.description": "{{Description}}",
  "query.url": "{{QueryURL}}",
  "query": "{{Query}}",
  "trigger.range": "{{TriggerTimeRange}}",
  "trigger.name": "{{TriggerTime}}",
  "alert.payload": "{{ResultsJson}}"
}
JSON
  webhook_type    = "Webhook"
}

Now that we’ve defined our webhook connection “resource” in Terraform, let’s apply those changes to our infrastructure and review the resulting artifacts in Sumo Logic’s UI.

We can execute terraform plan to show changes required by our current Terraform configuration. In this example, we have one resource pending creation, the webhook I defined above.

In this example, we’ll go right ahead and execute terraform apply to create or update our infrastructure. The output from this command is shown below.

As you can see, the resource named sumologic_connection.tines_webhook was created successfully 👏.

threatpunter detection-as-code % terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # sumologic_connection.tines_webhook will be created
  + resource "sumologic_connection" "tines_webhook" {
      + custom_headers     = {
          + "Content-Type" = "application/json"
        }
      + default_payload    = jsonencode(
            {
              + "alert.payload"    = "{{ResultsJson}}"
              + query              = "{{Query}}"
              + "query.url"        = "{{QueryURL}}"
              + "rule.description" = "{{Description}}"
              + "rule.name"        = "{{Name}}"
              + "trigger.name"     = "{{TriggerTime}}"
              + "trigger.range"    = "{{TriggerTimeRange}}"
            }
        )
      + description        = "Connection to send alert payloads to Tines webhook."
      + id                 = (known after apply)
      + name               = "Tines Webhook - Create GitHub Issues from Sumo Logic Alerts."
      + resolution_payload = (known after apply)
      + type               = "WebhookConnection"
      + url                = (sensitive value)
      + webhook_type       = "Webhook"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

sumologic_connection.tines_webhook: Creating...
sumologic_connection.tines_webhook: Creation complete after 0s [id=000000000003948D]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

We can view the webhook connection in Sumo Logic’s UI by navigating to Manage Data • Monitoring • Connections. This is an example of how Sumo Logic resources can be managed via Terraform.

Viewing the Tines webhook connection in Sumo Logic

Creating a new Okta detection rule

We’re ready to create our first detection rule as code in Terraform’s configuration language.

For this detection use case, let’s imagine that we work at a company where Okta administrator permissions should only ever be assigned to user account names that begin with admin. like admin.alice.smith@threatpunter.com and never standard user accounts like alice.smith@threatpunter.com.

We want to detect and alert on administrator permissions being assigned to non-admin accounts for the reasons below. I’m sure there are more, but these are the first few that came to mind.

• This is a violation of our fictitious company’s security standard/policy.
• It could indicate that an attacker has compromised an Okta administrator account and is assigning administrator permissions to another account to maintain persistence.
• An insider threat could be assigning administrator permissions to another account to maintain persistence or carry out actions using another account to evade detection.

I’m going to create the detection in the form of a Monitor in Sumo Logic. In a nutshell, a Sumo Logic Monitor tracks log data in real time and can send notifications when certain events happen. So our “detection rule” will technically be a “Monitor” in Sumo Logic speak.

Below is the logic for the detection rule. We want to generate an alert when the event user.account.privilege.grant appears in Okta’s system log where the target account name does not start with admin.

user.account.privilege.grant is the event that Okta logs when an administrator role is assigned to a user account.

_sourceCategory="okta" user.account.privilege.grant
| where eventType="user.account.privilege.grant" AND !(%"target[0].alternateId" matches /^admin\./)

I’ve created the detection rule as a Sumo Logic Monitor resource in Terraform file, rule_okta_administrator_role_assigned_to_non_admin_user_account.tf and included it in a code block below.

This resource utilizes a Terraform module, modules/sumo_monitor/main.tf. Thanks again to Adam Cole for sharing a sample module in his presentation. I based my module off of his example and updated the configuration & syntax for support with the latest Terraform provider for Sumo Logic Monitors (sumologic_monitor).

The detection rule is made up of the following components:

• Metadata about the detection for the security team to utilize such as maturity, version, rule_id, and when the rule was created/updated.
• Standard information like name, description, Sumo Logic query, and which webhook should be used to send alert payloads to Tines.
• The Sumo Logic folder (sumologic_monitor_folder.detections.id) where the detection (Monitor) should be created. We’ll define this folder in Terraform soon.

/*
name = "Administrator Role Assigned to Non-Admin User Account"
creation_date = "2023/07/22"
updated_date = "2023/07/22"
maturity = "Production"
version = "1.0.0"
rule_id: "97d6c856-93e8-40e3-9af7-f797a5c1435b"
platform: "Sumo Logic"
*/

module "rule_okta_administrator_role_assigned_to_non_admin_user_account" {
  source                 = "./modules/sumo_monitor"
  standard_name          = "Administrator Role Assigned to Non-Admin User Account"
  standard_description   = "Identifies when an administrator role is assigned to a non-admin Okta user account i.e. a standard user account that does not follow our company's admin account naming conventions. Investigate using playbook PB-100."
  standard_query         = <<EOF
_sourceCategory="okta" user.account.privilege.grant
| where eventType="user.account.privilege.grant" AND !(%"target[0].alternateId" matches /^admin\./)
EOF
  standard_folder        = sumologic_monitor_folder.detections.id
  tines_webhook          = sumologic_connection.tines_webhook.id
  tines_webhook_override = <<EOF
{
  "rule.name": "{{Name}}",
  "rule.description": "{{Description}}",
  "query.url": "{{QueryURL}}",
  "query": "{{Query}}",
  "trigger.range": "{{TriggerTimeRange}}",
  "trigger.name": "{{TriggerTime}}",
  "alert.payload": "{{ResultsJson}}"
}
EOF
}

Our rules will be stored in a folder called Detections. This sumologic_monitor_folder resource is defined in sumologic_folders.tf.

Validating and applying the Terraform configuration

Execute the commands below to validate and format your Terraform configuration.

threatpunter detection-as-code % terraform validate
Success! The configuration is valid.

threatpunter detection-as-code % terraform fmt -recursive

Tip: The Terraform extension for Visual Studio Code is helpful for syntax validation & highlighting and formatting while you’re working with Terraform’s configuration language.

Execute terraform apply to apply changes to the Sumo Logic environment. Note, I’ve omitted lines in the output of the command below for brevity with (…).

The folder for our detections (sumologic_monitor_folder.detections) was created followed by the detection rule (module.rule_okta_administrator_role_assigned_to_non_admin_user_account.sumologic_monitor.alert).

threatpunter detection-as-code % terraform apply
sumologic_connection.tines_webhook: Refreshing state... [id=000000000003948D]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # sumologic_monitor_folder.detections will be created
  + resource "sumologic_monitor_folder" "detections" {
      + content_type = "Folder"
      (...)

  # module.rule_okta_administrator_role_assigned_to_non_admin_user_account.sumologic_monitor.alert will be created
  + resource "sumologic_monitor" "alert" {
      (...)

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

sumologic_monitor_folder.detections: Creating...
sumologic_monitor_folder.detections: Creation complete after 2s [id=000000000003150F]
module.rule_okta_administrator_role_assigned_to_non_admin_user_account.sumologic_monitor.alert: Creating...
module.rule_okta_administrator_role_assigned_to_non_admin_user_account.sumologic_monitor.alert: Creation complete after 1s [id=0000000000031510]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Let’s review the detection in Sumo Logic. Looks good!

Wrap up

That’s it for part one! Thanks for making it this far — I hope you enjoyed it.

This post covered the following:

• The principles and benefits of Detection-as-Code (DAC) including references to awesome existing work by members of the security community.
• A set of tools that can be used to build and implement a DAC pipeline from scratch.
• How to configure an arsenal of tools to build the foundations for a DAC pipeline.
• A process for doing Detection Engineering as code using Terraform and Sumo Logic with a practical Okta threat detection use case.

Please check out the “existing work” section below for references to DAC and Detection Engineering resources that helped me on my journey throughout this project.

As a reminder, you can find my example code in the following GitHub repo: threat-punter/detection-as-code-example.

In part two of this series, I’ll provide explanations and code for creating CI/CD workflows to test our DAC pipeline and deploying changes. I’ll walk through a process for handling alert payloads with Tines, testing detections, and validating the alert pipeline. Finally, I’ll demonstrate how the DAC pipeline works end-to-end with a practical use case — detecting & responding to suspicious Okta behavior.

Existing work

Our progress and achievements are based on the foundations laid by those who came before us. Below is a list of resources that helped me on my journey throughout this project.

If you feel that I’ve missed any references, please let me know and I’ll look at adding them.

• Automating Detection-as-Code — John Tuckner
• Detection as Code: Detection Development Using CI/CD — Patrick Bareiss, Jose Hernandez
• Detection-as-code: Why it works and where to start — Kyle Bailey
• LifeOmic: Detection as code with Sumo Logic and Terraform — Adam Cole
• The Future Of Security Operations Roadshow: Using Github For Automated Case Management — Chris Long
• Can We Have “Detection as Code”? — Anton Chuvakin
• Elastic’s Detection Rules
• Splunk’s Detection Rules
• Google’s Detection Rules for Chronicle

Acknowledgements

• A huge thank you to the talented UX Researcher, Daphne Villarreal for reviewing this blog series and providing precious feedback.
• John Tuckner of Tines Labs for sharing knowledge and methodologies for automating security operations workflows freely & openly with the security community.
• Anton Ovrutsky for pointing me to the Terraform provider for Sumo Logic.