UDM Event Validation

Chris Martin (@thatsiemguy)
12 min readJul 6, 2023

--

The Universal Data Model (UDM) is the schema of Chronicle SIEM. Within UDM, Metadata Event Type represents a high-level description of the original log; however, in order to create a UDM Event or UDM Entity Event, you need to pass Validation.

What is Validation? How do you pass UDM Validation?

In this post I will attempt explore the Validation requirements for UDM Events.

At the time of writing UDM has 103 distinct Event Types

Overview

Understanding the UDM schema, and particularly UDM Validation, helps to solve the following challenges you may encounter when working with Chronicle SIEM, such as:

  • Parsing in Chronicle SIEM
    - be able to quickly identify which fields need be populated to assign the UDM metadata.event_type of choice
    - understand why a given UDM metadata.event_type is using a given event_type, e.g., a required Validation field is not present in the original log, and hence the Parser author choose the next best best (least worst) option, e.g., service_creation requires a machine identifier but perhaps you only have a user in the log, so you can’t assign the event_type
  • Chronicle Data Lake, aka BigQuery
    _ in order to write a SQL statement you’ll often need to coalesce fields together, and by having insight into which UDM fields will be present, e.g., in PROCESS_PRIVILEGE_ESCALATION requires a user be in principal, yet USER_LOGIN requires the user be in target
  • Chronicle Dashboards
    - As above, when building a Chronicle SIEM dashboard having insights into the UDM fields that will be present for a given event_type helps to ensure widgets are populated as needed
  • Chronicle SOAR
    - if you’re building a SOAR Playbook integration, having confidence on which fields will be present and mapped into the SOAR Ontology for a given event type can help
    - you can use the SOAR ontology mapping to look up the UDM to SOAR Ontology mapping, but you still need to know which UDM

In this post I document how you can test and verify UDM Validation yourself, but the summary of the results are as follows, with the below matrix providing the minimal viable UDM fields needed in order to populate a given UDM metadata.event_type.

An unofficial UDM Validation matrix

⚠️ Note, just because you can create a minimal UDM event, you should follow the UDM Usage guidance where possible to create an optimal UDM event, i.e., a UDM event that doesn’t only meet the Validation requirements but adds additional fields that provide futher context.

Creating a UDM Event

Before we cover Validation its useful to recap how you create UDM Events:

  1. Use a Parser to normalize a log to UDM.
  2. Create a UDM object as code via the Ingestion API

📝 I’ve written about these two topics before. For related reading, please see the following:

Using either approach will require you assign a Metadata Event Type, e.g., here’s a Parser excerpt assigning the Event Type as NETWORK_CONNECTION:

  mutate {
replace => {
"event1.idm.read_only_udm.metadata.event_type" => "NETWORK_CONNECTION"
}
}

However, when testing or submitting the Parser it returns the following error:

generic::unknown: 
invalid event 0: LOG_PARSING_GENERATED_INVALID_EVENT:
"generic::invalid_argument: udm validation failed: target field is not set"

In this case, you have attempted to create a metadata.event_type of NETWORK_CONNECTION and not populated the required target UDM object.

How do you know what fields you should populate in order to pass UDM Validation when writing a Chronicle Parser or creating a UDM Event?

The official documentation that covers this topic is the UDM Usage guide; however, as you’ll see in this post there are 1) validation requirements that are not documented, and 2) the UDM Usage guide proposes an optimal UDM event, it does not always clearly cover what the actual minimal UDM is.

💡 The cbn_cli.py or Parser Management feature provide more detail on the missing Validation than using udmevents method via the Ingestion API, e.g., using udmevents you receive the INVALID_ARGUMENT error, but with a GROK Parser you would receive the following, more detailed, error:

generic::unknown: invalid event 0: 
LOG_PARSING_GENERATED_INVALID_EVENT:
"generic::invalid_argument: udm validation failed: target field is not set"

UDM Validation

UDM Metadata Event Types have Validation requirements. These are mandatory fields that must be present in order to assign an Event Type.

The purpose of Validation is to ensure a base level of quality for UDM Event Types. For example, a USER_LOGIN event without a useris not useful.

Chronicle includes documentation on UDM Usage that provides a level of detail around Validation; however, as you’ll see in this post, it is not exhaustive.

Testing UDM Metadata Event Types

To test all 103 metadata.event_type I used the following approach:

  • a Colab notebook to create a minimal UDM event for each Metadata Event Type
  • the Ingestion API to submit the above event
  • verified against the the Chronicle UDM Usage guide where documentation was available

The test harness is included as the bottom of this document, and below follows the output of each test. Where available links to the UDM Usage documentation are included, but one of the observations of the tests is that not all UDM Event Validation requirements are documented, and in some cases appear to not match

Ultimately, below is a useful reference if you’re building a new Parser or UDM event via code to quickly double check the required fields.

⚠️ This is accurate at this moment in time, July 2023, and it may or will become out of date as things change, but you will have the process details below to quickly work out validation if not available in the UDM Usage guide.

ANALYST_ and DEVICE_

Notes:

  • I could not find any examples of ANALYST_ events, and so have omitted testing this event_type
  • DEVICE_ does not appear an active UDM event_type and failed when I tried to use it.

EMAIL

Event Types:

  • EMAIL_TRANSACTION
  • EMAIL_UNCATEGORIZED
  • EMAIL_URL_CLICK [Deprecated]

Notes:

  • network.email is NOT an Aliasing field
    - while the UDM Usage guidance is to not populate users into Principal or Target, if you require Aliasing and Enrichment for Search or Detection you will need to index into these fields, i.e., use a Parser Extension
  • about.file.hash is NOT an Indexed field for File Hashes
    - If you require Aliasing and Enrichment on Hashes, e.g., File Attachments, you will need to index into Principal or Target, i.e., Hash view won’t be populated from an attachment hash normalized to about.file.hash
  • EMAIL_URL_CLICK is marked as Deprecated
{
"metadata": {
"event_timestamp": "2023-07-06T06:26:10.448584Z",
"event_type": "EMAIL_TRANSACTION"
},
"principal": {
"ip": [
"1.2.3.4"
]
}
}

FILE

Event Types:

  • FILE_CREATION
  • FILE_DELETION
  • FILE_MODIFICATION
  • FILE_READ
  • FILE_OPEN
  • FILE_UNCATEGORIZED

Minimum UDM Object:

{
"metadata": {
"event_timestamp": "2023-07-03T12:14:38.164507Z",
"event_type": "FILE_COPY"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"file": {
"full_path": "foo.bar"
}
}
}

Event Types:

  • FILE_MOVE
  • FILE_SYNC

Minimum UDM Object:

{
"metadata": {
"event_timestamp": "2023-07-03T12:45:02.557878Z",
"event_type": "FILE_MOVE"
},
"src": {
"file": {
"full_path": "foo.bar"
}
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"file": {
"full_path": "foo.bar"
}
}
}

GENERIC_EVENT

Event Types:

  • GENERIC_EVENT

Minimal UDM Object:

{
"metadata": {
"event_timestamp": "2023-07-03T12:52:50.379205Z",
"event_type": "GENERIC_EVENT"
}
}

GROUP

Event Types:

  • GROUP_CREATION
  • GROUP_DELETION
  • GROUP_MODIFICATION
  • GROUP_UNCATEGORIZED

Minimal UDM Object:

{
"metadata": {
"event_timestamp": "2023-07-03T13:01:10.957803Z",
"event_type": "GROUP_CREATION"
},
"principal": {
"user": {
"userid": "pinguino"
}
},
"target": {
"group": {
"group_display_name": "foobar_users"
}
}
}

MUTEX

Event Types:

  • MUTEX_CREATION
  • MUTEX_UNCATEGORIZED

Notes:

  • You can only pass UDM Validation with a process object in principal, you can’t pass UDM Validation with a process in target
  • resource is marked as deprecated
{
"metadata": {
"event_timestamp": "2023-07-03T13:22:30.048989Z",
"event_type": "MUTEX_UNCATEGORIZED"
},
"principal": {
"hostname": "pinguino-01",
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"target": {
"resource": {
"name": "mutex",
"type": "MUTEX"
}
}
}

NETWORK

Event Types:

  • NETWORK_CONNECTION
  • NETWORK_FLOW
  • NETWORK_FTP
  • NETWORK_HTTP
  • NETWORK_SMTP

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-03T14:06:30.781880Z",
"event_type": "NETWORK_CONNECTION"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"hostname": "jamon-01"
}
}

Event Types:

  • NETWORK_DHCP

Notes:

  • network.dhcp uses the field names as per the DHCP rfc — https://tools.ietf.org/html/rfc2131
    - op = BOOTREQUEST, BOOTREPLY
    - chaddr = client hardware address
    - ciaddr = client IP address, only if client is BOUND, RENEW, REBINDING to ARP request
    - yiaddr = your client IP address
  • Chronicle SIEM only evaluates BOOTREQUEST or BOOTREPLY for Asset aliasing

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-04T18:52:22.142364Z",
"event_type": "NETWORK_DHCP"
},
"principal": {
"ip": [
"1.2.3.4"
]
},
"network": {
"application_protocol": "DHCP",
"dhcp": {
"opcode": "BOOTREPLY"
}
}
}

Event Types:

  • NETWORK_DNS

Notes

| #   | Type   | Meaning                                              |
|-----|--------|------------------------------------------------------|
| 1 | A | A host address |
| 2 | NS | An authoritative name server |
| 3 | MD | A mail destination (Obsolete - use MX) |
| 4 | MF | A mail forwarder (Obsolete - use MX) |
| 5 | CNAME | The canonical name for an alias |
| 6 | SOA | Marks the start of a zone of authority |
| 7 | MB | a mailbox domain name (EXPERIMENTAL) |
| 8 | MG | A mail group member (EXPERIMENTAL) |
| 9 | MR | A mail rename domain name (EXPERIMENTAL) |
| 10 | NULL | A null RR (EXPERIMENTAL) |
| 11 | WKS | A well known service description |
| 12 | PTR | A domain name pointer |
| 13 | HINFO | Host information |
| 14 | MINFO | Mailbox or mail list information |
| 15 | MX | Mail exchange |
| 16 | TXT | Text strings |
| 252 | AXFR | A request for a transfer of an entire zone |
| 253 | MAILB | A request for mailbox-related records (MB, MG or MR) |
| 254 | MAILA | A request for mail agent RRs (Obsolete - see MX) |
| 255 | * | A request for all records |

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-04T18:54:15.943301Z",
"event_type": "NETWORK_DNS"
},
"principal": {
"ip": [
"1.2.3.4"
]
},
"network": {
"application_protocol": "DNS",
"dns": {
"questions": {
"name": "www.acme.com"
}
}
}
}

PROCESS

Event Types:

  • PROCESS_INJECTION
  • PROCESS_LAUNCH
  • PROCESS_OPEN
  • PROCESS_TERMINATION
  • PROCESS_UNCATEGORIZED

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-03T20:24:29.437724Z",
"event_type": "PROCESS_LAUNCH"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}

Event Types:

  • PROCESS_MODULE_LOAD

Notes:

  • The documentation omits that UDM Validation requires a process in principal as well as target
{
"metadata": {
"event_timestamp": "2023-07-03T20:40:29.854097Z",
"event_type": "PROCESS_MODULE_LOAD"
},
"principal": {
"hostname": "pinguino-01",
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}

Event Types:

  • PROCESS_PRIVILEGE_ESCALATION

Notes:

  • The documentation omits that a user and process is required in principal

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-04T19:04:12.573338Z",
"event_type": "PROCESS_PRIVILEGE_ESCALATION"
},
"principal": {
"user": {
"userid": "pinguino"
},
"hostname": "pinguino-01",
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}

REGISTRY

Event Types:

  • REGISTRY_CREATION
  • REGISTRY_DELETION
  • REGISTRY_MODIFICATION
  • REGISTRY_UNCATEGORIZED
{
"metadata": {
"event_timestamp": "2023-07-04T19:12:19.489913Z",
"event_type": "REGISTRY_MODIFICATION"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"registry": {
"registry_key": "foo.bar"
}
}
}

RESOURCE

Event Types:

  • RESOURCE_CREATION
  • RESOURCE_DELETION
  • RESOURCE_PERMISSIONS_CHANGE
  • RESOURCE_READ
  • RESOURCE_WRITTEN

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-04T20:19:53.394745Z",
"event_type": "RESOURCE_CREATION"
},
"principal": {
"user": {
"userid": "pinguino"
}
},
"target": {
"resource": {
"name": "foo.bar"
}
}
}

SCAN

Event Types:

  • SCAN_FILE

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-06T06:57:08.443142Z",
"event_type": "SCAN_FILE"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"file": {
"full_path": "foo.bar"
}
}
}

Event Types:

  • SCAN_HOST
  • SCAN_NETWORK
  • SCAN_UNCATEGORIZED

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-06T06:58:32.624239Z",
"event_type": "SCAN_HOST"
},
"principal": {
"hostname": "pinguino-01"
}
}

Event Types:

  • SCAN_PROCESS
  • SCAN_PROCESS_BEHAVIORS

Notes:

  • SCAN_PROCESS_BEHAVIORS is deprecated and SCAN_PROCESS should be used instead.

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-06T06:59:17.690425Z",
"event_type": "SCAN_PROCESS"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}

Event Types:

  • SCAN_VULN_HOST
  • SCAN_VULN_NETWORK
{
"metadata": {
"event_timestamp": "2023-07-04T20:50:08.288914Z",
"event_type": "SCAN_VULN_NETWORK"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"extensions": {
"vulns": {
"vulnerabilities": {
"description": "Potentially Bad Traffic"
}
}
}
}

Event Types:

  • SCHEDULED_TASK_CREATION
  • SCHEDULED_TASK_DELETION
  • SCHEDULED_TASK_DISABLE
  • SCHEDULED_TASK_MODIFICATION
  • SCHEDULED_TASK_UNCATEGORIZED

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-04T21:02:05.417757Z",
"event_type": "SCHEDULED_TASK_CREATION"
},
"principal": {
"hostname": "pinguino-01",
"user": {
"userid": "pinguino"
}
},
"target": {
"resource": {
"resource_type": "TASK"
}
}
}

SERVICE

Event Types:

  • SERVICE_CREATION
  • SERVICE_DELETION
  • SERVICE_MODIFICATION
  • SERVICE_START
  • SERVICE_STOP
  • SERVICE_UNSPECIFIED

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-05T14:11:06.526110Z",
"event_type": "SERVICE_UNSPECIFIED"
},
"principal": {
"hostname": "foo.bar"
},
"target": {
"user": {
"userid": "pinguino-01"
},
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}

SETTING

Event Types:

  • SETTING_CREATION
  • SETTING_DELETION
  • SETTING_MODIFICATION
  • SETTING_UNCATEGORIZED

Notes:

  • resource.type is deprecated and resource_type should be used

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-05T14:15:27.070909Z",
"event_type": "SETTING_MODIFICATION"
},
"principal": {
"hostname": "foo.bar"
},
"target": {
"resource": {
"type": "SETTING",
"name": "foo.bar"
}
}
}

STATUS

Event Types:

  • STATUS_HEARTBEAT
  • STATUS_SHUTDOWN
  • STATUS_STARTUP
  • STATUS_UNCATEGORIZED
  • STATUS_UPDATE

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-05T14:19:15.126817Z",
"event_type": "STATUS_HEARTBEAT"
},
"principal": {
"hostname": "foo.bar"
}
}

SYSTEM

Event Types:

  • SYSTEM_AUDIT_LOG_WIPE
  • SYSTEM_AUDIT_LOG_UNCATEGORIZED

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-05T16:09:29.647627Z",
"event_type": "SYSTEM_AUDIT_LOG_WIPE"
},
"principal": {
"hostname": "foo.bar",
"user": {
"userid": "pinguino-01"
}
}
}

USER

Event Types:

  • USER_CHANGE_PASSWORD
  • USER_CHANGE_PERMISSIONS
  • USER_CHANGE_PASSWORD
  • USER_CHANGE_PERMISSIONS

Minimal UDM Event:


{
"metadata": {
"event_timestamp": "2023-07-05T16:20:16.408189Z",
"event_type": "USER_CHANGE_PERMISSIONS"
},
"principal": {
"user": {
"userid": "pinguino-01"
}
},
"target": {
"user": {
"userid": "jamon"
}
}
}

Event Types:

  • USER_COMMUNICATION
  • USER_UNCATEGORIZED
{
"metadata": {
"event_timestamp": "2023-07-05T16:27:22.013276Z",
"event_type": "USER_COMMUNICATION"
},
"principal": {
"user": {
"userid": "pinguino-01"
}
}
}

Event Types:

  • USER_BADGE_IN
{
"metadata": {
"event_timestamp": "2023-07-05T20:31:18.303584Z",
"event_type": "USER_BADGE_IN"
},
"target": {
"hostname": "jamon-01"
},
"extensions": {
"auth": {
"type": "MACHINE"
}
}
}

Event Types:

  • USER_CREATION
  • USER_DELETION
{
"metadata": {
"event_timestamp": "2023-07-05T20:44:10.642556Z",
"event_type": "USER_CREATION"
},
"principal": {
"hostname": "foo.bar",
"user": {
"userid": "pinguino"
}
},
"target": {
"user": {
"userid": "jamon"
}
}
}

Event Types:

  • USER_LOGIN
  • USER_LOGOUT
{
"metadata": {
"event_timestamp": "2023-07-05T20:48:50.657334Z",
"event_type": "USER_LOGIN"
},
"target": {
"user": {
"userid": "jamon"
}
},
"extensions": {
"auth": {
"type": "MACHINE"
}
}
}

Event Types:

  • USER_RESOURCE_ACCESS
  • USER_RESOURCE_CREATION
  • USER_RESOURCE_DELETION
  • USER_RESOURCE_UPDATE_CONTENT
  • USER_RESOURCE_UPDATE_PERMISSIONS
  • USER_STATS

Notes:

  • USER_STATS is marked as deprecated.

Minimal UDM Event:

{
"metadata": {
"event_timestamp": "2023-07-06T07:16:41.350672Z",
"event_type": "USER_RESOURCE_ACCESS"
},
"principal": {
"user": {
"userid": "pinguino"
}
},
"target": {
"resource": {
"name": "foo.bar"
}
}
}

Test Harness Setup

🐉 I am not a developer, and the below Python scripts are not production usage code, and intended only for learning and educational purposes.

You will require a JSON Developer Service Account with Ingestion API IAM permissions.

  1. Configure input variables:

# Chronicle JSON Developer Service account with Ingestion API IAM permissions
# - specifically: i) Malachite Ingestion Collector, ii) Malachite Ingestion Integration User
CREDENTIALS_FILE='/content/ing-<tla>-<random>.json'

# Can be found in your UI under Settings
CUSTOMER_ID="<your customer guid>"

# Can be 'na', 'europe', or 'asia'.
REGION="us"

2. Run the following helper functions:

import datetime
import json

def now():

# Get the current time
current_time = datetime.datetime.now()

# Format the current time
formatted_time = current_time.strftime('%Y-%m-%dT%H:%M:%S.%fZ')

return formatted_time

def subtract_offset(timestamp, offset):
"""
Subtracts the offset from the timestamp and returns the resulting timestamp.

Args:
timestamp: The timestamp to subtract the offset from.
offset: The offset to subtract from the timestamp.

Returns:
The resulting timestamp.
"""

# Convert the timestamp to a datetime object.
datetime_object = datetime.datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%fZ")

# Convert the offset to a timedelta object.
timedelta_object = datetime.timedelta(days=-offset)

# Subtract the offset from the datetime object.
new_datetime_object = datetime_object - timedelta_object

# Convert the new datetime object to a timestamp.
#new_timestamp = new_datetime_object.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
new_timestamp = new_datetime_object.isoformat(timespec='milliseconds') + "Z"

return new_timestamp

def region(region):
REGIONS = {
"europe": "https://europe-malachiteingestion-pa.googleapis.com",
"asia": "https://asia-southeast1-malachiteingestion-pa.googleapis.com",
"us": "https://malachiteingestion-pa.googleapis.com",
}
if region not in REGIONS:
raise ValueError("Invalid region")
return REGIONS[region]

def auth(credentials_file):
from google.auth.transport import requests
from google.oauth2 import service_account

AUTHORIZATION_SCOPES = ["https://www.googleapis.com/auth/chronicle-backstory","https://www.googleapis.com/auth/malachite-ingestion"]

credentials = service_account.Credentials.from_service_account_file(
str(credentials_file),
scopes=AUTHORIZATION_SCOPES)
return requests.AuthorizedSession(credentials)

def create_udm_event(event):
import requests

authenticated = auth(CREDENTIALS_FILE)

data = {}
data["customer_id"] = CUSTOMER_ID
data["events"] = []
data["events"].append(json.loads(event))
json_data = json.dumps(data)

print ('Event to send:\n' + json_data)

http_endpoint = '{}/v2/udmevents:batchCreate'.format(region(REGION))
headers = {'content-type': 'application/json'}
r = authenticated.post(url=http_endpoint, data=json_data, headers=headers)

print(r.text)
return r

3. For each Metadata Event Type run the below test harness to find the minimal viable UDM Event that passes UDM Validation:


metadata_event_type="FILE_MOVE"
principal_hostname="pinguino-laptop"
src_file_full_path="bar.foo"
target_file_full_path="foo.bar"

event={
'metadata':{
'event_timestamp': now(),
'event_type': metadata_event_type,
},
'src': {
'file': {
'full_path': target_file_full_path
}
},
'principal':{
'hostname':principal_hostname
},
'target': {
'file': {
'full_path': target_file_full_path
}
}
}

print(json.dumps(event, indent=4))

create_udm_event(json.dumps(event))

If your minimal UDM Event is successful, you’ll receive an API response as follows:

<Response [200]>

If there is a UDM Validation error you will receive an error as follows:

"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}

Summary

UDM Validation is a core concept to understand, and impacts many dimensions of your using the Chronicle SecOps platform. Hopefully this post helps to explain what Validation is, and how you can validate Validation itself.

Please note, this isn’t an official Chronicle product documentation, but if you do find any issues or have comments, please let me know.

--

--