UDM Event Validation
The Universal Data Model (UDM) is the schema of Chronicle SIEM. Within UDM, Metadata Event Type represents a high-level description of the original log; however, in order to create a UDM Event or UDM Entity Event, you need to pass Validation.
What is Validation? How do you pass UDM Validation?
In this post I will attempt explore the Validation requirements for UDM Events.
Overview
Understanding the UDM schema, and particularly UDM Validation, helps to solve the following challenges you may encounter when working with Chronicle SIEM, such as:
- Parsing in Chronicle SIEM
- be able to quickly identify which fields need be populated to assign the UDMmetadata.event_type
of choice
- understand why a given UDMmetadata.event_type
is using a givenevent_type
, e.g., a required Validation field is not present in the original log, and hence the Parser author choose the next best best (least worst) option, e.g.,service_creation
requires a machine identifier but perhaps you only have a user in the log, so you can’t assign theevent_type
- Chronicle Data Lake, aka BigQuery
_ in order to write a SQL statement you’ll often need to coalesce fields together, and by having insight into which UDM fields will be present, e.g., inPROCESS_PRIVILEGE_ESCALATION
requires a user be inprincipal
, yetUSER_LOGIN
requires the user be intarget
- Chronicle Dashboards
- As above, when building a Chronicle SIEM dashboard having insights into the UDM fields that will be present for a givenevent_type
helps to ensure widgets are populated as needed - Chronicle SOAR
- if you’re building a SOAR Playbook integration, having confidence on which fields will be present and mapped into the SOAR Ontology for a given event type can help
- you can use the SOAR ontology mapping to look up the UDM to SOAR Ontology mapping, but you still need to know which UDM
In this post I document how you can test and verify UDM Validation yourself, but the summary of the results are as follows, with the below matrix providing the minimal viable UDM fields needed in order to populate a given UDM metadata.event_type
.
⚠️ Note, just because you can create a minimal UDM event, you should follow the UDM Usage guidance where possible to create an optimal UDM event, i.e., a UDM event that doesn’t only meet the Validation requirements but adds additional fields that provide futher context.
Creating a UDM Event
Before we cover Validation its useful to recap how you create UDM Events:
- Use a Parser to normalize a log to UDM.
- Create a UDM object as code via the Ingestion API
📝 I’ve written about these two topics before. For related reading, please see the following:
Using either approach will require you assign a Metadata Event Type, e.g., here’s a Parser excerpt assigning the Event Type as NETWORK_CONNECTION:
mutate {
replace => {
"event1.idm.read_only_udm.metadata.event_type" => "NETWORK_CONNECTION"
}
}
However, when testing or submitting the Parser it returns the following error:
generic::unknown:
invalid event 0: LOG_PARSING_GENERATED_INVALID_EVENT:
"generic::invalid_argument: udm validation failed: target field is not set"
In this case, you have attempted to create a metadata.event_type
of NETWORK_CONNECTION
and not populated the required target UDM object.
How do you know what fields you should populate in order to pass UDM Validation when writing a Chronicle Parser or creating a UDM Event?
The official documentation that covers this topic is the UDM Usage guide; however, as you’ll see in this post there are 1) validation requirements that are not documented, and 2) the UDM Usage guide proposes an optimal UDM event, it does not always clearly cover what the actual minimal UDM is.
💡 The cbn_cli.py or Parser Management feature provide more detail on the missing Validation than using udmevents method via the Ingestion API, e.g., using udmevents you receive the INVALID_ARGUMENT error, but with a GROK Parser you would receive the following, more detailed, error:
generic::unknown: invalid event 0:
LOG_PARSING_GENERATED_INVALID_EVENT:
"generic::invalid_argument: udm validation failed: target field is not set"
UDM Validation
UDM Metadata Event Types have Validation requirements. These are mandatory fields that must be present in order to assign an Event Type.
The purpose of Validation is to ensure a base level of quality for UDM Event Types. For example, a USER_LOGIN
event without a user
is not useful.
Chronicle includes documentation on UDM Usage that provides a level of detail around Validation; however, as you’ll see in this post, it is not exhaustive.
Testing UDM Metadata Event Types
To test all 103 metadata.event_type
I used the following approach:
- a Colab notebook to create a minimal UDM event for each Metadata Event Type
- the Ingestion API to submit the above event
- verified against the the Chronicle UDM Usage guide where documentation was available
The test harness is included as the bottom of this document, and below follows the output of each test. Where available links to the UDM Usage documentation are included, but one of the observations of the tests is that not all UDM Event Validation requirements are documented, and in some cases appear to not match
Ultimately, below is a useful reference if you’re building a new Parser or UDM event via code to quickly double check the required fields.
⚠️ This is accurate at this moment in time, July 2023, and it may or will become out of date as things change, but you will have the process details below to quickly work out validation if not available in the UDM Usage guide.
ANALYST_ and DEVICE_
Notes:
- I could not find any examples of
ANALYST_
events, and so have omitted testing thisevent_type
DEVICE_
does not appear an active UDM event_type and failed when I tried to use it.
Event Types:
- EMAIL_TRANSACTION
- EMAIL_UNCATEGORIZED
- EMAIL_URL_CLICK [Deprecated]
Notes:
network.email
is NOT an Aliasing field
- while the UDM Usage guidance is to not populate users into Principal or Target, if you require Aliasing and Enrichment for Search or Detection you will need to index into these fields, i.e., use a Parser Extensionabout.file.hash
is NOT an Indexed field for File Hashes
- If you require Aliasing and Enrichment on Hashes, e.g., File Attachments, you will need to index into Principal or Target, i.e., Hash view won’t be populated from an attachment hash normalized toabout.file.hash
EMAIL_URL_CLICK
is marked as Deprecated
{
"metadata": {
"event_timestamp": "2023-07-06T06:26:10.448584Z",
"event_type": "EMAIL_TRANSACTION"
},
"principal": {
"ip": [
"1.2.3.4"
]
}
}
FILE
Event Types:
- FILE_CREATION
- FILE_DELETION
- FILE_MODIFICATION
- FILE_READ
- FILE_OPEN
- FILE_UNCATEGORIZED
Minimum UDM Object:
{
"metadata": {
"event_timestamp": "2023-07-03T12:14:38.164507Z",
"event_type": "FILE_COPY"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"file": {
"full_path": "foo.bar"
}
}
}
Event Types:
- FILE_MOVE
- FILE_SYNC
Minimum UDM Object:
{
"metadata": {
"event_timestamp": "2023-07-03T12:45:02.557878Z",
"event_type": "FILE_MOVE"
},
"src": {
"file": {
"full_path": "foo.bar"
}
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"file": {
"full_path": "foo.bar"
}
}
}
GENERIC_EVENT
Event Types:
- GENERIC_EVENT
Minimal UDM Object:
{
"metadata": {
"event_timestamp": "2023-07-03T12:52:50.379205Z",
"event_type": "GENERIC_EVENT"
}
}
GROUP
Event Types:
- GROUP_CREATION
- GROUP_DELETION
- GROUP_MODIFICATION
- GROUP_UNCATEGORIZED
Minimal UDM Object:
{
"metadata": {
"event_timestamp": "2023-07-03T13:01:10.957803Z",
"event_type": "GROUP_CREATION"
},
"principal": {
"user": {
"userid": "pinguino"
}
},
"target": {
"group": {
"group_display_name": "foobar_users"
}
}
}
MUTEX
Event Types:
- MUTEX_CREATION
- MUTEX_UNCATEGORIZED
Notes:
- You can only pass UDM Validation with a
process
object inprincipal
, you can’t pass UDM Validation with aprocess
intarget
resource
is marked as deprecated
{
"metadata": {
"event_timestamp": "2023-07-03T13:22:30.048989Z",
"event_type": "MUTEX_UNCATEGORIZED"
},
"principal": {
"hostname": "pinguino-01",
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"target": {
"resource": {
"name": "mutex",
"type": "MUTEX"
}
}
}
NETWORK
Event Types:
- NETWORK_CONNECTION
- NETWORK_FLOW
- NETWORK_FTP
- NETWORK_HTTP
- NETWORK_SMTP
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-03T14:06:30.781880Z",
"event_type": "NETWORK_CONNECTION"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"hostname": "jamon-01"
}
}
Event Types:
- NETWORK_DHCP
Notes:
network.dhcp
uses the field names as per the DHCP rfc — https://tools.ietf.org/html/rfc2131
- op = BOOTREQUEST, BOOTREPLY
- chaddr = client hardware address
- ciaddr = client IP address, only if client is BOUND, RENEW, REBINDING to ARP request
- yiaddr = your client IP address- Chronicle SIEM only evaluates BOOTREQUEST or BOOTREPLY for Asset aliasing
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-04T18:52:22.142364Z",
"event_type": "NETWORK_DHCP"
},
"principal": {
"ip": [
"1.2.3.4"
]
},
"network": {
"application_protocol": "DHCP",
"dhcp": {
"opcode": "BOOTREPLY"
}
}
}
Event Types:
- NETWORK_DNS
Notes
- The UDM fields for
network.dns
are based upon the DNS RFC Values — https://www.ietf.org/rfc/rfc1035.txt
- for QTypes the following is a hand reference:
| # | Type | Meaning |
|-----|--------|------------------------------------------------------|
| 1 | A | A host address |
| 2 | NS | An authoritative name server |
| 3 | MD | A mail destination (Obsolete - use MX) |
| 4 | MF | A mail forwarder (Obsolete - use MX) |
| 5 | CNAME | The canonical name for an alias |
| 6 | SOA | Marks the start of a zone of authority |
| 7 | MB | a mailbox domain name (EXPERIMENTAL) |
| 8 | MG | A mail group member (EXPERIMENTAL) |
| 9 | MR | A mail rename domain name (EXPERIMENTAL) |
| 10 | NULL | A null RR (EXPERIMENTAL) |
| 11 | WKS | A well known service description |
| 12 | PTR | A domain name pointer |
| 13 | HINFO | Host information |
| 14 | MINFO | Mailbox or mail list information |
| 15 | MX | Mail exchange |
| 16 | TXT | Text strings |
| 252 | AXFR | A request for a transfer of an entire zone |
| 253 | MAILB | A request for mailbox-related records (MB, MG or MR) |
| 254 | MAILA | A request for mail agent RRs (Obsolete - see MX) |
| 255 | * | A request for all records |
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-04T18:54:15.943301Z",
"event_type": "NETWORK_DNS"
},
"principal": {
"ip": [
"1.2.3.4"
]
},
"network": {
"application_protocol": "DNS",
"dns": {
"questions": {
"name": "www.acme.com"
}
}
}
}
PROCESS
Event Types:
- PROCESS_INJECTION
- PROCESS_LAUNCH
- PROCESS_OPEN
- PROCESS_TERMINATION
- PROCESS_UNCATEGORIZED
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-03T20:24:29.437724Z",
"event_type": "PROCESS_LAUNCH"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}
Event Types:
- PROCESS_MODULE_LOAD
Notes:
- The documentation omits that UDM Validation requires a
process
inprincipal
as well astarget
{
"metadata": {
"event_timestamp": "2023-07-03T20:40:29.854097Z",
"event_type": "PROCESS_MODULE_LOAD"
},
"principal": {
"hostname": "pinguino-01",
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}
Event Types:
- PROCESS_PRIVILEGE_ESCALATION
Notes:
- The documentation omits that a
user
andprocess
is required inprincipal
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-04T19:04:12.573338Z",
"event_type": "PROCESS_PRIVILEGE_ESCALATION"
},
"principal": {
"user": {
"userid": "pinguino"
},
"hostname": "pinguino-01",
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}
REGISTRY
Event Types:
- REGISTRY_CREATION
- REGISTRY_DELETION
- REGISTRY_MODIFICATION
- REGISTRY_UNCATEGORIZED
{
"metadata": {
"event_timestamp": "2023-07-04T19:12:19.489913Z",
"event_type": "REGISTRY_MODIFICATION"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"registry": {
"registry_key": "foo.bar"
}
}
}
RESOURCE
Event Types:
- RESOURCE_CREATION
- RESOURCE_DELETION
- RESOURCE_PERMISSIONS_CHANGE
- RESOURCE_READ
- RESOURCE_WRITTEN
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-04T20:19:53.394745Z",
"event_type": "RESOURCE_CREATION"
},
"principal": {
"user": {
"userid": "pinguino"
}
},
"target": {
"resource": {
"name": "foo.bar"
}
}
}
SCAN
Event Types:
- SCAN_FILE
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-06T06:57:08.443142Z",
"event_type": "SCAN_FILE"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"file": {
"full_path": "foo.bar"
}
}
}
Event Types:
- SCAN_HOST
- SCAN_NETWORK
- SCAN_UNCATEGORIZED
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-06T06:58:32.624239Z",
"event_type": "SCAN_HOST"
},
"principal": {
"hostname": "pinguino-01"
}
}
Event Types:
- SCAN_PROCESS
- SCAN_PROCESS_BEHAVIORS
Notes:
- SCAN_PROCESS_BEHAVIORS is deprecated and SCAN_PROCESS should be used instead.
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-06T06:59:17.690425Z",
"event_type": "SCAN_PROCESS"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}
Event Types:
- SCAN_VULN_HOST
- SCAN_VULN_NETWORK
{
"metadata": {
"event_timestamp": "2023-07-04T20:50:08.288914Z",
"event_type": "SCAN_VULN_NETWORK"
},
"principal": {
"hostname": "pinguino-01"
},
"target": {
"process": {
"file": {
"full_path": "foo.bar"
}
}
},
"extensions": {
"vulns": {
"vulnerabilities": {
"description": "Potentially Bad Traffic"
}
}
}
}
Event Types:
- SCHEDULED_TASK_CREATION
- SCHEDULED_TASK_DELETION
- SCHEDULED_TASK_DISABLE
- SCHEDULED_TASK_MODIFICATION
- SCHEDULED_TASK_UNCATEGORIZED
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-04T21:02:05.417757Z",
"event_type": "SCHEDULED_TASK_CREATION"
},
"principal": {
"hostname": "pinguino-01",
"user": {
"userid": "pinguino"
}
},
"target": {
"resource": {
"resource_type": "TASK"
}
}
}
SERVICE
Event Types:
- SERVICE_CREATION
- SERVICE_DELETION
- SERVICE_MODIFICATION
- SERVICE_START
- SERVICE_STOP
- SERVICE_UNSPECIFIED
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-05T14:11:06.526110Z",
"event_type": "SERVICE_UNSPECIFIED"
},
"principal": {
"hostname": "foo.bar"
},
"target": {
"user": {
"userid": "pinguino-01"
},
"process": {
"file": {
"full_path": "foo.bar"
}
}
}
}
SETTING
Event Types:
- SETTING_CREATION
- SETTING_DELETION
- SETTING_MODIFICATION
- SETTING_UNCATEGORIZED
Notes:
resource.type
is deprecated and resource_type should be used
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-05T14:15:27.070909Z",
"event_type": "SETTING_MODIFICATION"
},
"principal": {
"hostname": "foo.bar"
},
"target": {
"resource": {
"type": "SETTING",
"name": "foo.bar"
}
}
}
STATUS
Event Types:
- STATUS_HEARTBEAT
- STATUS_SHUTDOWN
- STATUS_STARTUP
- STATUS_UNCATEGORIZED
- STATUS_UPDATE
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-05T14:19:15.126817Z",
"event_type": "STATUS_HEARTBEAT"
},
"principal": {
"hostname": "foo.bar"
}
}
SYSTEM
Event Types:
- SYSTEM_AUDIT_LOG_WIPE
- SYSTEM_AUDIT_LOG_UNCATEGORIZED
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-05T16:09:29.647627Z",
"event_type": "SYSTEM_AUDIT_LOG_WIPE"
},
"principal": {
"hostname": "foo.bar",
"user": {
"userid": "pinguino-01"
}
}
}
USER
Event Types:
- USER_CHANGE_PASSWORD
- USER_CHANGE_PERMISSIONS
- USER_CHANGE_PASSWORD
- USER_CHANGE_PERMISSIONS
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-05T16:20:16.408189Z",
"event_type": "USER_CHANGE_PERMISSIONS"
},
"principal": {
"user": {
"userid": "pinguino-01"
}
},
"target": {
"user": {
"userid": "jamon"
}
}
}
Event Types:
- USER_COMMUNICATION
- USER_UNCATEGORIZED
{
"metadata": {
"event_timestamp": "2023-07-05T16:27:22.013276Z",
"event_type": "USER_COMMUNICATION"
},
"principal": {
"user": {
"userid": "pinguino-01"
}
}
}
Event Types:
- USER_BADGE_IN
{
"metadata": {
"event_timestamp": "2023-07-05T20:31:18.303584Z",
"event_type": "USER_BADGE_IN"
},
"target": {
"hostname": "jamon-01"
},
"extensions": {
"auth": {
"type": "MACHINE"
}
}
}
Event Types:
- USER_CREATION
- USER_DELETION
{
"metadata": {
"event_timestamp": "2023-07-05T20:44:10.642556Z",
"event_type": "USER_CREATION"
},
"principal": {
"hostname": "foo.bar",
"user": {
"userid": "pinguino"
}
},
"target": {
"user": {
"userid": "jamon"
}
}
}
Event Types:
- USER_LOGIN
- USER_LOGOUT
{
"metadata": {
"event_timestamp": "2023-07-05T20:48:50.657334Z",
"event_type": "USER_LOGIN"
},
"target": {
"user": {
"userid": "jamon"
}
},
"extensions": {
"auth": {
"type": "MACHINE"
}
}
}
Event Types:
- USER_RESOURCE_ACCESS
- USER_RESOURCE_CREATION
- USER_RESOURCE_DELETION
- USER_RESOURCE_UPDATE_CONTENT
- USER_RESOURCE_UPDATE_PERMISSIONS
- USER_STATS
Notes:
USER_STATS
is marked as deprecated.
Minimal UDM Event:
{
"metadata": {
"event_timestamp": "2023-07-06T07:16:41.350672Z",
"event_type": "USER_RESOURCE_ACCESS"
},
"principal": {
"user": {
"userid": "pinguino"
}
},
"target": {
"resource": {
"name": "foo.bar"
}
}
}
Test Harness Setup
🐉 I am not a developer, and the below Python scripts are not production usage code, and intended only for learning and educational purposes.
You will require a JSON Developer Service Account with Ingestion API IAM permissions.
- Configure input variables:
# Chronicle JSON Developer Service account with Ingestion API IAM permissions
# - specifically: i) Malachite Ingestion Collector, ii) Malachite Ingestion Integration User
CREDENTIALS_FILE='/content/ing-<tla>-<random>.json'
# Can be found in your UI under Settings
CUSTOMER_ID="<your customer guid>"
# Can be 'na', 'europe', or 'asia'.
REGION="us"
2. Run the following helper functions:
import datetime
import json
def now():
# Get the current time
current_time = datetime.datetime.now()
# Format the current time
formatted_time = current_time.strftime('%Y-%m-%dT%H:%M:%S.%fZ')
return formatted_time
def subtract_offset(timestamp, offset):
"""
Subtracts the offset from the timestamp and returns the resulting timestamp.
Args:
timestamp: The timestamp to subtract the offset from.
offset: The offset to subtract from the timestamp.
Returns:
The resulting timestamp.
"""
# Convert the timestamp to a datetime object.
datetime_object = datetime.datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%fZ")
# Convert the offset to a timedelta object.
timedelta_object = datetime.timedelta(days=-offset)
# Subtract the offset from the datetime object.
new_datetime_object = datetime_object - timedelta_object
# Convert the new datetime object to a timestamp.
#new_timestamp = new_datetime_object.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
new_timestamp = new_datetime_object.isoformat(timespec='milliseconds') + "Z"
return new_timestamp
def region(region):
REGIONS = {
"europe": "https://europe-malachiteingestion-pa.googleapis.com",
"asia": "https://asia-southeast1-malachiteingestion-pa.googleapis.com",
"us": "https://malachiteingestion-pa.googleapis.com",
}
if region not in REGIONS:
raise ValueError("Invalid region")
return REGIONS[region]
def auth(credentials_file):
from google.auth.transport import requests
from google.oauth2 import service_account
AUTHORIZATION_SCOPES = ["https://www.googleapis.com/auth/chronicle-backstory","https://www.googleapis.com/auth/malachite-ingestion"]
credentials = service_account.Credentials.from_service_account_file(
str(credentials_file),
scopes=AUTHORIZATION_SCOPES)
return requests.AuthorizedSession(credentials)
def create_udm_event(event):
import requests
authenticated = auth(CREDENTIALS_FILE)
data = {}
data["customer_id"] = CUSTOMER_ID
data["events"] = []
data["events"].append(json.loads(event))
json_data = json.dumps(data)
print ('Event to send:\n' + json_data)
http_endpoint = '{}/v2/udmevents:batchCreate'.format(region(REGION))
headers = {'content-type': 'application/json'}
r = authenticated.post(url=http_endpoint, data=json_data, headers=headers)
print(r.text)
return r
3. For each Metadata Event Type run the below test harness to find the minimal viable UDM Event that passes UDM Validation:
metadata_event_type="FILE_MOVE"
principal_hostname="pinguino-laptop"
src_file_full_path="bar.foo"
target_file_full_path="foo.bar"
event={
'metadata':{
'event_timestamp': now(),
'event_type': metadata_event_type,
},
'src': {
'file': {
'full_path': target_file_full_path
}
},
'principal':{
'hostname':principal_hostname
},
'target': {
'file': {
'full_path': target_file_full_path
}
}
}
print(json.dumps(event, indent=4))
create_udm_event(json.dumps(event))
If your minimal UDM Event is successful, you’ll receive an API response as follows:
<Response [200]>
If there is a UDM Validation error you will receive an error as follows:
"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}
Summary
UDM Validation is a core concept to understand, and impacts many dimensions of your using the Chronicle SecOps platform. Hopefully this post helps to explain what Validation is, and how you can validate Validation itself.
Please note, this isn’t an official Chronicle product documentation, but if you do find any issues or have comments, please let me know.