How to Integrate MISP and Chronicle SIEM
In this post, I will cover how to run a MISP (Malware Information Sharing Platform and Threat Sharing) TIP (Threat Intelligence Platform) instance on Google Cloud, optionally secure it with IAP (Identity Aware Proxy), and integrate it with Chronicle SIEM, all using Chronicle SIEM’s 3rd Party Ingestion Scripts, also known as Cloud Functions.
🐉 This is a community-driven integration with Chronicle SIEM, and is not an official integration. Please treat this as a best-effort release at this time.
Why integrate MISP with Chronicle SIEM?
Near real-time and historical Attribute matching
Integrating MISP Events into Chronicle SIEM enables near real-time detection of threats using Chronicle SIEM YARA-L Rules and Entity Graph. By mapping MISP Events to the Chronicle SIEM Entity Graph, you can craft YARA-L rules that perform entity-to-entity relationships. This enables you to display related attributes for a given detection.
The example Detection results from another MISP YARA-L rule, a MD5 Hash based rule, show not only the specific Attribute used to match the Detection, highlighted in yellow in the screenshot below, but also related Attributes for the given MISP event that have been ingested into Entity Graph.
The original MISP Attribute is also visible, and via the normalized MISP Event Attribute into UDM, you can pivot via URL back to the MISP Event itself.
📝Note, at the time of writing Alert Graph does not support showing Entity to Entity relations, so Alert View is more optimal for viewing Detection results.
Chronicle SIEM can be used to match historical MISP Attribute data of types Domain, Hostname, or IP using automated IOC matching.
For all types of historical MISP Attribute data Chronicle SIEM Reference Lists and YARA-L Retrohunts can be used, e.g., the below YARA-L rule can be run as a RetroHunt to match historical indicators:
rule misp_ioc_retrohunt_event_1604 {
meta:
events:
( $e.metadata.event_type = "PROCESS_LAUNCH" or
$e.metadata.event_type = "FILE_CREATION"
)
and
( strings.concat("file", strings.concat(",",$e.principal.process.file.md5 )) in %misp_event_1604 ) or
( strings.concat("file", strings.concat(",",$e.principal.process.file.sha1 )) in %misp_event_1604 ) or
( strings.concat("file", strings.concat(",",$e.principal.process.file.sha256 )) in %misp_event_1604 ) or
( strings.concat("file", strings.concat(",",$e.target.process.file.md5 )) in %misp_event_1604 ) or
( strings.concat("file", strings.concat(",",$e.target.process.file.sha1 )) in %misp_event_1604 ) or
( strings.concat("file", strings.concat(",",$e.target.process.file.sha256 )) in %misp_event_1604 )
outcome:
$risk_score = 100
condition:
$e
}Chronicle SIEM Reference Lists are single-dimensional, but you can import “CSV” data and wrangle the required UDM fields to support multiple types of MISP attributes within a single Reference List, within the default limit of 7 Reference List calls per Detection Rule.
This is useful when you want to keep the indicators specific to a given campaign, i.e., a specific MISP Event.
📝 Chronicle Entity Graph loads data in batches every six hours, so it is not strictly real-time. IOC and Attribute matching is generally a historical activity, and it is important to be aware ingested MISP events into Entity Graph will not be immediately usable in Detection Engine until the next batch load runs.
A broader range of Attributes matching
Chronicle SIEM can automatically match IOCs based on IP or Domain, but by integrating MISP, a wider range of Attribute types can be used for Detection, including multi-value Entity Graph entries.
As of this writing, the following indicator types are supported (and more to be added) by a custom MISP_IOC parser:
ip-src
ip-dst
url
domain
hostname
sha1
sha256
md5
filename
regkey
whois-registrant-email
user-agent
chrome-extension-id
vulnerabilityAccess to Public & Private IOC Feeds
Chronicle SIEM includes a wide range of built-in threat feeds, but organizations that want to run their own threat intelligence platform or already run MISP can integrate them with Chronicle SIEM to leverage their intelligence capabilities.
MISP Primer
What is MISP?
MISP is an open source Threat Intelligence Platform that allows users to share Indicators of Compromise (IOCs) both publicly and privately. It is funded by the European Union and the Computer Incident Response Center Luxembourg.
Key MISP concept to be familiar with include:
- Attribute: Indicators used to detect suspicious or malicious cyber activity. For a full list of Attributes see the MISP API specifications
- Event: Contextually related information represented as Attributes and Object
- Object: Advanced combinations of related Attributes.
- Feed: A source of correlations for all of your Events and Attributes without the need to import them directly into your MISP system
- Tag: Allows an arbitrary link between Events to be created
- TLPs: the Traffic Light Protocol, provides a simple and intuitive schema for indicating when and how sensitive information can be shared, facilitating more frequent and effective collaboration
Source: https://misp.gitbooks.io/misp-book/content/GLOSSARY.html
Installing MISP on GCP
MISP can be run on a variety of platforms, including Google Cloud. If you are already running MISP in another environment, you can still use the Chronicle Cloud Function by refactoring it or by providing an export from the MISP REST API using another mechanism.
Prerequisites
- Chronicle SIEM Ingestion API JSON Developer Account
- if you do not have a copy of your Ingestion API credentials contact your Chronicle Partner or Chronicle Support - Access to Google Cloud Platform with permissions to create a dedicated Project
- Patience
- this is a first release of this integration and while all best efforts have been made to document the setup steps, there may be some customization for environment specific configurations required
Virtual Machine Sizing
The first step is to determine the Virtual Machine sizing requirements. As per the MISP recommendations, “starting with a 2+ core VM and 8 to 16 GB of memory should be plenty”.
Creating the MISP VM
The next step is to create a Virtual Machine. MISP supports the following Linux distributions:
- Ubuntu
- RHEL
- Debian
I created a GCE (Google Compute Engine) VM without a public IP address and used IAP to provide secured remote access. IAP is a more secure way to access GCE VMs than using a public IP address. You can also create a GCE VM with a public IP address, but be sure to apply an appropriate Firewall policy to restrict access as required.
Installing MISP
Once the VM is up and running, and you have SSH’d in, I recommend using the INSTALL-misp.sh script, which is a one-click installation script that makes it easy to get MISP up and running.
# Please check the installer options first to make the best choice for your install
wget --no-cache -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh
bash /tmp/INSTALL.sh
# This will install MISP Core
wget --no-cache -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh
bash /tmp/INSTALL.sh -cCongratulations! Your MISP instance is now installed. It is recommended that you review the hardening steps in the installation guide to improve the security of your MISP instance.
Configure MISP
To configure your MISP instance follow the MISP gitbooks quickstart guide:
A convenient way to add all publicly available MISP Feeds is to use the below JSON Feed file from the MISP GitHub repo.
To use the Chronicle MISP Cloud Function, you will need to create an API user. This can be done under Administration > Add User. The API key is a sensitive piece of information, so be sure to safely and securely store it.
Setup Identity Aware Proxy (IAP)
IAP is used to publish access to the MISP web interface in a secure manner.
This is an optional step, but a highly recommended one over exposing a VM with a Public IP address directly.
⚠️ Please note, this is not a detailed step by step guide within this post, and familiarity with IAP is recommended.
- Setup IAP within your GCP Project, or Shared VPC
2. Following the below instructions, skip step 1 (Create a Compute Engine template) as you already have a GCE VM, and follow the instructions from step 2 onward.
https://cloud.google.com/iap/docs/tutorial-gce
3. When you have a GCP Load Balancer with a TLS Certificate and Domain name, re-visit your MISP Server Settings & Maintence to configure the MISP.external_baseurl and MISP.baseurl
Chronicle MISP Cloud Function Setup
Setup the MISP Cloud Function Service Account
To run the MISP Cloud Function, you must create a dedicated service account and grant it the required permissions to access the credentials stored in Secret Manager.
- Create a dedicated service account to run the MISP Cloud Function:
SA_NAME="sa-misp-to-chronicle"
SA_DESCRIPTION="MISP to Chronicle SIEM SA"
SA_DISPLAY_NAME="MISP Service Account"
gcloud iam service-accounts create $SA_NAME --description="$SA_DESCRIPTION" --display-name="$SA_DISPLAY_NAME"2. To run the Cloud Function, you must grant the required IAM role to your MISP service account.:
# Grant Cloud Function Invoker IAM Role
PROJECT_ID="<your project id>"
gcloud projects add-iam-policy-binding $PROJECT_ID--member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --role='roles/cloudfunctions.developer'3. To create a Secret to hold the Chronicle SIEM Ingestion API credentials, follow these steps:
# Create a Secret for the Chronicle Ingestion API
SECRET_ID="chronicle_service_account_ingestion"
CRED_PATH="/home/your_username/creds.json"
gcloud secrets create $SECRET_ID --replication-policy="automatic"
gcloud secrets versions add projects/$PROJECT_ID/secrets/$SECRET_ID --data-file=$CRED_PATH
rm $CRED_PATH
gcloud secrets add-iam-policy-binding $SECRET_ID --member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --role='roles/secretmanager.secretAccessor'4. To create a Secret to hold the MISP API credentials, follow these steps:
# Create a Secret for the MISP API
SECRET_ID="misp_api"
CRED_PATH="/home/your_username/creds.json"
gcloud secrets create $SECRET_ID --replication-policy="automatic"
gcloud secrets versions add projects/$PROJECT_ID/secrets/$SECRET_ID --data-file=$CRED_PATH
rm $CRED_PATH
gcloud secrets add-iam-policy-binding $SECRET_ID --member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --role='roles/secretmanager.secretAccessor'Create a Serverless VPC
The MISP Cloud Function requires a Serverless VPC to be created in order to access your MISP VM. A Serverless VPC is a network that is isolated from the public internet. This makes it more secure for accessing your MISP VM.
gcloud services enable vpcaccess.googleapis.com
gcloud compute networks vpc-access connectors create CONNECTOR_NAME \
--region REGION \
--subnet SUBNET \
# If you are not using Shared VPC, omit the following line.
--subnet-project HOST_PROJECT_ID \
# Optional: specify minimum and maximum instance values between 2 and
# 10, default is 2 min, 10 max.
--min-instances MIN \
--max-instances MAX \
# Optional: specify machine type, default is e2-micro
--machine-type MACHINE_TYPEAlternatively you can configure the Serverless VPC via the GCP Console GUI.
Deploy the Chronicle MISP Cloud Function
Chronicle provides a set of cloud functions that can be used to ingest data from third-party APIs into Chronicle SIEM. These cloud functions are serverless, which means that they do not require any infrastructure to be provisioned or maintained. This makes them a cost-effective and scalable way to ingest data into Chronicle SIEM.
To integrate MISP into Chronicle SIEM you need to deploy the MISP Cloud Function.
The instructions for deploying the MISP cloud function are included in the Git repo above.
I recommend running the following commands from Google Cloud Shell to make the MISP cloud function deployment process easier.
- Clone the Chronicle Cloud Functions Git repo:
git clone https://github.com/chronicle/ingestion-scripts.git
cd ingestion-scripts
cd misp
cp -r ../common/ .2. Configure the MISP Cloud Function environment variables YML file:
nano .env.yml
# this can be found in the Chronicle UI under Settings > Profile
CHRONICLE_CUSTOMER_ID: <your Chronicle customer GUID>
# the region as in the URL of your Chronicle UI
CHRONICLE_REGION: <your Chronicle region, e.g., europe>
CHRONICLE_SERVICE_ACCOUNT: projects/<project>/secrets/<secret_name>/versions/1
CHRONICLE_NAMESPACE: <namespace>
API_KEY: projects/<project>/secrets/<secret_name>/versions/1
TARGET_SERVER: <your MISP IP address>
ORG_NAME: <your MISP Org Name>3. Before deploying the MISP Cloud Function, make the following changes to the main.py file:
- If you’re not using a signed SSL certificate then add
verify=Falsetomain.py
> req = requests.post(url, json=params, headers=headers, verify=False)
---
< req = requests.post(url, json=params, headers=headers)- Chronicle SIEM has a limit of 1 Megabyte (MB) per log message, but a MISP event can be several MB in size. To work around this, the following Python code snippet separates each attribute in an event into a single, smaller log message.
< # Iterate through all the events and ingest data into Chronicle.
< for data in response_events.get("response", []):
< event_json = data.get("Event", {})
<
< # Remove unwanted key-value and append the
< # updated dictionary to data_list.
< updated_dict = {
< key: event_json.get(key)
< for key in event_json
< if key not in KEYS_TO_REMOVE
< }
< data_list.append(updated_dict)
---
> dict_example = {}
>
> for response in response_events["response"]:
> for key, value in response.items():
> for key2, value2 in value.items():
> if type(value2) == type(str()):
> dict_example[key2] = value2
> if key2 == "Org":
> dict_example[key2] = value2
> if key2 == "Orgc":
> dict_example[key2] = value2
> if key2 == "Attribute":
> for idx, attr in enumerate(value2):
> new_dict = {}
> new_dict = dict_example.copy()
> new_dict["Attribute"] = attr
> new_dict_json = json.dumps(new_dict)
> data_list.append(new_dict)
> if key2 == "Object":
> for i in range(len(value2)):
> for key3, value3 in value2[i].items():
> if type(value3) == type(str()):
> dict_example[key3] = value3
> if key3 == "Org":
> dict_example[key3] = value3
> if key3 == "Orgc":
> dict_example[key3] = value3
> if key3 == "Attribute":
> for idx, attr in enumerate(value3):
> new_dict = {}
> new_dict = dict_example.copy()
> new_dict["Attribute"] = attr
> data_list.append(new_dict)
4. Deploy the MISP Cloud Function with at least 1 vCPU and 4GB of RAM. The default shared core and 256 MB of RAM is not sufficient to run the Cloud Function.
VPC_CONNECTOR="p-connector-shared-euw3"
FUNCTION_NAME="cf-misp-to-chronicle"
gcloud functions deploy $FUNCTION_NAME -entry-point main -trigger-http -runtime python39 -env-vars-file .env.yml -impersonate-service-account $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com -region="europe-west4" - service-account="$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" -gen2 -vpc-connector="$VPC_CONNECTOR" -egress-settings all --memory="4096MB"Congratulations! You’ve deployed the Chronicle MISP Cloud Function.
Scheduling the MISP Cloud Function
The MISP Cloud Function requires a Cloud Scheduler task in order to run at regular intervals. The Cloud Schedule Cron Job should match the POLL_INTERVAL as configured in your Cloud Function, e.g., 5 minutes by default.
CLOUD_SCHEDULER_JOB_LOCATION="europe-west1"
JOB_NAME="misp-to-chronicle"
gcloud scheduler jobs create http $JOB_NAME --location $CLOUD_SCHEDULER_JOB_LOCATION --schedule "*/5 * * * *" --time-zone "Etc/UTC" --uri "https://$FUNCTION_LOCATION-$PROJECT_ID.cloudfunctions.net/$FUNCTION_NAME" --http-method POST --message-body "}" --headers="Content-Type=application/json" --oidc-service-account-email $SA_NAME@$PROJECT_ID.iam.gserviceaccount.comChronicle MISP_IOC Parser
There is no default parser associate with the Chronicle MISP_IOC label at the time of writing but I will be proposing the below become a default; however, until such time you will need to submit the below parser via the Chronicle CBN tool, the Chronicle CLI, or the UI Parser Management preview.
Customize the Parser to match your environment
Before deploying the parser, review the initial section of the parser and update to match your environment, specifically configure:
- the base MISP URL to match your instance
- strip protocol from URLs
- this enables you to match URL attributes to the format of your log data, i.e., either with or without the HTTP(s) protocol prefix
filter {
# Set to match your MISP (external) base URL
mutate {
replace => {
# set to your external MISP URL
"misp_url" => "misp.your.domain..com"
# if your log sources don't include the HTTP(S) protocol set to true
"strip_protocol_from_urls" => "true"
}
}
# Initialize sentinel fields required for logic conditions
mutate {
replace => {
"Orgc.name" => ""
"Attribute.first_seen" => ""
"Attribute.last_seen" => ""
"Attribute.uuid" => ""
"Attribute.category" => ""
"Attribute.comment" => ""
"Attribute.timestamp" => ""
"Attribute.type" => ""
"id" => ""
"orgc_id" => ""
"org_id" => ""
"date" => ""
"threat_level_id" => ""
"info" => ""
"published" => ""
"uuid" => ""
"attribute_count" => ""
"analysis" => ""
"timestamp" => ""
"distribution" => ""
"proposal_email_lock" => ""
"locked" => ""
"publish_timestamp" => ""
"sharing_group_id" => ""
"disable_correlation" => ""
"extends_uuid" => ""
"protected" => ""
}
}
json {
source => "message"
array_function => "split_columns"
on_error => "_not_json"
}
if [_not_json] {
drop { tag => "TAG_MALFORMED_MESSAGE" }
} else {
mutate {
replace => {
"event.idm.entity.metadata.product_name" => "MISP Threat Sharing"
"event.idm.entity.metadata.vendor_name" => "misp-project.org"
}
}
# Unique MISP Event ID
if [uuid] != "" {
mutate {
replace => {
"event.idm.entity.metadata.product_entity_id" => "%{uuid}"
}
}
}
# Summary of the MISP Event
if [info] != "" {
mutate {
replace => {
"event.idm.entity.metadata.description" => "%{info}"
}
}
}
if [id] != "" {
mutate {
replace => {
"threat_det.url_back_to_product" => "https://%{misp_url}/events/view/%{id}"
}
}
}
# Provide summary of how many indicators were part of the MISP event
if [attribute_count] != "" {
mutate {
convert => {
"attribute_count" => "string"
}
on_error => "_catch_ex"
}
mutate {
replace => {
"_attribute_count.key" => "attribute_count"
"_attribute_count.value" => "%{attribute_count}"
}
}
mutate {
merge => {
"event.idm.entity.metadata.source_labels" => "_attribute_count"
}
}
}
# context on the event's distribution status
if [distribution] != "" {
# 0 - Your organization only
# 1 - This community only
# 2 - Connected communities
# 3 - All communities
# 4 - Sharing group
# 5 - Inherit Event
mutate {
convert => {
"distribution" => "string"
}
on_error => "_catch_ex"
}
mutate {
gsub => [
"distribution","0","0 - Your organization only"
]
}
mutate {
gsub => [
"distribution","1","1 - This community only"
]
}
mutate {
gsub => [
"distribution","2","2 - Connected communities"
]
}
mutate {
gsub => [
"distribution","3","3 - All communities"
]
}
mutate {
gsub => [
"distribution","4","4 - Sharing group"
]
}
mutate {
gsub => [
"distribution","4","5 - Inherit Event"
]
}
mutate {
replace => {
"_distribution.key" => "distribution_string"
"_distribution.value" => "%{distribution}"
}
}
mutate {
merge => {
"event.idm.entity.metadata.source_labels" => "_distribution"
}
}
}
# the organization that created the event
if [Orgc][name] != "" {
mutate {
replace => {
"threat_det.threat_feed_name" => "%{Orgc.name}"
}
}
}
if [threat_level_id] != "" {
# 1 - High
# 2 - Medium
# 3 - Low
# 4 - Undefined
mutate {
replace => {
"threat_det.severity_details" => "%{threat_level_id}"
}
}
if [threat_level_id] == "1" {
mutate {
replace => {
"threat_det.severity" => "HIGH"
}
}
} else if [threat_level_id] == "2" {
mutate {
replace => {
"threat_det.severity" => "MEDIUM"
}
}
} else if [threat_level_id] == "3" {
mutate {
replace => {
"threat_det.severity" => "LOW"
}
}
} else {
mutate {
replace => {
"threat_det.severity" => "UNKNOWN_SEVERITY"
}
}
}
}
if [Attribute][uuid] != "" {
mutate {
replace => {
"threat_det.threat_id" => "%{Attribute.uuid}"
}
}
}
if [Attribute][category] != "" {
mutate {
merge => {
"threat_det.category_details" => "Attribute.category"
}
}
}
# >>> Tag Extraction
for index, tag in Attribute.Tag {
mutate {
merge => {
"threat_det.category_details" => "tag.name"
}
}
}
# <<< Tag Extraction
# Final Threat Detection Merge
# - put any Threat Detect fields before this merge
mutate {
merge => {
"event.idm.entity.metadata.threat" => "threat_det"
}
}
# store the specific Indicator type, e.g., sha256, whois-registrant
if [Attribute][type] != "" {
mutate {
replace => {
"threat_det.description" => "%{Attribute.type}"
}
}
}
if [Attribute][comment] != "" {
mutate {
replace => {
"threat_det.summary" => "%{Attribute.comment}"
}
}
}
# Validation requirement
# - at least one value must be poopulated into inteval time
# - start with the timestamp of the indicator, and override with first and last seen if available
if [Attribute][timestamp] != "" {
date {
match => ["Attribute.timestamp", "UNIX"]
target => "event.idm.entity.metadata.interval.start_time"
}
}
if [Attribute][first_seen] != "" {
# Date plugin doesn't support six fractions of a second granularity
# - the below regex extracts two groups of 23 and 6 chars to reduce this to three fractional seconds
mutate {
gsub => [
# 2022-09-23T00:00:00.000000+00:00
"Attribute.first_seen", "(.......................)...(......)", "$1$2"
]
}
date {
match => ["Attribute.first_seen", "yyyy-MM-ddTHH:mm:ss.SSSZZ"]
target => "event.idm.entity.metadata.interval.start_time"
}
}
if [Attribute][last_seen] != "" {
mutate {
gsub => [
"Attribute.last_seen", "(.......................)...(......)", "$1$2"
]
}
date {
match => ["Attribute.last_seen", "yyyy-MM-ddTHH:mm:ss.SSSZZ"]
target => "event.idm.entity.metadata.interval.end_time"
}
}
# >>>>>>>>>>>>>>>>>>>>>>>>
# >>> Start IOC extraction
# >>>>>>>>>>>>>>>>>>>>>>>>
if [Attribute][type] == "chrome-extension-id" and [Attribute][value] != "" {
mutate {
convert => {
"Attribute.value" => "string"
}
on_error => "_catch_ex"
}
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "RESOURCE"
"event.idm.entity.entity.resource.name" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "ip-src" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "IP_ADDRESS"
}
merge => {
"event.idm.entity.entity.ip" => "Attribute.value"
}
}
}
if [Attribute][type] == "ip-dst" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "IP_ADDRESS"
}
merge => {
"event.idm.entity.entity.ip" => "Attribute.value"
}
}
}
if [Attribute][type] == "ip-dst|port" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "IP_ADDRESS"
"event.idm.entity.entity.port" => "%{column2}"
}
convert => {
"event.idm.entity.entity.port" => "integer"
}
merge => {
"event.idm.entity.entity.ip" => "column1"
}
}
}
}
if [Attribute][type] == "ip-src|port" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "IP_ADDRESS"
"event.idm.entity.entity.port" => "%{column2}"
}
convert => {
"event.idm.entity.entity.port" => "integer"
}
merge => {
"event.idm.entity.entity.ip" => "column1"
}
}
}
}
if [Attribute][type] == "domain|ip" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "DOMAIN_NAME"
"event.idm.entity.entity.hostname" => "%{column2}"
}
convert => {
"event.idm.entity.entity.port" => "integer"
}
merge => {
"event.idm.entity.entity.ip" => "column1"
}
}
}
}
if [Attribute][type] == "domain" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "DOMAIN_NAME"
"event.idm.entity.entity.hostname" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "hostname" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "DOMAIN_NAME"
"event.idm.entity.entity.hostname" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "user-agent" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "RESOURCE"
"event.idm.entity.entity.network.http.user_agent" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "url" and [Attribute][value] != "" {
if [strip_protocol_from_urls] == "true" {
mutate {
gsub => [
"Attribute.value", "http(?:s)?://(.*)","$1"
]
}
}
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "URL"
"event.idm.entity.entity.url" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "filename|md5" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.md5" => "%{column2}"
"event.idm.entity.entity.file.full_path" => "%{column1}"
}
}
}
}
if [Attribute][type] == "filename|sha1" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.sha1" => "%{column2}"
"event.idm.entity.entity.file.full_path" => "%{column1}"
}
}
}
}
if [Attribute][type] == "filename|sha256" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.sha256" => "%{column2}"
"event.idm.entity.entity.file.full_path" => "%{column1}"
}
}
}
}
if [Attribute][type] == "sha256" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.sha256" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "sha1" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.sha1" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "md5" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.md5" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "filename" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "FILE"
"event.idm.entity.entity.file.full_path" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "regkey" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "RESOURCE"
"event.idm.entity.entity.registry.registry_key" => "%{Attribute.value}"
}
}
}
if [Attribute][type] == "regkey|value" and [Attribute][value] != "" {
csv {
source => "Attribute.value"
separator => "|"
on_error => "_ex_csv"
}
if ![_ex_csv] {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "RESOURCE"
"event.idm.entity.entity.registry.registry_key" => "%{column1}"
"event.idm.entity.entity.registry.registry_value_name" => "%{column2}"
}
}
}
}
if [Attribute][type] == "email" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "USER"
}
merge => {
"event.idm.entity.entity.user.email_addresses" => "Attribute.value"
}
}
}
if [Attribute][type] == "email-src" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "USER"
}
merge => {
"event.idm.entity.entity.user.email_addresses" => "Attribute.value"
}
}
}
if [Attribute][type] == "email-dst" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "USER"
}
merge => {
"event.idm.entity.entity.user.email_addresses" => "Attribute.value"
}
}
}
if [Attribute][type] == "whois-registrant-email" and [Attribute][value] != "" {
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "USER"
}
merge => {
"event.idm.entity.entity.user.email_addresses" => "Attribute.value"
}
}
}
if [Attribute][type] == "vulnerability" and [Attribute][value] != "" {
# the extensions UDM object is not extended into Entity model
mutate {
replace => {
"event.idm.entity.metadata.entity_type" => "RESOURCE"
"event.idm.entity.entity.resource.name" => "%{Attribute.value}"
}
}
}
# <<< End IOC extraction
if [Attribute][type] =~ /ip-src|ip-dst|url|domain|hostname|sha1|sha256|md5|filename|regkey|whois-registrant-email|user-agent|chrome-extension-id|vulnerability/ {
mutate {
merge => {
"@output" => "event"
}
}
}
}
}Summary
I will post more examples of using the MISP integration in Chronicle SIEM in the future, such as more YARA-L rule examples. For now, I hope this gives you an idea of how you can integrate MISP TIP with the Chronicle SIEM platform and leverage Chronicle’s strengths for IOC matching. I will also focus in a future post on how to use Chronicle Reference Lists for historical IOC matching.
