Ingest PAN-OS Alerts into Cortex XDR Pro Endpoint

Photo by callmefred.com

In this blog I’m sharing my experience leveraging Cortex XDR API’s to tailor-fit this product into specific customer requirements.

bet that if you’re a Systems Engineer operating in EMEA you’ve argued more than once about the way products and licenses are packaged from an opportunity size point of view (too big for your customer base). There are many reasons that explain why a given vendor would like to have a reduced set of SKUs. But, no doubt, the fewer SKUs you have the lower the chances are for a given product to fit exactly into the customer’s need from a sizing standpoint.

At Palo Alto Networks we’re no different, but we execute on two principles that help bridge the gaps:

  • Provide APIs for our products
  • Encourage the developer community to use them through our Developer Relations team

The use case

A Cortex XDR Pro Endpoint customer was interested in ingesting threat logs from their PAN-OS NGFW into his tenant to stitch them with the agent’s alerts and incidents. The customer was aware that this requirement could be achieved by sharing the NGFW state into the cloud-based Cortex Data Lake. But the corresponding SKU was overkill from a feature point of view (it provides not only alert stitching but ML baselining of traffic behaviour as well) and an unreachable cost from a budgeting perspective.

Cortex XDR Pro provides a REST API to ingest third-party alerts to cover this specific use case. It is rate limited to only 600 alerts per minute per tenant but was more than enough for my customer because they were only interested in medium to critical alerts that appeared at a much lower frequency in their network. What about leveraging this API to provide an exact match to my customer’s requirement?

On the other end, PAN-OS can be configured to forward these filtered alerts natively to any REST API with a very flexible payload templating feature. So at first it looked like we could “connect” the PAN-OS HTTP Log Forwarding feature with the Cortex XDR Insert Parsed Alert API. But a deep dive analysis revealed inconsistencies between the mandatory timestamp field format (it must be presented as UNIX milliseconds for XDR to accept it)

It was too close to give up. I just needed a transformation pipeline that could be used as a middleman between the PAN-OS HTTP log forwarding feature and the Cortex XDR Insert Parsed Alert API. Provided I was ready for a middleman, I’d like it to enforce the XDR API quota limitations (600 alerts per minute / up to 60 alerts per update)

Developers to the rescue

I engaged our Developer Relations team in Palo Alto Networks because they’re always eager to discuss new use cases for our APIs. A quick discussion ended up with the following architecture proposal:

  • An HTTP server would be implemented to be used as the endpoint for PAN-OS alert ingestion. Basic authentication would be implemented by providing a pre-shared key in the Authentication header.
  • A transformation pipeline would convert the PAN-OS payload into a ready-to-consume XDR parsed alert API payload. The pipeline would take care, as well, to enforce the XDR API quota limits buffering bursts as needed.
  • The XDR API client implementation would use an Advanced API Key for authentication.
  • Everything would be packaged in a Docker container to ease its deployment. Configuration parameters would be provided to the container as environment variables.

Implementation details ended up shared in a document available in cortex.pan.dev. If you’re in the mood of creating your own implementation I highly recommend you taking the time to go over the whole tutorial. If you just want to get to the point then you can use this ready-to-go container image.

Let’s get started

I opted for using the available container image. Let me guide you through my experience using it to fulfil my customer’s request.

First of all the application requires some configuration data that must be provided as a set of environmental variables. A few of them are mandatory, others are just optional with default values.

The following are the required variables (the application will refuse to start without them)

  • API_KEY: XDR API Key (Advanced)
  • API_KEY_ID: The XDR API Key identifier (its sequence number)
  • FQDN: Fully Qualified Domain Name of the corresponding XDR Instance (i.e. myxdr.xdr.us.paloaltonetworks.com)

The following are optional variables

  • PSK: the server will check the value in the Authorization header to accept the request (default to no authentication)
  • DEBUG: if it exists then the engine will be more verbose (defaults to false)
  • PORT: TCP port to bind the HTTP server to (defaults to 8080)
  • OFFSET: PAN-OS timestamp does not include time zone. By default, they will be considered in UTC (defaults to +0 hours)
  • QUOTA_SIZE: XDR ingestion alert quota (defaults to 600)
  • QUOTA_SECONDS: XDR ingestion alert quota refresh period (defaults to 60 seconds)
  • UPDATE_SIZE: XDR ingestion alert max number of alerts per update (defaults to 60)
  • BUFFER_SIZE: size of the pipe buffer (defaults to 6000 = 10 minutes)
  • T1: how often the pipe buffer polled for new alerts (defaults to 2 seconds)

For more details, you can check public repository documentation

Step 1: Generate an Advanced API key on Cortex XDR

Connect to your Cortex XDR instance and navigate to Setting > API Keys Generate an API Key of type Advanced granting the Administrator role to it (that role is required for Alert ingestion)

Step 2: Run the container image

Assuming you have Docker installed on your computer the following command line command would pull the image and run the micro-service with configuration options passed as environmental variables.

docker run -rm -p 8080:8080 -e PSK=hello -e FQDN=xxx.xdr.us.paloaltonetworks.com -e API_KEY=<my-api-key> -e API_KEY_ID=<my-key-id> -e DEBUG=yes ghcr.io/xhoms/xdrgateway

The Debug option provides more verbosity and it is recommended for initial experimentation. If everything goes as expected you’ll see a log message like the following one.

2021/03/17 21:32:25 starting http service on port 8080

Step 3: Perform a quick test to verify the micro-service is running

Use any API test tool (I like good-old Curl) to push a test payload.

curl -X POST -H "Authorization: hello" "http://127.0.0.1:8080/in" -d '{"src":"1.2.3.4","sport":1234,"dst":"4.3.2.1","dport": 4231,"time_generated":"2021/01/06 20:44:34","rule":"test_rule","serial":"9999","sender_sw_version":"10.0.4","subtype":"spyware","threat_name":"bad-bad-c2c","severity":"critical","action":"alert"}---annex---"Mozi Command and Control Traffic Detection"'

You should receive an empty status 200 response and see log messages in the server console like the following ones:

2021/03/17 21:52:16 api - successfully parsed alert
2021/03/17 21:52:17 xdrclient - successful call to insert_parsed_alerts

Step 4: Configure HTTP log Forward on the Firewall:

PAN-OS NGFW can forward alerts to HTTP/S endpoints. The feature configuration is available in Device > Server Profiles > HTTP and, for this case, you should use the following parameters

  • IP : IP address of the container host
  • Protocol : HTTP
  • Port: 8080(default)
  • HTTP Method POST

Some notes regarding the payload format

  • URI Format:“/in” is the endpoint where the micro-service listen for POST requests containing new alerts
  • Headers: Notice we’re setting the value hello in the Authorization header to match the -e PSK=hello configuration variable we passed to the micro-service
  • Payload: The variable $threat_name was introduced with PAN-OS 10.0. If you’re using older versions of PAN-OS the you can use the variable $threatid instead

The last action to complete this job is to create a new Log Forwarding profile that will consume our recently created HTTP server and to use it in all security rules we want their alerts being ingested into XDR. The configuration object is available at Objects > Log Forwarding

Step 5: Final checks

As your PAN-OS NGFW starts generating alerts, you should see activity in the micro-service’s output log and alerts being ingested into the Cortex XDR instance. The source tag Palo Alto Networks — PAN-OS clearly indicates these alerts were pushed by our deployment.

Production-ready version

There a couple of additional steps to perform before considering the micro-service production-ready.

  • The service should be exposed over a secure channel (TLS). Best option is to leverage the preferred forward-proxy environment available in our container environment. Notice the image honours the PORT env variable and should work out-of-the-box almost everywhere
  • A restart-policy should be in place to restart the container in case of a crash. It is not a good idea to run more than one instance in a load balancing group because the quota enforcement won’t be synchronised between them

Summary

Palo Alto Networks products and services are built with automation and customisation in mind. They feature rich API’s that can be used to tailor them to specific customer needs.

In my case these APIs provided the customer with a choice:

  • Either wait for the scope of the project (and its budget) to increase in order to accommodate additional products like Cortex Network Traffic Analysis or
  • Deploy a compact “middle-man” that would connect the PAN-OS and Cortex XDR API’s

This experience empowered me by acquiring DevOps knowledge (building micro-services) I’m sure I’ll use in many opportunities to come. Special thanks to The Developer Relations team at Palo Alto Networks who provided documentation and examples, and were eager to explore this new use case. I couldn’t have done it without their help and the work they’ve been putting into improving our developer experience.