Stackdriver Push to Splunk

During my career (in technology), I have dealt with many clients to whom security was one of the main areas of concern.

As such, there’s always room for improvement but without a shed of a doubt, communications direction and stateful firewalls are some of the very first elements to consider.

When it comes to logging and audit information, as a rule of thumb, it’s good to have a log aggregator stored outside of the scope of a cloud provider. A great log correlation out there is Splunk. It has all of the bells and whistles needed to have a cutting edge AI powered rule engine. It allows you to have self healing as well as event correlation.

On the one hand there’s the system logs, on the other the hypervisor logs. When it comes to cloud environments, each player has its custom solutions. In the case of Google Cloud Platform (gcp), the name of the game is Stackdriver.

Stackdriver stores and ships logs from/to different platforms. In order to achieve push communication from gcp (specifically from Stackdriver) onto Splunk, the initiator must be Stackdriver.

Stackdriver has a few alternatives to export logs to different platforms, amongst them, the one that suffices all of the requirements to connect to Splunk on a push basis is PubSub.

PubSub is a managed service providing message queue communications. This means that, each time a message arrives, an event gets triggered. Such event queues the message onto Splunk.

Desired event capture and export to Splunk

Following the architecture diagram, we must configure PubSub to reach the Splunk service.

Disclamer:

• PubSub only sends messages to pre-validated domains on gcp.
• PubSub’s authentication mechanism is predefined and cannot be modified at runtime.
• PubSub’s url endpoint must support SSL with a valid certificate issued by a predefined third party certificate authority.

As such, there are a couple of modifications that need to happen in Splunk (cloud or otherwise) in order for us to make the pieces work together.

For starters, we need a custom domain name that can be validated in gcp. Usually you could just use CloudDNS to achieve this task.

Secondly, we need to generate a way to validate Splunk’s HEC via url request parameter (remember, we cannot modify PubSub’s auth mechanism).

Finally, we need to have a valid SSL terminator on Splunk validated by a third party (for this part, we recommend to just use Let’s Encrypt which is a valid and trusted certificate authority for PubSub).

On the Splunk side:

• Generate an SSL certificate (using Let’s Encrypt) and add it as a frontend for Splunk (including HEC requests).

Configure Splunk instance to use Lets Encrypt certificate chain

• Install Base64 binary from Splunkbase.

Upload the downloaded base64 app to the Splunk instance.

• Configure HEC so that the token can be passed as an argument as part of the URL (Allow Query String).

Create an HEC to feed data onto the Splunk instance

• Allow HEC over SSL.

Allowing HEC over SSL by enabling all tokens from the General section

• Finally specify SSL certificates to be used by HEC (priv_key should include the full chain with the encrypted private key)

server.conf with SSL using Let’s Encrypt certificate chain

On the gcp Side:

• Validate the Splunk domain on gcp.

• Build a PubSub topic with the correct format containing the Splunk HEC endpoint (the enpoind must be in the format https://SPLUNK_URL:8088/services/collector/raw?token=TOKEN_ID).

PubSub to send messages using a Push subscription to Splunk

• Generate a sink for Stackdriver to push logs onto PubSub.

Create a StackDriver log sink and send messages to the previously created PubSub topic

Once every piece is in place and talking to each other, this is how a messge sent from PubSub looks from the Splunk perspective:

Raw message from PubSub

Or its equivalent via the WebUI:

Splunk showcasing a message received via PubSub

Note that the message is by default base64 encoded by PubSub. By using the following query the message can be decoded at runtime within Splunk.

message.data="*" | fields message.data,subscription,message.messageId,message.publishTime | rex field=message.data "(?<data>.*)" | base64 field=data mode=append action=decode | table message.messageId, message.publishTime, subscription, base64

A graphical representation of a sample result via Splunk WebUI:

Splunk using the base64 module to decode the contents provided by PubSub

Conclusion:

It has been proven that there is at least one method to connect Stackdriver to Splunk via push without the intervention of third party or even custom or homebrew code.

I understand that this method lacks message acknowledgement and traffic throttle when large bursts of events occur.

There is plenty of room for improvement in order to build a solution from the current proposal.

It is my understanding that whichever the official proposal for message push onto Splunk will be, it should come in the form of a cloud native initiative. Whenever that solution get issued I’ll be ready to try it out. In the meantime this suffices.