Google Cloud log export to Splunk
So you are developing a on the Google Cloud. Your gracious infrastructure host has kindly provided you with an existing, perfectly good log collection and analysis tool, called Stackdriver. Everything is sorted, all is good. Except one day you are told to send your logs off to some far away place, running a strange and mysterious logging system.
If that system happens to be Splunk, I have some good news for you:
During my internship at Telenor Digital this summer, I’ve been developing a prototype for a bridge between these two log collecting titans. The experienced Splunk admin will now frown and exclaim “You didn’t make that GCP Splunk addon!”, and he would be right, because I didn’t. But I used it to make something useful for the cloud.
You see, an existing Splunk setup prefers to receive logs from new Splunk instances, so called forwarders. The forwarders are connected to the source of the logs, and then pass them on to some Splunk Indexer. If you pay Google good money to host your project on the cwo log collecting titans. The expperienced Splunk admin will now frown and exclaim “You didn’t make that GCP Splunk addon!”, and he wouldloud, you’ll feel pretty bummed out about having some server box lying around in your server-room/dank basement, just so you can run said forwarder. This is why the sensible thing is to run the forwarder in the cloud as well, and why I hereby introduce the GCP-Splunk-bridge.
The GCP-Splunk bridge works by using the Google Pub/Sub system to transport logs from Stackdriver Logging to a virtualized forwarder, which in turn connects to your Splunk cluster of choice. Deploying this thing manually would be a giant pain, so everything is packaged “neatly” using Docker, Terraform and Helm.
Why is there a Bucket?
The bucket stores secrets, certificates and terraform remote states, keeping them out of your repository.
What is that NAT-Gateway doing there?
The people managing your firewall are not to fond of your cloud-based ephemeral IPs. The Gateway ensures that all egress traffic has a whitelist-able static IP.
Are you really using the pre-made Splunk Docker image?
I’m enhancing it.
In theory at least, this prototype should scale well. The potential bottlenecks (Pub/Sub pipe, forwarder instance, and NAT-Gateway) can be scaled horizontally to the point where they might support a good amount of GB/s, enough for most ordinary cases. Doing so will require some tinkering around with the config files, but is probably a cleaner solution than setting up new forwarder boxes in the attic.
Do keep in mind though: I’m an intern, I don’t really know what I’m doing, so you should probably look a bit more closely at this thing before deploying it and forgetting about it.