Chaos Observability with Humio
New Support in the Chaos Toolkit Incubator for Surfacing Chaos events into Humio
This post is written by Russ Miles and was orginally posted Februray 21.
Building on yesterday’s article that showed how the Chaos Toolkit could now surface the commands and progress of chaos experiments into Slack, amongst other places, using the new notifications support, today it was time to take this support and integrate it into the cloud native observability solution, Humio.
Introducting the new incubator project: ChaosToolkit-Humio
Following our process for creating new integrations for the Chaos Toolkit, we’ve created a new Chaos Toolkit Incubator project for this extension called, naturally, chaostoolkit-humio.
When you add chaostoolkit-humio to your Chaos Toolkit python environment, the extension automatically listens to all the Chaos Toolkit notifications and sends them as events to Humio:
Now you can inspect current and past events in Humio and answer questions such as “What chaos was running on what systems at that point in time?”.
Next steps
Through the new Humio extension and integration, Chaos Toolkit chaos experiments can now be brought into your overall log aggregation and observability strategy.
Next, we plan to take the Humio integration further so that we can introduce probes for you to inspect, as part of your actual chaos experiments, the output that appears in Humio. When this is done, you’ll be able to incorporate those observations into your chaos experiments taking full advantage of the huge quantities of logs available in a typical Humio deployment, sourced from all over your production system.
Thanks to Martin W. Lassen.
