“Chaos Should Never Be A Surprise…”

The Chaos Toolkit now includes support for runtime notifications, and Slack Integration

Russ Miles
Chaos Toolkit
2 min readFeb 20, 2018

--

One of the rules of chaos engineering is that “chaos should never be a surprise”.

When you’re executing a chaos experiment everyone should know about it. Now, thanks to a great suggestion from Simon Johansson in the community, the Chaos Toolkit includes runtime notifications that make it simple to listen to the executing of an experiment and surface that progress in lots of novel ways.

Communicating Chaos Experiment Progress with Runtime Hooks

The Chaos Toolkit now triggers a set of notifications that you can listen to during the various runtime operations of the toolkit, including:

  • Validate Flow Events: Surfaces that experiment validation has started, failed or has completed.
  • Discovery Flow Events: Surfaces that system discovery has started, failed or has completed.
  • Init Flow Events: Surfaces that experiment initialisation, or creation, has been started, failed of has completed.
  • Run Flow Events: Surfaces that an experiment has been started, failed or has completed.

Each set of notifications expose details of a particular point in the lifecycle of a Chaos Toolkit operation and can be used to then surface that information…

An Example in the Incubator: Slack Integration

At that same time as adding this new support for runtime events we also added an example of their usage.

There is now Slack integration in the Chaos Toolkit Incubator that uses these runtime events to surface Chaos Toolkit operation execution into a specific Slack channel.

Surfacing a Chaos Experiment execution into Slack

Summary

Runtime chaos experiment hooks provide a lot of exciting opportunities for communicating the status and progress of your Chaos Toolkit operations, in particular the status of running chaos experiments.

Look out for more examples that take advantage of this new feature over the coming weeks as we increase the observability of chaos.

--

--

Russ Miles
Chaos Toolkit

People, Team and Organizational Developer. Writer, psychologist, speaker and humanistic Head of Engineering. https://twitter.com/russmiles