Kubernetes Time Traveling — Chaos Engineering with Gremlin

Tammy Bryant (Butow)
Sep 25 · 4 min read
Image for post
Image for post

This tutorial shares how you can utilize the Gremlin Time Travel attack to change clock time. This attack is cloud-agnostic and will work across AWS, GCP, Azure, DigitalOcean, and more.

Here are a few reasons to use the Time Travel attack:

  • Ensure your systems can effectively handle certificate expiration
  • Prepare for unknown-unknown incidents caused by clock skew
  • Prepare for unexpected downtime

Prerequisites

  • A Gremlin account (sign up here)
  • Your Gremlin daemon credentials
  • A kubernetes cluster

Time Travel a Kubernetes node using Gremlin

Kubernetes architecture is commonly 1 primary and 2 or more nodes which are replicated from the primary. When the primary dies the nodes are ready to replace it. When one node dies another will be ready to replace it.

Step 1 — Install the Gremlin Agent

The simplest way to install the Gremlin agent on your Kubernetes cluster is to use Helm. If you do not already have Helm installed, go here to get started. Once Helm is installed and configured, the next steps are to add the Gremlin repo and install the client.

Add the Gremlin Helm chart:

Create a namespace for the Gremlin Kubernetes client:

Next you will run the helm command to install the Gremlin client. In this command there are three placeholder variables that you will need to replace with real data. Replace $GREMLIN_TEAM_ID with your Team ID from Step 1.1, and replace $GREMLIN_TEAM_SECRET with your Secret Key from Step 1.1. Replace $GREMLIN_CLUSTER_ID with a name for the cluster.

If you are using Helm v3, run this command:

For more information on the Gremlin Helm chart, including more configuration options, check out the chart on Github.

Step 2 — View the current clock time and disable NTP

Use the built-in Linux date tool check the current system time

You will see a result similar to the following:

Disable NTP on the instance:

Step 3 — Creating a Time Travel Attack against a Kubernetes node using the Gremlin App

You can use the Gremlin App or the Gremlin API to trigger Gremlin Attacks and Scenarios. You can view the available range of Gremlin Attacks in Gremlin Help.

To create a Time Travel Scenario, click Scenarios in the left Navigation bar click to create a new Scenario

Host targeting should be selected by default. Click on the Exact button to expand the list of available hosts, and select one of them. You’ll see the Blast Radius for the attack is limited to 1 host.

Image for post
Image for post

Click “Choose a Gremlin,” and then select State and Time Travel. Leave the Length set to 60 seconds. Leave the radio button for NTP set to “No,” as we’ve already disabled NTP on the host. Leave the offset set to 86400 second. That’s the amount of clock drift that will be introduced. Then hit the green Unleash Gremlin button.

Image for post
Image for post

Next click to save your scenario:

Image for post
Image for post

Now you can run your Kubernetes Time Travel Scenario:

Image for post
Image for post

When your Scenario is finished you will be prompted to add your results to the Gremlin App.

Image for post
Image for post

Step 4— Check the new adjusted clock time

Using the built-in Linux date tool check the adjusted system time:

Conclusion

How does changing the clock time impact your Kubernetes cluster? Share your findings in the comments below!

Chaos Engineering

Tammy Bryant (Butow)

Written by

Chaos Engineering
Tammy Bryant (Butow)

Written by

Principal Site Reliability Engineer @GremlinInc http://gremlin.com | Chaos Engineering ☁️ 💻 ⚡️💀 Previously @DigitalOcean @Dropbox @NAB @QUT

Chaos Engineering

Chaos Engineering

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store