Failover Testing

Twin Sisters mean redundant

This is the sixth article in our Running Kafka in Kubernetes publication. Refer to https://medium.com/kafka-in-kubernetes/accessing-kafka-broker-87aa7928a6e9 to see the previous article.

Motivation

Failover is important in building a distributed system. Our Kafka workload contains 3 replicas and our Kubernetes runs on 2 nodes. In this article, you will see how we perform failover testing.

Change Replicas

Let’s add replica of Kafka to 2

Let’s add replicas to 3

All Replicas are running

Manual Verification of PV

In order to to manual verification of the PV, let’s create 1 file into each of the Kafka container

This is from kafka-0

This is from kafka-1

This is from kakfa-2

Let’s look at where each Pod is located

Right now, I am going to move force kafka Pods to run all in ip-172–31–12–155.

Let’s taint the master node

And now scale kafka to 0

Now scale back to 3

Notice that all kafka pods run in ip-172–31–12–155

Let’s see the volume

Below shows kafka-0 with file intact

Below shows kafka-1 with file intact

Below shows kafka-2 with file intact

Verification From Kafka Consume

Conclusion

You have seen how we performed the failover verification. This proves that NSF Client Provisioner that we use works perfectly.