Events and Blue Green Deployments: Why won’t they get along
Recently I was asked by my manager to have a look on a deployment pattern, called Blue-Green where the source are the traffic is Kafka Messages making it a slice of Event Driven Architecture. Pretty unconventional right? One thing crossed my mind almost instantly, why would we do it, in fact why one would ask for it. Let’s address the elephant in the room.
Blue green deployment
It’s a strategy where you have two identical yet different environments. The idea is that one of these will be running in Live mode (Blue) while other will be used as Live proving for next release (Green). Once you are happy with testing and performance on Green set, you move the traffic completely or gradually (Using a load balancer, as per business need) from active Blue to Green and once you are happy with everything on your Green, it becomes Blue and previous Blue is now ready to be used as Live proving for Next release.
The benefits of this are obvious.
- Easy rollbacks.
- Confidence in release by testing on prod.
- Minimal downtime (may be even zero based on network configuration)
One of the common issue would be lost traffic if there is any application failure on Green and we need to rollback, however with traffic mirroring in asynchronous mode this problem can be resolved as well.
Kafka events or messages
If you reached here, there are high chances that you already know what Kafka is, however for those who don’t know, it’s an Event Streaming framework which can be used to Publish, store and consume events based on your use case. Every message which is published to Kafka, either to be consumed or stored is called an Event. For more details please refer https://kafka.apache.org/.
Now if you haven’t identified the issue yet, then let me reveal it. It’s nature of traffic or source of traffic. Basically blue green deployment strategy has a key element which is element of Control i.e. Load Balancer. Because one of most common form of traffic on Load Balancer is HTTP, we can control it using some parameters and filtering rules, we can instantly change the target from a Proxy and roll it back. However when the source message is an Event rather a Request, the fundaments change. If you consumer is polling Kafka for events then it’s controlled by Consumer Group and will have a metadata associated with it. There can be N number of consumers out there and they can either all be in same consumer group (which is not likely to be in a production application) or they can be in different groups while some sharing same group. Now whenever an application is started (or consumer), it starts polling the event bus and processes it.
In case of blue green deployment, you have two set of application but there is no control on traffic because consumers are stand alone, self readied, pro active clients and doesn’t wait for a push on an Endpoint, as soon as the messages are published in the topic, it’s ready to be consumed. Now we have two options here
- We create new consumers in same consumer group and take advantage of meta data stored in Kafka for that group so that new consumer will be assigned to some partition given that there are free partitions (Generally there won’t be as per Kafka design) and then you need to trigger a rebalance. This can cause some disturbance and then rollback isn’t very easy because consumed messages by new consumer will not be easy to replay for same consumer group.
- If we create new consumers in new consumer group then we are going to have a lot of duplicate messages which might cause a problem for many use cases.
Hence we see there isn’t any definite control on traffic when the source is a Event bus and Blue green might not be apt for a Event Driven Architecture. Blue green is good for System oriented architecture making Web calls and there is a layer of load balancing available.
