Running Kafka in a Hybrid Cloud Environment
New configuration options coming in Kafka 0.8.1
With Kafka 0.8.0, if you deploy brokers in a cloud environment, you can run into trouble configuring clients that are not in the same cloud environment. This article describes changes coming in Kafka 0.8.1 to address this.
Why do you need clients outside the cloud environment of the broker?
- You may want to setup Kafka replication using the MirrorMaker tool.
- You may have systems in an existing data center that need to publish to a Kafka cloud deployment.
What’s the issue?
Most often with on-demand compute systems such as EC2 and OpenStack, virtual machines only see their private IP addresses (typically in an RFC 1918 address space) which are not reachable from the outside. With OpenStack, you can assign “floating IPs” (Elastic IPs on EC2) to a virtual machine which forwards traffic to the VM but does not expose the IP as an interface that the VM can bind to.
As shown below, what you want is for external clients to use the floating IP to connect to the broker and for the broker itself to bind it’s private “fixed IP”.
The reason this doesn’t work for Kafka 0.8.0 is that a single configuration parameter is used both for binding a socket on the broker and for advertising the broker in ZooKeeper for clients to discover.
With this patch, which will be included in the Kafka 0.8.1 release, these two settings can now be configured independently.
host.name=<fixed IP>
advertised.host.name=<floating IP>The host.name parameter configures the interface that the broker will bind to. If not set, the server will bind to all interfaces on the VM.
The advertised.host.name parameter specifies the hostname the broker will advertise to producers and consumers. If not set, it uses the value of host.name if set; otherwise it uses the hostname of the VM.
