Hazelcast Cluster Setup in Kubernetes

Here we describe the general steps to set up a Hazelcast cluster in Kubernetes:

1. Create a Docker image with Hazelcast and your application code. You can use a Dockerfile to build the image and copy the Hazelcast configuration file and any other required files.
2. Create a Kubernetes deployment for the Hazelcast cluster with the Docker image you created in step 1 or you can use existing Hazelcast inside docker registry. In the deployment configuration, specify the number of replicas you want for the cluster. For example:

Hazelcast Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hazelcast-deployment
spec:
  replicas: 6
  selector:
    matchLabels:
      app: hazelcast
  template:
    metadata:
      labels:
        app: hazelcast
    spec:
      containers:
      - name: hazelcast
        image: your-docker-image or from publish docker registry
        ports:
        - containerPort: 5701
        volumeMounts:
        - name: hazelcast-config-volume
          mountPath: /opt/hazelcast/config/
      volumes:
      - name: hazelcast-config-volume
        configMap:
          name: hazelcast-config

Hazelcast Service

Create a Kubernetes service to expose the Hazelcast deployment to other pods in the cluster. For example:

apiVersion: v1
kind: Service
metadata:
  name: hazelcast-service
spec:
  selector:
    app: hazelcast
  ports:
  - name: hazelcast
    port: 5701

Hazelcast Cluster Configuration

Create a Kubernetes ConfigMap to store the Hazelcast configuration file (e.g. hazelcast.xml) .

apiVersion: v1
kind: ConfigMap
metadata:
  name: hazelcast-config
data:
  hazelcast.xml: |
    <hazelcast>
      <network>
        <join>
          <multicast enabled="false" />
          <tcp-ip enabled="true">
            <interface>eth0</interface>
            <member-list>
              <member>hazelcast-service.default.svc.cluster.local</member>
            </member-list>
          </tcp-ip>
        </join>
      </network>
    </hazelcast>

In Kubernetes, eth0 is a network interface inside a container. When a container is launched, it has its own network namespace, and by default, the first network interface is named eth0. The eth0 interface is used to communicate with other containers in the same pod or with the Kubernetes cluster network.

In the context of Hazelcast, eth0 interface can be configured in the hazelcast.xml file to enable Hazelcast to communicate with other members in the cluster using the Kubernetes service IP addresses.

Creating Cluster using Statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hazelcast-cluster
spec:
  serviceName: hazelcast
  replicas: 3
  selector:
    matchLabels:
      app: hazelcast
  template:
    metadata:
      labels:
        app: hazelcast
    spec:
      containers:
      - name: hazelcast
        image: you image registry
        volumeMounts:
        - name: config
          mountPath: /opt/hazelcast/
        ports:
        - containerPort: 5701
      volumes:
      - name: config
        configMap:
          name: hazelcast-config
---
apiVersion: v1
kind: Service
metadata:
  name: hazelcast
spec:
  selector:
    app: hazelcast
  clusterIP: None
  ports:
  - name: hazelcast
    port: 5701
    targetPort: 5701

<hazelcast xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.hazelcast.com/schema/config
                               http://www.hazelcast.com/schema/config/hazelcast-config-4.0.xsd">
  <network>
    <join>
      <kubernetes enabled="true"
                   property="kubernetes-service"
                   service-dns="hazelcast-cluster-headless"
                   service-port="5701">
        <namespace>default</namespace>
      </kubernetes>
    </join>
  </network>
</hazelcast>

If we provide a Kubernetes service as a seed node in the Hazelcast cluster, then all the pods associated with that service would be used as the initial list of members to join the cluster. This approach has some benefits such as automatic discovery of new members as pods are created or deleted, load balancing and service discovery are handled by Kubernetes.

However, there are some considerations to keep in mind. First, the service should be a headless service to ensure that each pod in the StatefulSet gets its own DNS entry, which will be used to identify each member in the cluster. Second, when using a Kubernetes service as a seed node, it is recommended to set the prefer-ip-address attribute to true in the Hazelcast configuration to ensure that the pods use the IP address instead of the hostname to communicate with each other. Finally, it's recommended to have at least three seed nodes for better fault tolerance and stability in the cluster.

Accessing the Hazelcast Cluster from your code

Config config = new Config();
config.getNetworkConfig().getJoin().getTcpIpConfig().addMember("hazelcast-service.default.svc.cluster.local");
HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);

That’s it folks!

Happy Coding!