Wildfly 10 kubernetes cluster

How to create a Wildfly cluster on kubernetes with KUBE_PING

3 min readOct 22, 2018

Introduction

There are many legacy applications that have been developed to run on Wildfly 10.1 clusters (configurations ha or full-ha), but can’t migrate them easily to a kubernetes environment.

In this post I’ll show how to do it using a simple and open-source way from the creators of JGroups.

It works within any kubernetes infrastructure (Amazon EKS, Google GKE, Azure AKS, minikube, provisioning via KOPS etc).

Wildfly cluster

Wildfly uses JGroups internally to discover other Wildflys and to be able to synchronize its services (clustering).

If, for example, two wildflys with high availability configuration (ha or full-ha) runs on a physical machine, they will automatically discover themselves and create a cluster with each other. This is due to the fact that a local network supports multicast and wildfly is configured by default to use udp multicast as the discovery protocol.

But multicast is disabled by default on kubernetes clusters. Thus, even if you deploy a wildfly’s with more than 1, they will not be clustered.

So for wildflys to discover and synchronize (cluster), you must enable a specific JGroups discovery protocol (the KUBE_PING below).

KUBE_PING

KUBE_PING is a discovery protocol for JGroups cluster nodes managed by Kubernetes.
Since Kubernetes is in charge of launching nodes, it knows the IP addresses of all pods it started, and is therefore the best place to ask for cluster discovery.
Discovery is therefore done by asking Kubernetes API for a list of IP addresses of all cluster nodes.
The protocol spins up a local HTTP Server which is used for sending discovery requests to all instances and wait for the responses.

For Wildfly 10.1 you should use the jgroups-kubernetes artifacts in version 0.9.3 to be compatible with Wildfly’s internal JGroups.

JGroups is responsible for finding out what other wildflys there are in the cluster to synchronize them. For JGroups to work in an kubernetes cluster, it must be configured for tcp stack and use the KUBE_PING discovery protocol.

<protocol type=”org.jgroups.protocols.kubernetes.KUBE_PING” module=”org.jgroups.kubernetes”>

KUBE_PING does a direct query to the KUBERNETES API to discover the IPs of the PODs that will form the cluster. To do this, enable the query for the serviceaccount used:

$ kubectl create rolebinding default-viewer \
 — clusterrole=view \
 — serviceaccount=default:default \
 — namespace=default

In order for KUBE_PING to health check on other PODs, you must enable ports 7600 and 8888 in the Wildfly deployment:

Basic Steps

The complete project is at: https://gitlab.com/Fernandes/wildfly10-kubeping

Download or compile the kube_ping artifacts in a wildfly compatible version (for wildfly 10.1 the version is 0.9.3);
Create a docker with the artifacts in a wildfly’s module;
Modify the wildfly configuration (standalone-ha.xml or standalone-full-ha.xml) to use the new discover protocol KUBE_PING;
Deploy on kubernetes and check the logs to confirm wildfly’s cluster forming.

kube_ping artifacts

Wildfly’s module

Wildfly’s modified configuration

Kubernetes deployment

Verifying wildfly’s cluster logs

To make it easier to view the cluster being formed, let’s use standalone-full-ha.xml.

Note on ActimeMQ password:

<cluster password=”${jboss.messaging.cluster.password:mypass”/>

Correct log showing boot of undertow responsible for health check for KUBE_PING

18:57:15,530 INFO  [org.jgroups.protocols.kubernetes.KUBE_PING] (MSC service thread 1-4) Starting UndertowServer on port 8888 for channel address: wildfly10-kubeping-789477d79-6xjgq

Open the wildfly’s logs and verify that they are synchronizing the infinispan caches.

18:57:41,604 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,wildfly10-kubeping-789477d79-6xjgq) ISPN000094: Received new cluster view for channel server: [wildfly10-kubeping-789477d79-6xjgq|1] (2) [wildfly10-kubeping-789477d79-6xjgq, wildfly10-kubeping-789477d79-k4r67]
18:57:41,624 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,wildfly10-kubeping-789477d79-6xjgq) ISPN000094: Received new cluster view for channel web: [wildfly10-kubeping-789477d79-6xjgq|1] (2) [wildfly10-kubeping-789477d79-6xjgq, wildfly10-kubeping-789477d79-k4r67]
18:57:41,632 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,wildfly10-kubeping-789477d79-6xjgq) ISPN000094: Received new cluster view for channel hibernate: [wildfly10-kubeping-789477d79-6xjgq|1] (2) [wildfly10-kubeping-789477d79-6xjgq, wildfly10-kubeping-789477d79-k4r67]
18:57:41,633 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,wildfly10-kubeping-789477d79-6xjgq) ISPN000094: Received new cluster view for channel ejb: [wildfly10-kubeping-789477d79-6xjgq|1] (2) [wildfly10-kubeping-789477d79-6xjgq, wildfly10-kubeping-789477d79-k4r67]

All set!!!
Wildfly’s cluster are online on kubernetes!!

To see new PODs entering or being removed from the cluster, you can use the command kubectl scale --replicas=4 -f kubernetes.