Activating an Apache Ignite cluster on Kubernetes

Stephen Darlington
Dec 7, 2018 · 3 min read

A couple of weeks ago I did a talk at the OpenStack conference about deploying Apache Ignite on the Kubernetes container orchestration system. As with any talk, with only so much time to work with, some things ended up on the cutting room floor.

One of those things was how to activate the cluster when using Ignite’s native persistence support. I mentioned that you needed to do it — since it’s one of the things that complicates using a distributed database in these environments — but I didn’t really say how. In the demo, I just logged straight into one of the already running pods and ran control.sh --activate. You could argue that this is not an unreasonable thing to do. Activating the cluster is something you only do once so a bit of a hack, something that’s not very Kubernetes-ish, wouldn’t be the worst thing you could do.

But, we can do better.

One option might be to use kubectl exec. You can run a command something like this:

kubectl exec -it ignite-0 --namespace=ignite -- /opt/ignite/apache-ignite-fabric/bin/control.sh --activate

That would work. But to my way of thinking this is no better than directly ssh-ing into the pod and manually running a command. Having said that, one nice thing about this command is that it relies on one of the features of StatefulSets: unique and predictable names for all the pods. The first pod is always going to have a zero postfix.

My second thought was to expose the appropriate ports in the cluster and connect from the client using Ignite’s --host and --port options. I was pretty confident that this would do the trick:

./control.sh --host service --port 30000 --activate

I was wrong.

In hindsight it was pretty obvious why. While the service was exposing itself (pardon the expression) on port 11211, the script was trying to access it on port 30000. That would be confusing enough, but there were probably other ports that it was trying to access (47100, 47500), all of which would have been available on the wrong port. Even using an ssh tunnel I couldn’t get it working. That’s not to say that it couldn’t be made to work, but we can do better.

The real way to do it is to create a Kubernetes job.

I was trying to avoid this since I thought it might be quite tricky but it turned out to be fairly straightforward.

apiVersion: batch/v1
kind: Job
metadata:
name: ignite-activate
spec:
ttlSecondsAfterFinished: 100
template:
spec:
containers:
— name: ignite-image
image: apacheignite/ignite:2.6.0
command: [“/opt/ignite/apache-ignite-fabric/bin/control.sh”, “ — activate”, “ — host ignite”]
restartPolicy: Never

Then you kick it off with kubectl create -f ignite-activate.yaml.

It’s not rocket-science, but here a couple of small things that I missed on my first pass. Number one, you and I know that it only needs to be run once, that it’s not a daemon. Kubernetes doesn’t know that, we need to tell it. That’s the restart policy. Without that line, it’ll keep kicking off the job.

Next, the “ttlSecondsAfterFinished” line. This is currently in alpha, so, depending on the version of the server that you’re running, it either might not work at all or you’ll need to enable experimental features. But it’s neat so I thought I should include it. After running the job, it waits 100 seconds — after you’ve had time to check the logs — it tidies up after itself, deleting pods, and any other resources it used. Without that flag, you’ll have to do that step yourself. So much for automation.

In the end, the “best” way might also actually be the easiest. How often does that happen?

Stephen Darlington

Written by

Started coding on a Sinclair Spectrum in 1985. Thinking about upgrading soon. I write about my experiences in the software industry.