Deployment Guide for Apache SeaTunnel Kubernetes Cluster Mode with Zeta Engine

Apache SeaTunnel
CodeX
Published in
4 min readApr 30, 2024

--

SeaTunnel provides a method to run the Zeta engine in cluster mode, allowing Kubernetes to locally deploy the Zeta engine for more efficient application deployment and management. In this guide, we will explore more about running the Zeta engine in cluster mode with SeaTunnel on Kubernetes and understand how to leverage the advantages of the Zeta engine better.

  1. Upload SeaTunnel to the server. Previously, I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script.

After executing the install-plugin, the lib directory contains the following:

tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3

2. Build the SeaTunnel image. Create a Dockerfile in the same directory where SeaTunnel is installed. Configure as follows, you can choose the version yourself:

FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel

Execute the command:

docker build -t seatunnel:2.3.3 -f Dockerfile.

3. View the image:

docker images

The image appears as follows:

4. Load the image into Kubernetes, here I’m using Minikube for demonstration:

minikube image load seatunnel:2.3.3

Refer to the above link for details: Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide

5. Create config maps as follows:

kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml
kubectl create configmap hazelcast --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml

6. Use Reloader to automatically restart pods after updating configmaps:

wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
kubectl apply -f reloader.yaml

7. Create seatunnel-cluster.yml as follows:

apiVersion: v1
kind: Service
metadata:
name: seatunnel
spec:
selector:
app: seatunnel
ports:
- port: 5801
name: seatunnel
clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: seatunnel
annotations:
configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
serviceName: "seatunnel"
replicas: 3
selector:
matchLabels:
app: seatunnel
template:
metadata:
labels:
app: seatunnel
spec:
containers:
- name: seatunnel
image: seatunnel:2.3.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5801
name: client
command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
resources:
limits:
cpu: "1"
memory: 4G
requests:
cpu: "1"
memory: 2G
volumeMounts:
- mountPath: "/opt/seatunnel/config/hazelcast.yaml"
name: hazelcast
subPath: hazelcast.yaml
- mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
name: hazelcast-client
subPath: hazelcast-client.yaml
- mountPath: "/opt/seatunnel/config/seatunnel.yaml"
name: seatunnelmap
subPath: seatunnel.yaml
volumes:
- name: hazelcast
configMap:
name: hazelcast
- name: hazelcast-client
configMap:
name: hazelcast-client
- name: seatunnelmap
configMap:
name: seatunnelmap

8. Execute:

kubectl apply -f seatunnel-cluster.yml

9. Modify the configuration in config maps:

kubectl edit cm hazelcast
Modify the cluster address.
Here, headless service access mode is used.
The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local

Example:

  • seatunnel-0.seatunnel.default.svc.cluster.local
  • seatunnel-1.seatunnel.default.svc.cluster.local
  • seatunnel-2.seatunnel.default.svc.cluster.local

Reminder: Use spaces instead of tabs. Otherwise, it will cause errors.

kubectl edit cm hazelcast-client
kubectl edit cm seatunnelmap

Modify it to your own HDFS address.

10. You can see the following result:

11. After all nodes have finished updating and are running, you can enter the container to check if the paths have been modified:

kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml

12. Check the logs inside the container:

tail -200f logs/seatunnel-engine-server.log

We can see that the cluster is running normally.

13. Run tasks:

You can open a new connection, log in to another pod node, and execute tasks to test the cluster:

kubectl exec -it seatunnel-1 /bin/bash
bin/seatunnel.sh --config config/v2.streaming.conf.template

We can see that tasks have also started running in other pods.

That’s all for running the Zeta engine in cluster mode with SeaTunnel on Kubernetes!

--

--

Apache SeaTunnel
CodeX
Writer for

The next-generation high-performance, distributed, massive data integration tool.