Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide

Apache SeaTunnel
CodeX
Published in
3 min readApr 23, 2024

--

Editor | Debra Chen
SeaTunnel offers a method to run the Zeta Engine (local mode) on Kubernetes, allowing Kubernetes to run the Zeta Engine locally, achieving more efficient application deployment and management. In this article, we will explore more information about running SeaTunnel with k8s to run the Zeta engine in local mode, understanding how to better leverage the advantages of the Zeta Engine.

  1. Upload SeaTunnel to the server.
    I have previously unpacked and executed install-plugin.sh. Here, for convenience, I'm using the seatunnel after executing the install-plugin.shscript.

The lib directory after executing the install-plugin contains the following:

tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz 
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3

2. Build the SeaTunnel image.

Create a Dockerfile in the same directory where you installed SeaTunnel. Configure it as follows, you can choose the version:

FROM openjdk:8

ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel

Execute the command:

docker build -t seatunnel:2.3.3 -f Dockerfile .

3. View the image:

docker images

4. Load the image into Kubernetes. Here, I’m demonstrating using minikube:

minikube image load seatunnel:2.3.3

5.

minikube image ls

Check the image

6. Create seatunnel.streaming.conf:

env {
execution.parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 2000
}

source {
FakeSource {
result_table_name = "fake"
row.num = 160000
schema = {
fields {
name = "string"
age = "int"
}
}
}
}

transform {

}

sink {
Console {}
}

7. Create a ConfigMap:

kubectl create cm seatunnel-config \
--from-file=seatunnel.streaming.conf=seatunnel.streaming.conf

8. Create seatunnel.yaml as shown below:

apiVersion: v1
kind: Pod
metadata:
name: seatunneltest
spec:
containers:
- name: seatunnel
image: seatunnel:2.3.3
command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel.sh --config /data/seatunnel.streaming.conf -e local"]
volumeMounts:
- name: seatunnel-config
mountPath: /data/seatunnel.streaming.conf
subPath: seatunnel.streaming.conf
volumes:
- name: seatunnel-config
configMap:
name: seatunnel-config
items:
- key: seatunnel.streaming.conf
path: seatunnel.streaming.conf
~

Execute:

kubectl apply -f seatunnel.yaml

View pod results:

--

--

Apache SeaTunnel
CodeX
Writer for

The next-generation high-performance, distributed, massive data integration tool.