Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide

Editor | Debra Chen
SeaTunnel offers a method to run the Zeta Engine (local mode) on Kubernetes, allowing Kubernetes to run the Zeta Engine locally, achieving more efficient application deployment and management. In this article, we will explore more information about running SeaTunnel with k8s to run the Zeta engine in local mode, understanding how to better leverage the advantages of the Zeta Engine.

1. Upload SeaTunnel to the server.
   I have previously unpacked and executed install-plugin.sh. Here, for convenience, I'm using the seatunnel after executing the install-plugin.shscript.

The lib directory after executing the install-plugin contains the following:

tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz 
sh apache-seatunnel-2.3.3/bin/install-plugin.sh 
tar -czvf  apache-seatunnel-2.3.3-bin.tar.gz  apache-seatunnel-2.3.3

2. Build the SeaTunnel image.

Create a Dockerfile in the same directory where you installed SeaTunnel. Configure it as follows, you can choose the version:

FROM openjdk:8

ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel

Execute the command:

docker build -t seatunnel:2.3.3 -f Dockerfile .

3. View the image:

docker images

4. Load the image into Kubernetes. Here, I’m demonstrating using minikube:

minikube image load seatunnel:2.3.3

5.

minikube image ls

Check the image

6. Create seatunnel.streaming.conf:

env {
  execution.parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 2000
}

source {
    FakeSource {
      result_table_name = "fake"
      row.num = 160000
      schema = {
        fields {
          name = "string"
          age = "int"
        }
      }
    }
}

transform {

}

sink {
  Console {}
}

7. Create a ConfigMap:

kubectl create cm seatunnel-config \
--from-file=seatunnel.streaming.conf=seatunnel.streaming.conf

8. Create seatunnel.yaml as shown below:

apiVersion: v1
kind: Pod
metadata:
  name: seatunneltest
spec:
  containers:
  - name: seatunnel
    image: seatunnel:2.3.3
    command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel.sh --config /data/seatunnel.streaming.conf -e local"]
    volumeMounts:
      - name: seatunnel-config
        mountPath: /data/seatunnel.streaming.conf
        subPath: seatunnel.streaming.conf
  volumes:
        - name: seatunnel-config
          configMap:
            name: seatunnel-config
            items:
            - key: seatunnel.streaming.conf
              path: seatunnel.streaming.conf
~

Execute:

kubectl apply -f seatunnel.yaml

View pod results: