I want my_secret.txt file accessible in pod created by KubernetesPodOperator as /app/tmp/my_secret.txt .
Create kubernetes secret
Create kubernetes secret describe in this article.
apiVersion: v1
kind: Secret
metadata:
# secret name in kubernetes
name: k8s-secret
namespace: composer-user-workloads
data:
# key is filename
# value is the content of the file in base64
"my_secret.txt": {{.Values.my_secret_base64}}Apply it via helm or kubectl to kubernetes cluster
Dag
...
from airflow.kubernetes.secret import Secret
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
...
secret = Secret('volume', '/app/tmp', 'k8s-secret')
# same as below
# secret_volume = V1Volume(name='my-secret-vol', secret=V1SecretVolumeSource(secret_name='k8s-secret'))
# secret_volume_mount = V1VolumeMount(mount_path='/app/tmp', name='my-secret-vol', read_only=True)
...
kubernetes_min_pod = KubernetesPodOperator(
task_id="my-pod",
name="my-pod",
secrets=[secret, ],
# verify file existence and its content
cmds=["/bin/bash", "-c", "ls /app/tmp && cat /app/tmp/my_secret.txt"],
# composer user workload
namespace="composer-user-workloads",
image="gcr.io/gcp-runtimes/ubuntu_20_0_4",
# kubeconfig file
config_file="/home/airflow/composer_kube_config",
kubernetes_conn_id="kubernetes_default",
)
...Log
...
{before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 1st time calling it.
{pod_manager.py:395} INFO - [base] my_secret.txt
{pod_manager.py:395} INFO - [base] world peace
...