The Prefect Blog
Published in

The Prefect Blog

How to use Kubernetes with Prefect: Part 2

Creating custom Kubernetes Job blocks

boats in Amsterdam
Source: pixabay.com

Create a Kubernetes Job block

Example 1: K8s block with EXTRA_PIP_PACKAGES and a few tweaks

blank block creation form for prefect k8s
  • Set Image to prefecthq/prefect:2.6.8-python3.10. If you don’t specify a Docker image tag, Prefect creates a block with an image that matches the Python and Prefect versions inferred from your environment. As the Prefect docs state “It’s a good practice to use Docker images with specific Prefect versions in production.” So here we show how to specify the image version. See more discussion of Prefect’s Docker images in the docs.
  • Set KubernetesImagePullPolicy to Always. This policy ensures you get the freshest version of the specified image, so long as the image repository is available. Docker’s layer caching is smart, so this choice shouldn’t be a performance drag if your image is designed with caching in mind.
  • Change Job Watch Timeout Seconds to 6000. Changes the limit to 100 minutes.
  • Change Pod Watch Timeout Seconds to 6000. Changes the limit to 100 minutes.

Create deployment

screenshot of completed flow run

Example 2: Custom Docker Image

  1. Create the Dockerfile and requirements.txt files.
  2. Build the Docker image.
  3. Push the Docker image to your Docker registry.
  4. Create a K8s block and specify your image in the Image field.
  5. Build and apply your deployment. Note that we need to rebuild our deployment if we change our infrastructure block. ⚠️
  6. Start your agent.
  7. Run your deployment.
  8. Profit. 🤑

Create Dockerfile and requirements.txt

FROM prefecthq/prefect:2.6.8-python3.10
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host pypi.python.org --no-cache-dir

Build image

Push image

Create K8s block

block form for creating the block that shows custome image

Build deployment

Start agent

Run deployment

terminal output screenshot showing flow ran

Example 3: Custom K8s manifest

Create K8s Base Job Manifest

apiVersion: batch/v1
kind: Job
metadata:
# labels are required, even if empty
labels: {}
spec:
template:
spec:
completions: 1
containers: # the first container is required
- env: [] # env is required, even if empty
name: prefect-job
parallelism: 1
restartPolicy: Never

Create K8s Job Bock

from prefect.infrastructure import KubernetesJob

k8s_custom_block = KubernetesJob(
image="prefecthq/prefect:2-python3.10",
env={"EXTRA_PIP_PACKAGES": "s3fs"},
job=KubernetesJob.job_from_file("./k8s_flow_run_job_manifest.yaml"),
image_pull_policy="Always",
job_watch_timeout_seconds=6000,
pod_watch_timeout_seconds=6000,
)

if __name__ == "__main__":
k8s_custom_block.save("marvin3", overwrite=True)
k8s base job manifest prefect block section

Build deployment

Run deployment

output from flow run in terminal screenshot

Recap

Wrap

brightly colored houses on water with small boats in front
Source: pixabay.com

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jeff Hale

I write about data science. Join my Data Awesome mailing list to stay on top of the latest data tools and tips: https://dataawesome.com