Building and Testing a Modern Data Analytics Suite Locally with Kubernetes before Launching to Cloud — 2. ClickHouse & Taxbit

From the previous blog, we set up the local Kubernetes environment and install Airflow easily. In the realm of data analytics, agility and efficiency in setting up your environment can significantly enhance your workflow. With Kubernetes becoming the backbone of scalable applications, it’s only fitting to integrate powerful data warehouses like ClickHouse within this ecosystem. This blog post guides you through deploying Bitnami’s ClickHouse on Minikube and setting up Tabix as an intuitive SQL editor UI.

Deploying Bitnami ClickHouse on Minikube

Bitnami’s pre-packaged applications simplify Kubernetes deployments. Here’s how to deploy ClickHouse and integrate it with Airflow on Minikube:

Step 1: Set Up the Helm Chart Repository

Add Bitnami’s Helm chart repository to Helm:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Step 2: Install ClickHouse

Deploy ClickHouse using Helm with the following command:

helm install clickhouse bitnami/clickhouse --namespace default --create-namespace

This command installs ClickHouse with default settings. For custom installations (e.g., setting a password), create a values.yaml and deploy:

helm install clickhouse bitnami/clickhouse -f values.yaml --namespace default

Step 3: Verify Deployment

Ensure the deployment is successful:

kubectl get pods -n default -l app.kubernetes.io/name=clickhouse

Step 4: Port Forwarding for ClickHouse UI

Set up port forwarding to access ClickHouse UI from your local machine:

kubectl port-forward --namespace default svc/clickhouse 8123:8123 &

Access ClickHouse UI via http://localhost:8123.

Integrating Airflow with ClickHouse

Create a new Airflow connection for ClickHouse :

• Conn Id: clickhouse_default
• Conn Type: HTTP
• Host: clickhouse.default.svc.cluster.local
• Schema: default
• Login: your-clickhouse-username
• Password: your-clickhouse-password
• Port: 8123

Deploying Tabix: A Web-Based SQL Editor for ClickHouse

Although Bitnami’s ClickHouse doesn’t include an Admin UI, Tabix offers a sleek web-based alternative. To deploy Tabix in Minikube:

Step 1: Create a Deployment for Tabix

Save a YAML deployment configuration for Tabix, then apply it to your cluster the deployment code.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tabix
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tabix
  template:
    metadata:
      labels:
        app: tabix
    spec:
      containers:
        - name: tabix
          image: spoonest/tabix:latest
          ports:
            - containerPort: 80

kubectl apply -f tabix-deployment.yaml

Step 2: Expose Tabix as a Service

Create a service for Tabix and apply it:

apiVersion: v1
kind: Service
metadata:
  name: tabix
spec:
  type: NodePort
  selector:
    app: tabix
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080

kubectl apply -f tabix-service.yaml

Step 3: Access the Tabix UI

Start the Minikube tunnel if required and access Tabix using the service URL. The default port configured in the service file is 30080.

Step 4: Connect Tabix to ClickHouse

Configure the connection in Tabix UI, you’ll need to provide the connection details to your ClickHouse service:

• Host: Use the ClusterIP of your ClickHouse service or the Minikube IP with the NodePort. You can find this by running kubectl get svc.
• Port: This is the port exposed by the ClickHouse service.
• Database: The ‘default’ is usually default.
• Username: If you’ve set one, the ‘default’ is usually default.
• Password: If required.

With these steps, you now have ClickHouse deployed within Minikube and Tabix as an interactive interface. It’s a testament to the flexibility and power of Kubernetes, even for local development.