One Command to Rule Them All: Revolutionizing Data Engineer Onboarding with Docker Magic.

Constellar tech corner illustration (generated by DALL-E)

Welcome to Constellar’s tech corner, where innovation meets practicality. Today, we’re excited to share a transformative approach to onboarding data engineers that’s as streamlined as your morning coffee routine.

Imagine a new engineer joining your team and, on their very first day, setting up their entire development environment with a single command. This isn’t just a dream — it’s reality, thanks to Docker Compose.

Illustration image generated by DALL-E (prompt is in ALT text)

The “Aha!” Moment

Picture this: a data engineer sits with a cup of coffee, the steam swirling up like the ideas in their mind. They’re thinking about the onboarding process — those first few days of navigating new databases can be as daunting as opening a puzzle box with no instructions. But then, in the midst of a caffeine-induced epiphany, it hits them: Why not make this simpler? Why not indeed!

Enter Docker Compose

Imagine if setting up your development environment was as easy as making instant noodles — just add water, and voilà! Well, we’ve concocted a recipe just as simple for our data engineers, and it’s called docker-compose.yml. Just one command, and you’ll have your database, migration tool, and application running faster than you can say “containerization”.

version: '3.1'

services:
  db:
    image: postgres:latest
    restart: always
    env_file:
      - .env
    ports:
      - "${POSTGRES_PORT}:5432"

  flyway:
    image: flyway/flyway
    command: -url=${FLYWAY_URL} -schemas=${FLYWAY_SCHEMAS} -user=${FLYWAY_USER} -password=${FLYWAY_PASSWORD} migrate
    volumes:
      - ./sql:/flyway/sql
    depends_on:
      - db
    environment:
      - FLYWAY_URL
      - FLYWAY_SCHEMAS
      - FLYWAY_USER
      - FLYWAY_PASSWORD

  python-app:
    build: .
    depends_on:
      - flyway
    env_file:
      - .env
    volumes:
      - .:/code
    working_dir: /code
    command: python sample_script.py

With this file, the terminal becomes a magic wand. A flick of the wrist (or more accurately, a tap of the fingers) and docker-compose up — build animates the lifeless code into bustling containers. It’s like watching a symphony where each musician knows precisely when to come in.

Flyway Migrations: Choreographed SQL Dance

Now, we’re not ones to let our database do a freestyle, so we brought in Flyway to choreograph the SQL dance. With each migration script named in perfect harmony, like V20240410__Create_table_name.sql, Flyway orchestrates the movements so gracefully, you’d think it’s the SQL equivalent of Swan Lake.

For more details about the naming convention and concept of Flyway, please visit their official documentation.

Python Application: The Symphony Conductor

The Python app, on the other hand, is the conductor, awaiting the baton pass from Flyway. It stands ready with its sample_script.py, and when the migrations are set, it leaps into action, querying and managing data like a maestro leading a crescendo.

import os
import time
import psycopg2
from psycopg2 import OperationalError

DB_NAME = os.getenv('POSTGRES_DB')
DB_USER = os.getenv('POSTGRES_USER')
DB_PASSWORD = os.getenv('POSTGRES_PASSWORD')
DB_HOST = 'db'  # Name of the service in docker-compose

def connect_to_db(attempts=5, delay=5):
    for attempt in range(attempts):
        try:
            conn = psycopg2.connect(
                dbname=DB_NAME,
                user=DB_USER,
                password=DB_PASSWORD,
                host=DB_HOST
            )
            print("Connection to database established")
            return conn
        except OperationalError as e:
            print(f"Connection attempt {attempt + 1}/{attempts} failed: {e}")
            time.sleep(delay)
    print("Could not connect to the database")
    return None

conn = connect_to_db()
if conn:
    cur = conn.cursor()

    cur.execute("SELECT * FROM {table_name}")
    records = cur.fetchall()

    for row in records:
        print(row)

    cur.close()
    conn.close()

The .env: Secret Keeper

Ah, the .env file — our trusted keeper of secrets, holding the environment variables close, like a gambler clutching their cards. It whispers to the containers, “Here’s what you need to know,” and like that, they’re off to the races, with no sensitive data left hanging out to dry.

Running the Magic Command

You are all set! The next step is as straightforward as it gets. At Constellar, we ensure that even our newest engineers can get their environment set up with a single command. Here’s how you can do it too:

• Open your Terminal or Command Prompt: Navigate to the directory where your docker-compose.yml file is located.
• Run the Docker Compose Command: Type the following command and press Enter:

docker-compose up --build

This command will start the process of building and running all the services defined in your Docker Compose file. It pulls the necessary Docker images, builds your application from the Dockerfile if specified, and creates the containers as defined.

• Observe the Magic: Watch as Docker Compose orchestrates the setup of your entire development environment. This process might take a few minutes, depending on your system and network speed.

Below is a screenshot showing the terminal output once the containers are successfully created and running. This visual confirmation is what we love to see — a fully operational development environment set up with zero hassle.

The screenshot should display the terminal with logs indicating that the services are up and running, such as:

Next Steps

With your containers up and running, you’re now ready to start coding! Here are a few tips on what to do next:

• Check the container status: Use the command docker ps to see all the containers that are currently running.
• Access your application: Depending on your setup, navigate to the local address where your application is hosted, usually something like http://localhost:8000.
• Begin development: Start by exploring the codebase, making changes, and seeing the immediate updates in your application, thanks to Docker’s volume mapping.

This practical guide ensures that every Constellar engineer — and hopefully every reader — can leverage Docker Compose to maximize productivity from day one.

From Constellar’s Perspective: Streamlining Onboarding

By adopting Docker Compose, Constellar has not only simplified the technical setup for our new engineers but also encapsulated our commitment to efficiency and innovation. This approach has cut our onboarding times significantly, allowing our team members to dive straight into productivity.

Conclusion: Constellar’s Commitment to Innovation

As you’ve seen, Docker Compose is more than a tool for Constellar — it’s a testament to our drive for simplifying complex processes. We hope our story inspires other organizations to consider how small changes can make significant impacts.

At Constellar, we’re always looking ahead, ready to embrace the next challenge with open arms and innovative solutions. Until next time, keep exploring the possibilities that technology can bring into your professional life. 💻