Most data teams building a data lakehouse think of governance, schema conformity, transactional integrity, or versioning as an afterthought. By the time they get to it, there are already three definitions of “active customer,” two pipelines implementing logic for flavours of the same use case, and several dashboards reading from the same source but producing different numbers for the same metric — quietly eroding trust in the entire data platform.

The cost of getting data governance right is tangible. It prevents data swamps, compliance exposure, and AI systems that hallucinate because the data they were trained on was never reliable to begin with.

The decisions you make in the early phase of a greenfield lakehouse project have a lasting impact on the data platform. I learned this by having lived through the consequences of both getting it right early and retrofitting it later. This post is an account of what building a governed lakehouse actually looks like in practice.

Why do organizations need a data lakehouse?

It is important to understand what a data lakehouse is and what problems does it solve.

Earlier, organisations managed their data lifecycle through three separate systems:

• a data warehouse for structured analytics
• a data lake for raw and unstructured data
• separate ML platforms for model development
• the BI layer on top of the above systems

Each system had its own access control, audit trail and governance framework. This resulted in an architecture where moving data between systems was quite cumbersome along with the added overhead of maintaining multiple copies of the same data, in different formats, with different freshness guarantees, compounding over time.

The lakehouse pattern combines all of this and aims to simplify the data architecture. It combines the flexibility and cost-effectiveness of a data lake with the reliability and query performance of a data warehouse in a single, open-format and unified architecture. Structured, semi-structured, unstructured, and streaming data: all of these coexist in the same platform. Analytics, ML, and AI workloads run against the same underlying data. And there is one governance layer for all of it.

This is technically possible due to the underlying storage format layer. Traditional data lakes lacked ACID transaction support, which makes it possible for failed jobs to corrupt files, multiple pipelines to write concurrently could compromise data integrity, and schema enforcement was absent by default. Modern open table formats like Delta Lake and Apache Iceberg provide ACID guarantees, schema enforcement, time travel, and audit history on top of Parquet, thereby bringing data warehouse reliability to the flexibility of object storage. That is the foundation everything else is built on.

In Databricks, the Delta Lake’s transaction log, house within the _delta_log directory is the mechanism that implements the above concepts of ACID guarantees, schema enforcement, time travel and audit history.

Every write to a Delta table appends a JSON entry to the log recording exactly what changed, when, and by which operation. This means failed jobs do not corrupt or incorrectly/partially write to the table but instead preserve a clean state, and time travel queries ( VERSION AS OF, TIMESTAMP AS OF) are first-class operations which helps users restore previous versions.

However, having a reliable storage format does not automatically mean it is a governed lakehouse.

What is “well-governed” in the context of a data lakehouse?

A well-governed lakehouse:

• contains data that can be trusted
• is secure
• is discoverable
• is reliable.

Governance is baked into the architecture from the beginning, not retrofitted afterward.

It also means the organisation is at some stage of a data maturity curve.

1. In the early stages, the focus is on historical reporting: structured data with batch loads, SQL queries, precanned dashboards.
2. As maturity increases, teams add streaming data, ML workloads, predictive analytics.
3. At the most mature end, organisations build AI systems that learn from their data and its surrounding metadata: RAG pipelines, agents, natural language interfaces.

Each stage of that curve makes higher demands on the governance layer beneath it.

Let us walk through how that looks like in practice.

Layered architecture with clear data contracts

The Medallion architecture: the Bronze / Silver / Gold pattern is widely adopted across the industry.

• Bronze is raw data, which is stored exactly as it is received, is append-only, and contains ingestion metadata.
• Silver is cleaned, conformed, and deduplicated data.
• Gold is aggregated and optimised data, ready for consumption.

However, the architecture is not effective if it is inconsistently implemented. The layers work only with a clear contract definition for each layer:

• what data enters
• in what form
• with what guarantees

This is typically effective when enforced via a combination of process and system checks.

In Databricks, Delta Lake schema enforcement makes this esay. Setting mergeSchema to false (the default) means a pipeline writing a column that does not exist in the target table will fail explicitly rather than silently modify or drift the schema to an undesired structure.

Combined with enforceSchema in Auto Loader, this enforces the schema at the Bronze ingestion layer so that upstream schema changes are caught here, rather than being discovered much downstream when the data has already landed in a Gold table.

For the Silver-to-Gold contract specifically, Delta Lake constraints add another enforcement layer:

ALTER TABLE gold.daily_account_revenue
ADD CONSTRAINT revenue_non_negative
CHECK (daily_revenue_usd >= 0);

A constraint violation at write time blocks the operation and surfaces the problem to the pipeline owner immediately. This is enforcement built into the storage layer, not maintained via a separate data quality check or tooling.

In practice, the contract between the Silver and the Gold layers is where ambiguity typically sets in and also where most governance failures can occur. Teams often debate upon what goes into which layer and what the contracts for the layers are. It is important to achieve a clear consensus early on and stick to it.

If a Gold table is rebuilt from a Silver table whose schema was changed without notice, every downstream consumer breaks. Clear contracts mean documented grain, documented SLAs, and a defined breaking-change policy (most importantly involving Standard Operating Procedures around communication to all stakeholders).

The layer boundaries might feel like overhead at the beginning. As the data platform matures, they ensure that the platform is consistently governed as the number of producers and consumers multiplies.

Global naming convention

When several teams are building pipelines independently, naming conventions diverge over time. A column called customer_id in one domain turns out to be a different entity from customer_id in another - same name, different grain, different update cadence. Joins that look correct produce wrong results. Data quality checks pass and produce data inconsistency or corruption downstream. Reports diverge.

The conventions that mattered most in practice: domain-prefixed table names, consistent use of surrogate versus natural keys across the platform, standardised date column naming ( created_at, updated_at, effective_from, effective_to), and agreed abbreviation rules for column names in data products.

In Unity Catalog, the three-level namespace ( catalog.schema.table) makes it possible to define domain boundaries. A table registered as prod.sales.sales_bookings is architecturally separated from prod.product.events. Users cannot join across domains without explicitly referencing both namespaces. This can be effective only if it is complemented by effective naming conventions, however, it enforces domain separation at the metadata layer.

When data products are declared declaratively, such as the schema, column descriptions, grain, SLAs — naming consistency becomes enforceable rather than advisory. Teams cannot easily drift from a standard that is codified in the data product specification.

On a slightly different note, such problems are usually discovered by Data Quality checks at several points in the pipeline. If a team does not have such checks in place, these problems are reported by customers or business users, which results in delayed problem discovery, wrong decisions being made due to incorrect data and more such probelms. Identifying the source of the problem becomes a tedious task as well, since without the Data Quality checks, it is hard to know where the problem is originating.

Domain ownership

It is not scalable to have a central data team own every table in the lakehouse. Data mesh principles guide such scenarios: the teams closest to the data understand it best, and should own not just the data, but the entire data lifecycle and its governance. In practice, this concept of domain ownership means that the team producing the data is responsible for its quality, its schema evolution, and its SLAs. The platform team provides the guardrails: Unity Catalog access policies, schema validation, lineage tracking and other tooling, but does not own the data itself.

Each table, schema, and catalog in an Unity Catalog has an explicit owner, an user or a group, and permissions are configured and inherited hierarchically. Assigning ownership of the prod.sales schema to the Sales team means they control access grants for their own data products. The platform team retains metastore-level administration without owning the data.

Enforcing these standards prevents teams from building pipelines that do not conform to agreed contracts or using different naming conventions, thereby also preventing them from building tables with fields which cannot be joined reliably across domains.

In practice, the platform team also defines and enforces standards across the metastore — table property requirements, mandatory column tags for PII, required grain and owner properties at table registration. These are enforced via Unity Catalog table properties.

Enforcing these standards prevents teams from building pipelines that do not conform to agreed contracts or using different naming conventions, thereby also preventing them from building tables with fields which cannot be joined reliably across domains.

When a team is responsible for their data products:

• they write column descriptions that explain business meaning, not just data type.
• they define the grain explicitly.
• they version their schemas.
• they own their SLA and are paged when they miss it.

The difference between a team that owns their data and a team that merely produces it is visible in the quality of the metadata.

Data contracts

A data contract is the explicit agreement between a data producer and its consumers. It mainly covers four things: schema, grain, SLAs, and the breaking-change policy.

Schema: what columns exist, their types, and which are nullable. In Delta Lake, the DESCRIBE DETAIL SQL command and the transaction log provide the schema history, including for prior schema version and the timestamp of each of the changes.

Grain: what one row represents: one event, one account per day, one transaction. This is the most commonly undocumented property, which also means that Data Engineers can easily overlook this and implement inconsistent grains. In Databricks, the grain is best documented as a Unity Catalog table property:

ALTER TABLE silver.transactions
SET TBLPROPERTIES (
  'grain' = 'one row per transaction_id',
  'owner' = 'payments-engineering@company.com',
  'sla_hours' = '4'
);

These properties are queryable via SHOW TBLPROPERTIES and surfaced in the Unity Catalog UI, making the grain a first-class and discoverable attribute.

SLA: when is the data expected to be available and how fresh is it?

Breaking-change policy: what constitutes a breaking change, how much notice consumers receive, and what the migration path looks like.

An example scenario: a producer table’s column type was recently changed from integer to string during a routine pipeline refactor. The downstream Gold table schema data quality check caught it six hours later, after three dashboards had already served incorrect aggregations to business stakeholders. Tracing the root cause took two engineers almost a day. The fix took twenty minutes thereafter. A contract with a breaking-change policy would have prevented the type change from reaching downstream without a version increment and consumer notification.

Governance and access

Data classification precedes access control decisions. Before you can apply access controls, you need to know if the incoming data contains: PII, user generated content, financial data, internal-only metrics, publicly referenceable aggregates.

For instance, Databricks Unity Catalog’s tagging system handles this if used consistently.Tags applied at the column level propagate to downstream derived tables through lineage — meaning a column tagged pii:email in Bronze retains that classification when it appears in Silver and Gold, without requiring manual re-tagging at each layer:

ALTER TABLE bronze.raw_events
ALTER COLUMN user_email
SET TAGS ('pii' = 'email', 'sensitivity' = 'high');

Unity Catalog’s column masking policies then apply automatically based on these tags. So, a role without PII access sees user_email as **** without needing masking code to be implemeted as part of the data pipeline:

CREATE ROW FILTER mask_pii_email ON bronze.raw_events
AS (user) -> CASE
  WHEN is_member('pii-access') THEN true
  ELSE user_email IS NULL
END;

Access controls in most cloud platforms include RBAC (role-based) and ABAC (attribute-based) access controls which provide controls at different granularities. RBAC is table-level and schema-level. ABAC is row-level and column-level. For example, they could be used to mask a PII column for one role while exposing it to another.

PII handling includes masking, but it also includes retention policies, audit logging of who accessed what, and ensuring that PII does not leak through derived columns or aggregations. Governance in regulated environments has to account for derived sensitivity, not just source sensitivity.

Delta Lake’s VACUUM command honours retention thresholds. For PII data with hard deletion requirements, the correct and complete way to delete the data is via the column-level deletion combined with REORG TABLE ... APPLY (PURGE) to physically remove the deleted data from Parquet files. Marking the data as deleted in the transaction log would not suffice.

Data discovery and audits

Data discovery is a hard problem to solve and it becomes challenging over time. This is a problem that partially occurs due to missing or incomplete Data Lineage.

When a number in a board-level dashboard is wrong, the first question is always “where does this come from?” Without an end-to-end data lineage, that question takes days to answer without end-to-end data flow lineage. With lineage, it is a simple query of the tables storing the lineage relationships.

Unity Catalog captures column-level lineage automatically for SQL operations on Databricks: a Gold table metric traced back to the Silver layer’s transformations and back to the Bronze layer’s ingestion, with every intermediate table and column visible in the lineage graph. No additional steps or configuration are required for setting up lineage for SQL workloads.

Python and Spark DataFrame operations require the appropriate writing patterns (using spark.sql() or Delta Lake merge operations) to appear in the lineage graph.

What a catalogue is actually for — versus what people think it is for: most teams treat the data catalogue as a documentation project. A place to write descriptions and tag tables. That is the wrong framing. A catalogue is a runtime artefact. It reflects the live state of the data platform: what tables exist, what they contain, who owns them, when they were last updated, and how they relate to each other. Documentation that lives separately from the pipeline inevitably falls behind the pipeline. Descriptions written at data product creation time — as part of the schema definition — stay current because they are versioned alongside the schema.

Tagging and classification tiers work best when they are mandatory at data product registration time, not optional at consumption time. A sensitivity tag added when a table is first defined is almost always more accurate than one added retroactively when a compliance audit surfaces it.

Data observability

Data observability covers five properties: freshness, volume, distribution, schema, and lineage.

Freshness: is the data as recent as defined in the SLA? On Delta Lake, DESCRIBE HISTORY provides the timestamp of the most recent write operation - making freshness checks queryable without external tooling:

SELECT timestamp, operation, operationParameters
FROM (DESCRIBE HISTORY silver.transactions)
WHERE operation = 'WRITE'
ORDER BY timestamp DESC
LIMIT 1;

Volume: are row counts within expected bounds? Are we recording the % of deviation from the average row counts for the past 30 days? Delta Lake table history also records the number of files added and removed per operation, giving a write time volume signal without a full table scan.

Distribution: are column value distributions stable? A shift in the distribution of a key dimension is often the first signal of an upstream data quality issue

Schema: did the schema change without a corresponding contract update? Delta Lake’s transaction log records every schema change with a timestamp, making it auditable by comparing the current schema against a prior schema version:

from delta.tables import DeltaTable

dt = DeltaTable.forName(spark, "silver.transactions")
history = dt.history(10)

Lineage: can you trace a value in a Gold table back to its source system?

The ownership question matters more than the tooling. A team that owns their data product owns the observability signals for that product.

• They set the expected volume numbers.
• They define what “freshness” means for their SLA.
• They are paged when the distribution shifts.

By making observability declarative, we built it into pipelines without making it another task to be performed later. Expected row count ranges, null rate thresholds, and distribution are defined in the data product specification alongside the schema. The pipeline validates against them at each layer. Violations block promotion from Bronze to Silver. This is not a separate observability system — it is the pipeline enforcing the contract at runtime.

Semantic layer

One governed definition per metric. This is the principle that most data teams agree with consistently.

A semantic layer sits between the physical data model and the consumer — whether that consumer is a BI tool, an analyst running SQL, or an AI agent generating queries. It defines what “revenue” means, what “active user” means, what time zone “today” refers to. One definition, one place, applied consistently.

Without it, different teams build their own definitions in their own tools. The organisation ends up with five different revenue numbers depending on which dashboard you open. This is not a data quality problem, it is a governance problem. The data is correct; the definitions are inconsistent.

The semantic layer is also where the connection to AI readiness becomes concrete. An AI agent generating SQL against your lakehouse will use whatever metric definitions it can infer from column names and table descriptions. If those definitions are inconsistent, the agent produces inconsistent results.

AI-readiness

An LLM querying your lakehouse — whether through a natural language query interface, a RAG pipeline, or a custom agent depends entirely on the quality of your metadata. Column descriptions that say “flag” or “id” are useless to a language model. Column descriptions that say “binary indicator set to 1 when the account has had an active subscription within the last 30 days” are genuinely useful. The difference between a Genie Space that returns accurate results and one that hallucinates is largely determined by the quality of the column-level annotations in Unity Catalog.

AI-readiness means:

• descriptions are written for a language model, not a data engineer.
• embeddings exist for columns and tables so that semantic search over the data catalogue works.
• agent-accessible metadata: what tables exist, what they contain, how they relate — is current and accurate. This is possible if it is maintained at data product definition time rather than retroactively.

A lakehouse without this is not AI-ready. Point an LLM at undocumented tables with opaque column names and you will get confident, plausible, wrong answers. The data quality problem that used to surface as a wrong number in a dashboard now surfaces as a confident hallucination in an AI system that business users trust more, not less, than the dashboard it replaced.

Walking through one example end to end

Follow a single transaction entity from source through to the reporting layer to see how naming, contracts, lineage, and quality checks touch the same record at different points.

Bronze: the transaction arrives from the source system exactly as received as a raw JSON, appended to the Bronze table with an ingested_at timestamp. No transformation. The schema is documented; the grain is one event per row.

Silver: the transaction is cleaned, typed, deduplicated, and conformed to the platform naming convention. txn_id, account_sk, transaction_amount_usd, transaction_ts. The Silver contract specifies grain (one transaction per txn_id), nullable columns, and a freshness SLA of T+4 hours. A row count check validates that Silver contains within 1% of the expected Bronze volume. A null check validates that account_sk is never null.

Gold: the transaction is aggregated to the account-day grain: daily_account_revenue, transaction_count, avg_transaction_value. The Gold contract specifies that "revenue" means sum(transaction_amount_usd) where transaction_status = 'settled'. This definition is registered in the semantic layer. Every downstream consumer such as a BI tool, an analyst SQL query or an AI agent uses the same definition.

Lineage: an Unity Catalog lineage trace from the Gold metric back to Bronze is available. When a reporting discrepancy surfaces, the trace takes minutes, not days.

This is what governance built into the architecture looks like, as opposed to governance implemented afterward.

Ungoverned data platforms

These are some problems arise in a data platform without governance:

• Definition drift — the same metric calculated differently by different teams, diverging silently over months
• Undocumented PII — personal data in columns that are not tagged, not masked, and not audited; a compliance exposure that nobody knows about until it matters
• Broken lineage — a pipeline refactored six months ago but the lineage graph never updated, so the dependency map is wrong
• Dashboard divergence — two dashboards showing different numbers for the same metric, both technically correct by their own definitions, neither trustworthy
• Small-file proliferation — hundreds of thousands of tiny Parquet files accumulating in a Bronze table with no compaction policy, degrading query performance progressively until someone notices.

Conclusion

Based on past experience, implementing the below items up front simplifies your data platform and saves time and effort down the line:

• Defining the data contract structure before any team writes their first pipeline.
• Agreeing on the naming conventions before any table is created.
• Making column descriptions mandatory at schema definition time.
• Instrumenting lineage from the start.

The hardest part of building a governed lakehouse is not the tooling. It is the organisational discipline to maintain standards as the number of teams, tables, and consumers grows. Platform guardrails help, but they do not substitute for teams that genuinely own their data products.

Governance is an engineering discipline — built into the architecture, maintained by the teams that own the data, and enforced by the platform.

Further reading

For practitioners who want to go deeper on the topics covered in this post:

Books

The Data Lakehouse for Dummies, 2nd Databricks Special Edition by Ari Kaplan and Amit Kara.

Fundamentals of Data Engineering by Joe Reis and Matt Housley.

Data Management at Scale by Piethein Strengholt.

Official documentation and reference material Delta Lake Documentation Unity Catalog Documentation Apache Iceberg Documentation Blog posts and articles

Databricks Engineering Blog: Practitioner posts from the Databricks engineering team on Delta Lake internals, Unity Catalog features, Spark optimisation, and MLOps. The posts on Delta Lake deletion vectors, liquid clustering, and Unity Catalog lineage are particularly relevant to this post’s topics.

Martin Kleppmann — Designing Data-Intensive Applications (selected chapters): Please note, this not a lakehouse book specifically, but Chapter 3 (storage engines) and Chapter 11 (stream processing) provide the systems foundation that makes Delta Lake’s design choices legible. Strongly recommended for anyone working at the storage and streaming layer.

Data Council and Subsurface Conference Talks — Practitioner talks from data engineers at companies operating lakehouses at scale.

Originally published at harishkesavarao.github.io

A Data Scientist’s notebook optimised for exploratory work on a small sample will often need significant rework before it can run reliably at production scale. This reflects a natural difference in priorities between DE and DS: notebooks favour iteration speed, while production pipelines prioritise reliability, scalability, and cost efficiency. Bridging that gap requires both disciplines to have a working understanding of each other’s constraints.

Recently, I partnered with Data Scientists to scale a Semantic Search and Theming application from an exploratory notebook to a production data pipeline. This post describes the journey — what I had to learn, what we had to align on, and how the code evolved from an experimental notebook to a production pipeline.

Learning Data Science concepts

Over the past few years, Data Engineering’s primary use case has evolved from pure analytics — for technical and business decision makers — to a combination of analytics and machine learning model support for Data Scientists and data analysts.

This shift requires a Data Engineer to gain a deep understanding about what they are building. For semantic search and theming specifically, the questions I needed to answer before I could contribute meaningfully were:

• What does semantic search actually do, and how is it different from keyword search?
• What is an embedding, and why does chunking large text documents before embedding matter?
• What is the difference between cosine similarity and dot product search, and what does that mean for how you store and query vectors?
• Which embedding models are appropriate for the kinds of text being processed?

These are not questions a Data Engineer needs to answer at a research level. But understanding them well enough to make architecture decisions — which vector storage to use, how to design a chunking pipeline, what is the right compute type for generating embeddings at scale — is the prerequisite to ensure that the application works as intended for the user count and data volume in production.

A Data Engineer’s job in this collaboration is to educate Data Scientists on the nuances of production data engineering with regards to their use cases, agree on trade-offs and roadmap priorities together, and build the infrastructure that will scale their Search and Theming logic.

Understanding business value

Before any technical work, both Data Engineers and Data Scientists need alignment on three questions:

• Who are the customers, and what are they trying to do with the data?
• What problem are we actually solving — not just technologically, but the business value for the customers?
• What does “good enough” look like for the first version?

Once we had answers to the above questions, we used it to formulate a problem statement. A closer look at the problem statement often reveals that it can be decomposed into smaller, prioritised use cases with clearer success criteria. Getting frequent and early feedback from both Data Scientists and business users — before the pipeline is complete — prevents the risk of building something that no one might be using in the future.

Additionally, it is also possible that there are other users or teams trying to solve the same set of problems (technologically) or to unlock the same kinds of business value as us. Identifying such users or teams, collaborating with them to learn and share lessons and also to reduce redundancies and overlaps to promote reusability is a valuable exercise for an organization.

For semantic search and theming, the business question was: can a user describe what they are looking for in natural language and get relevant and accurate results back, without needing to know the exact terminology used in the underlying data?

What the Data Scientist needed vs. what production required

Features and scalability

The standard pipeline structure

For any production data pipeline — whether analytics or ML — these considerations are relevant:

1. Ingest data at an agreed latency (batch, micro-batch, or streaming)
2. Transform data to meet the use case’s needs
3. Load into appropriate storage with read/write optimisation
4. Validate data quality at multiple points in the pipeline

For ML pipelines, a fifth step applies: serve — making the output (embeddings, features, model outputs) available to downstream consumers reliably and at low latency.

The notebook-to-pipeline gap

When a Data Scientist’s notebook becomes a production workload, the following changes are almost always required:

pandas → Spark

The most common refactor. pandas operates on a single node. As data volume scales, this becomes a compute bottleneck. The refactor requires replacing pandas operations with Spark DataFrame equivalents and registering Python functions as UDFs for operations that cannot be expressed natively.

Notebook-style, single-node:

import pandas as pd

df = pd.read_csv("dataset.csv")
results = df[df["category"] == "billing"]["description"].apply(embed_text)
print(results.head())p

Production-ready, Spark:

from pyspark.sql import functions as F
from pyspark.sql.types import ArrayType, FloatType

@udf(returnType=ArrayType(FloatType()))
def embed_text_udf(text: str):
    return embed_text(text)

df = (
    spark.table("dataset")
    .filter(F.col("category") == "billing")
    .withColumn("embedding", embed_text_udf(F.col("description")))
)

# Write directly — never collect() in production
df.write.format("delta").mode("overwrite").save("/mnt/embeddings/billing")

Two things to note in the production version: the UDF wraps the embedding function so it runs distributed across the cluster, and the result is written directly to Delta Lake rather than collected to the driver. Collecting a large embedding dataset to the Spark driver is one of the most common causes of out-of-memory failures in production.

Removing or converting print/display/show statements

Every print(), display(), or show() call in a Spark context collects data to the driver. In a notebook, this is fine — the dataset is small. In production, it is expensive and in some cases causes driver OOM errors.

Replace with logging:

import logging

logger = logging.getLogger(__name__)

# Instead of: print(df.count())
row_count = df.count()
logger.info(f"Processed {row_count} records in embedding pipeline")

# Instead of: df.show()
# Log a pre-calculated summary, not the raw data
logger.info(f"Embedding pipeline complete. Sample schema: {df.schema}")

Logging statements should display pre-calculated results or perform minimal actions on DataFrames. Avoid triggering new Spark actions inside logging calls.

Data chunking

As data volumes grow, processing everything in a single run becomes impractical — both for compute cost and for pipeline resumability. Chunk by date or another appropriate attribute:

from datetime import date, timedelta

def get_date_chunks(start_date: date, end_date: date, chunk_days: int = 7):
    current = start_date
    while current < end_date:
        chunk_end = min(current + timedelta(days=chunk_days), end_date)
        yield current, chunk_end
        current = chunk_end

for chunk_start, chunk_end in get_date_chunks(start_date, end_date):
    chunk_df = (
        spark.table("dataset")
        .filter(
            (F.col("created_date") >= chunk_start) &
            (F.col("created_date") < chunk_end)
        )
    )
    process_and_write(chunk_df, chunk_start, chunk_end)

Chunking also makes pipeline failures recoverable — a failed chunk can be reprocessed without rerunning the entire historical load.

Statistics and profiling

Data profiling operations — row counts, null checks, distribution summaries — are useful during development but expensive at scale. Make them optional or conditional:

ENABLE_PROFILING = False  # set via config or environment variable

if ENABLE_PROFILING:
    logger.info(f"Null count in description: {df.filter(F.col('description').isNull()).count()}")
    logger.info(f"Category distribution: {df.groupBy('category').count().collect()}")

API reliability

ML pipelines typically call external APIs — embedding model endpoints, LLM APIs, enrichment services. Notebooks call these once per cell and fail fast. Production pipelines need to handle transient failures gracefully.

Implement retries with exponential backoff and explicit handling for rate limit responses:

import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def get_session_with_retries(
    retries: int = 3,
    backoff_factor: float = 2.0,
    statuses: tuple = (429, 500, 502, 503, 504),
) -> requests.Session:
    session = requests.Session()
    retry = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=statuses,
        respect_retry_after_header=True,  # honours 429 Retry-After headers
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount("https://", adapter)
    return session


def embedding_api(text: str, session: requests.Session) -> list[float]:
    response = session.post(
        "https://your-embedding-endpoint/embed",
        json={"text": text},
        timeout=30,
    )
    response.raise_for_status()
    result = response.json()

    # Validate result structure before returning
    if "embedding" not in result or not isinstance(result["embedding"], list):
        raise ValueError(f"Unexpected API response structure: {result.keys()}")

    return result["embedding"]

Two details worth calling out: respect_retry_after_header=True honours the Retry-After header that rate-limited APIs return (common with OpenAI, Anthropic, and most embedding endpoints), and the result validation after raise_for_status() catches cases where the API returns 200 with malformed output — which happens more often than it should in production.

The pipeline architecture

The journey from notebook to production pipeline follows a consistent path:

The collaboration between Data Engineers and Data Scientists is iterative — not a single handoff. Data Scientists validate that the refactored pipeline produces the same results as the notebook. Data Engineers validate that the pipeline meets production reliability and cost requirements. Several iterations are normal before both criteria are satisfied simultaneously.

Conclusion

The hardest part of productionising a Data Science application is not the technical refactoring — it is the alignment on what “production-ready” means between the two disciplines that have different definitions of the term.

For a Data Scientist, production-ready means the model works correctly. For a Data Engineer, production-ready means the pipeline runs reliably at scale, fails gracefully, and can be maintained by someone other than the person who built it. Both definitions are correct. Getting to a version that satisfies both requires trade-offs and some rounds of discussions — the table in this post is a useful starting point for that conversation.

Data Engineers and Data Scientists solve complementary problems. The notebook provides the logic and the validated approach. The production pipeline provides the infrastructure that makes that approach work at scale. Neither is complete without the other.

Looking ahead…

Every new year gives us an opportunity for a fresh, promising perspective to existing aspects of our life. Career, personal goals, habit building and the world around us overall.

I think I can sum up my 2015 with the words — grateful, enduring, thought-provoking, maturing.

While 2015 made great strides for technology, what lies ahead of us in 2016 and beyond is more exciting than ever.

Space travel

SpaceX made a milestone success of getting a launched spacecraft back to the Earth. That, in itself, drastically reduces the cost to put man and material in space. I will not be surprised that space tourist travel will become mainstream before the middle of this century. More than making space travel possible, learning more about how humans adapt to it physically and psychologically is key. Making space travel safer and reliable is extremely important for it to develop as an industry and for people to spend money on it.

It took a better part of a century for us to master air travel. Space travel may not take that long, but we are, for the first time, looking beyond the Earth. That is literally unchartered territory. It will be interesting to wait and see what unfolds in 2016 and beyond.

Technology

Drones have become more commonplace than ever before. I saw a newly constructed office space using drones to create a timeline video of the entire construction process using drones. Drones saved lives by surveying inaccessible locations, during floods and other disasters.

Hover-boards are interesting too. Once the manufacturers sort the explosion/fire problem with reasonable safety for the devices, I think they will replace or compliment segways and help people having a lot of walking to do as part of their jobs — such as warehouses or parks.

Connected living

An aisle at Best Buy features a host of gadgets to control the home with a smartphone/tablet. This, I would think, is just the beginning of connecting every conceivable electronic item in your home to the smartphone. Life will certainly be different with the home constantly surrounded by wi-fi enabled devices. We still have a long way to go before mainstream adoption. Time will tell.

Nature

2015, especially, the second half, proved to be very different from others for most parts of the world in terms of weather. Unprecedented warm temperatures, snowfall, rain, floods, heat waves etc. shattered weather records in multiple continents.

We did see a significant step towards combating climate change with the agreement in France this year. Pollution in some of the most densely populated cities in the world is one reason to cut down on carbon emissions, even with ongoing debate of the impact of humans on climate change. Beijing and New Delhi especially posed a grim picture of how bad pollution can get.

Information

People were able to receive and share information much faster than before, in 2015. Thanks to Facebook and Twitter, my home city was able to get back on its feet, after a devastating and unprecedented flooding and cyclones. Volunteers, relief work, rescue operations etc. were quickly coordinated using Facebook, Whatsapp and Twitter. For the first time, people commented that Facebook was not just for sharing content about the fun times we all have, Facebook saves lives too.

Closing the year 2015, welcoming 2016…

Thinking back about the year 2015, I just like all of you, have a lot to learn from, reflect upon, grow out of and continue what we all have been doing. It may well be just a landmark in a calendar system, but, the New Year can be an opportunity for us to reflect, refine and reform and for us to get better at what we all do.

Good luck and best wishes for a Happy and Joyous 2016!

The English language always fascinates me. More than anything else, I love to read — news, stories from business leaders, successful entrepreneurs, so on and so forth. I have always wanted to write and I did for quite some time in 2008. After that, time got the better of me.

I stumbled upon Medium for reading an article by one of the most renowned names in Carnatic Music, Sanjay Subrahmanyan. I started liking the theme of Medium, the structures, tags and the most important component, the people writing in Medium.

I hope to continue the streak of writing and in the event, interact with the wonderful people here.