WTF is a composable data product?

Scotty Pate 5 min read

Data products are like sex in high school: everyone is talking about it, everyone assumes everyone else is doing it, and almost nobody is doing it well. (Dan Ariely made this joke about big data in 2013 — a decade on, we've only had to change the noun.) Every vendor deck, product manager, and data executive of the last five years has recited "treat your data like a product". I've been a data developer for fifteen years, so I've seen what actually lives behind that slide: a spaghetti dependency graph that would embarrass its owners, if anyone could figure out who the owners were.

The tell is the word itself. A product implies a boundary: something you can ship, version, own, and point at. A schema in a shared warehouse is not that. A dbt project is not that. A dashboard is definitely not that.

This post is about the unit.

The problem, in one paragraph

In a shared warehouse, every table is reachable, so every dashboard, notebook, and ML job wires itself to whatever layer is convenient. None of those dependencies are declared anywhere. The producer of the data has no idea you started using one of their tables as input to your real-time data application. Your "use case" is a fuzzy region of a spaghetti DAG with no owner, no promises, and no safe way to change. A pipeline doesn't fix this, because a pipeline has no edges: where does it start, where does it end, what does it promise, who is allowed to use it? If you want the full crucifixion — the vendor bill, the DevOps history lesson, the antipattern bingo — that's its own post. Here, let's just fix it.

The unit: a cell

Software solved this exact problem decades ago with the module: private internals, a small public interface, a version number, an owner. A composable data product — we call it a cell — applies that to data. It bundles four things that normally drift apart into one versioned, deployable directory:

  • Logic — the SQL transforms, run in order, producing one atomic snapshot.
  • Output — not "a table you hope is right," but a snapshot that is machine-verified against a declared shape on every build. Declared column missing? Declared grain not actually unique in the data? The build fails. Not a warning, not a test someone forgot to run — a failure.
  • Interface — an explicit export list, like pub in a programming language. It inverts the warehouse default: everything is private unless declared. Internals can churn freely; non-exported tables don't just lack documentation, they have no route at all.
  • Access — who can read the exports, declared in the same file, enforced default-deny at the serving API.

The mental model is a software module for data. Inside: source reads, scratch tables, intermediate state — all private, all owned by one clearly named use case. Outside: a small, versioned public surface that is the written-down requirements. The implicit dependencies the warehouse bakes in become explicit declarations a producer can see and refuse.

Why "composable"

Because cells have explicit, versioned interfaces, they build on each other the way code libraries do. A cell declares another cell's export as a source, by name and version, without a central team brokering the handoff. A breaking change ships as a new major version served alongside the old one — orders@1 and orders@2 at the same time — so consumers migrate on their own clock, not yours. You get a graph of small products, each with a stable public surface, private guts, and an owner, instead of one warehouse where everything depends on everything and nobody knows how.

Compatible with your stack, not a competitor

Here's the part people get wrong: a composable data product does not ask you to rip anything out.

If you run a warehouse, keep it. It's genuinely good at what it was built for — one shared model, operated by one team — and a cell treats it as one input among others: a cell can read warehouse tables, lake files, and other cells' exports in the same breath. Your ingestion vendor keeps landing data. Your dbt project keeps building the shared model. What changes is what happens at the edge, where a specific use case meets its consumers. Instead of pointing a dashboard at a mart table and praying, you draw a boundary: these transforms, this contract, this versioned API, this owner.

It's not a semantic layer either. A graph of independently-authored contracts is not automatically one coherent data model. It's a serving boundary. The warehouse answers "what does the organization know?" A cell answers "what does this use case promise, to whom, at what version?" Those are different questions, and you can answer the second one without re-litigating the first. For the use cases that never needed the central stack at all, a cell is the escape hatch: it runs perfectly well against Parquet on object storage with no warehouse in sight.

What this looks like in practice

Datamake is an open-source build system for exactly this unit. A cell is plain SQL plus one file:

cell: orders

transforms:
  - sql/stg_orders.sql
  - sql/orders_daily.sql

interface:
  - name: orders_daily
    version: 2.1.0
    grain: [order_date, region]
    schema: { order_date: date, region: string, revenue: decimal }

Three commands take that from nothing to a versioned, contract-checked REST API:

$ datamk init orders && cd orders
$ datamk run    # atomic snapshot, verified against the interface
$ datamk serve  # REST + OpenAPI on :8080

It's one Rust binary with an embedded DuckDB engine. Storage is Parquet, the catalog is DuckLake — open formats end to end, so your data is never held hostage. No cluster to stand up, no orchestration platform, no procurement cycle. Apache-2.0, zero dollars, and the whole thing runs on your laptop before lunch.

That's the unit. Not a platform, not a mesh, not a philosophy — a directory you can build, verify, version, and serve, that plugs into the stack you already have.

For the deeper reasoning — diagrams included — read the concept doc. Or skip the reading and run datamk init.

← All posts