Datamake blog

The modern data stack is fundamentally broken

2026-07-30T00:00:00+00:00

Here's a fun exercise. Count the purchase orders between "we have data" and "someone answered a question with it" at a company running the modern data stack.

You need an ingestion vendor to copy data out of your SaaS tools. You need a cloud warehouse to put it in. You need a hosted transformation platform to turn the raw copies into something usable, and an orchestrator to run it all on a schedule. That's four vendors, four procurement cycles, four onboarding calls with a customer success manager — each one starting at five figures a year, and that's the entry price, before usage-based billing finds its stride.

And here's the part that should make you angry: at the end of all that, you still haven't shipped anything. You've built a capability. The stack is up, the DAG is green, the invoices are recurring — and the actual thing someone wanted, a trustworthy answer served to a dashboard or an app, is still a ticket in the central data team's queue.

Worse: every problem you had before the stack, you still have. Nobody agrees on what "revenue" means. The dashboard broke because someone renamed a column. The question that started this whole journey is three sprints deep in a backlog. You didn't buy solutions to those problems. You only bought a more expensive place for them to live.

Then the second wave of vendors shows up to sell you solutions to the problems the first wave created. A data observability tool, because nobody can tell when a pipeline silently breaks. A data catalog, because nobody knows what any of the 4,000 tables are for. A reverse-ETL tool, because after paying to move data into the warehouse you now need to pay to move it back out. Each of these is a real product solving a real problem, but the problem is downstream of an architectural choice nobody remembers making.

Everything software learned not to do, sold back to you as architecture</h2>
Twenty years ago, software worked the way data works now. There was a central IT team. You filed a ticket to get a server. Applications were monoliths deployed by an ops group that didn't write them, on a schedule measured in quarters, and the people who understood the code and the people who ran it in production were different people who communicated by blame.
Software didn't fix that by buying better ticketing systems. It changed the shape of ownership. Monoliths broke into modules and services with explicit interfaces. "You build it, you run it" replaced the wall between dev and ops. The central team stopped being a gatekeeper that did everyone's work and became a platform team that paved roads for others to drive on. Nobody walked it back, because it worked.
Data is the last department still running the old playbook — and the modern data stack didn't retire it, it renovated it. A central team, a shared monolith, a ticket queue, and a release process everyone routes around. And the monolith at the center is worse than an organizational throwback; it's a technical one. Look at what the warehouse actually is, in software terms: one global namespace of shared mutable state, where every consumer couples directly to whatever internals are convenient, integration happens by reading someone else's tables, and changes ship with no version bump. Global state. Depending on internals. Integration through a shared database. Breaking changes without versions. Every one of those is a named antipattern that fails code review in software. Stacked together, in data, they're the reference architecture.
So the breakage isn't bad luck, it's destiny. In the warehouse, every table is reachable, so every dashboard, notebook, and ML job wires itself to whatever layer is convenient — staging, intermediate, mart — and none of those dependencies are declared anywhere. Your "use case" is a fuzzy region of a spaghetti DAG. No table can tell you which use case it exists for, what it promises, or whether it's safe to delete. Every column you ship might be load-bearing for a dashboard you've never heard of, so when someone renames it, there's no deprecation notice — it just breaks on a Tuesday, and the postmortem is a Slack thread. Ownership goes diffuse, every change starts with archaeology, and development slows to the speed of fear, with a central team holding it together through tribal knowledge and human toil. That's what the five figures × N vendors actually buys: the organizational pattern software spent two decades escaping, implemented on top of the technical patterns software spent two decades banning, with a subscription ring of tools to make living inside it more comfortable.
A pipeline doesn't fix this, because a pipeline has no edges. Where does it start? Where does it end? What does it promise? Who is allowed to use it? Those answers live in someone's head, a wiki page, and three Slack threads.
The way out is not another vendor</h2>
If the diagnosis is "the monolith has no boundaries," the fix isn't a fifth subscription that monitors the boundaries you don't have. The fix is the same one software found: a unit of ownership with private internals, an explicit versioned interface, and a name on it. Data has that unit now — it's called a composable data product, and we wrote up exactly what it is</a>.
The part worth stressing: it's not a rip-and-replace. A composable data product is compatible with the modern data stack, not a competitor to it. Your warehouse doesn't disappear; it becomes one input among others. The stack keeps doing what it's genuinely good at — one shared model, operated by one team — and the use cases that never needed all that get edges, contracts, and owners instead of a spot in the queue.
Software made this turn twenty years ago and never looked back. Data's turn is overdue.

WTF is a composable data product?

2026-07-24T00:00:00+00:00

Data products are like sex in high school: everyone is talking about it, everyone assumes everyone else is doing it, and almost nobody is doing it well. (Dan Ariely made this joke</a> about big data in 2013 — a decade on, we've only had to change the noun.) Every vendor deck, product manager, and data executive of the last five years has recited "treat your data like a product". I've been a data developer for fifteen years, so I've seen what actually lives behind that slide: a spaghetti dependency graph that would embarrass its owners, if anyone could figure out who the owners were.
The tell is the word itself. A product implies a boundary: something you can ship, version, own, and point at. A schema in a shared warehouse is not that. A dbt project is not that. A dashboard is definitely not that.
This post is about the unit.
The problem, in one paragraph</h2> In a shared warehouse, every table is reachable, so every dashboard, notebook, and ML job wires itself to whatever layer is convenient. None of those dependencies are declared anywhere. The producer of the data has no idea you started using one of their tables as input to your real-time data application. Your "use case" is a fuzzy region of a spaghetti DAG with no owner, no promises, and no safe way to change. A pipeline doesn't fix this, because a pipeline has no edges: where does it start, where does it end, what does it promise, who is allowed to use it? If you want the full crucifixion — the vendor bill, the DevOps history lesson, the antipattern bingo — that's its own post</a>. Here, let's just fix it.
The unit: a cell</h2>
Software solved this exact problem decades ago with the module: private internals, a small public interface, a version number, an owner. A composable data product — we call it a cell — applies that to data. It bundles four things that normally drift apart into one versioned, deployable directory:

Logic — the SQL transforms, run in order, producing one atomic snapshot.</li>
Output — not "a table you hope is right," but a snapshot that is machine-verified against a declared shape on every build. Declared column missing? Declared grain not actually unique in the data? The build fails. Not a warning, not a test someone forgot to run — a failure.</li>
Interface — an explicit export list, like pub</code> in a programming language. It inverts the warehouse default: everything is private unless declared. Internals can churn freely; non-exported tables don't just lack documentation, they have no route at all.</li>
Access — who can read the exports, declared in the same file, enforced default-deny at the serving API.</li> </ul> The mental model is a software module for data. Inside: source reads, scratch tables, intermediate state — all private, all owned by one clearly named use case. Outside: a small, versioned public surface that is the written-down requirements. The implicit dependencies the warehouse bakes in become explicit declarations a producer can see and refuse.Why "composable"</h2> Because cells have explicit, versioned interfaces, they build on each other the way code libraries do. A cell declares another cell's export as a source, by name and version, without a central team brokering the handoff. A breaking change ships as a new major version served alongside the old one — orders@1</code> and orders@2</code> at the same time — so consumers migrate on their own clock, not yours. You get a graph of small products, each with a stable public surface, private guts, and an owner, instead of one warehouse where everything depends on everything and nobody knows how. Compatible with your stack, not a competitor</h2> Here's the part people get wrong: a composable data product does not ask you to rip anything out. If you run a warehouse, keep it. It's genuinely good at what it was built for — one shared model, operated by one team — and a cell treats it as one input among others: a cell can read warehouse tables, lake files, and other cells' exports in the same breath. Your ingestion vendor keeps landing data. Your dbt project keeps building the shared model. What changes is what happens at the edge, where a specific use case meets its consumers. Instead of pointing a dashboard at a mart table and praying, you draw a boundary: these transforms, this contract, this versioned API, this owner. It's not a semantic layer either. A graph of independently-authored contracts is not automatically one coherent data model. It's a serving boundary. The warehouse answers "what does the organization know?" A cell answers "what does this use case promise, to whom, at what version?" Those are different questions, and you can answer the second one without re-litigating the first. For the use cases that never needed the central stack at all, a cell is the escape hatch: it runs perfectly well against Parquet on object storage with no warehouse in sight. What this looks like in practice</h2> Datamake</a> is an open-source build system for exactly this unit. A cell is plain SQL plus one file: cell: orders transforms: - sql/stg_orders.sql - sql/orders_daily.sql interface: - name: orders_daily version: 2.1.0 grain: [order_date, region] schema: { order_date: date, region: string, revenue: decimal }</code></pre> Three commands take that from nothing to a versioned, contract-checked REST API: $ datamk init orders && cd orders $ datamk run # atomic snapshot, verified against the interface $ datamk serve # REST + OpenAPI on :8080</code></pre> It's one Rust binary with an embedded DuckDB engine. Storage is Parquet, the catalog is DuckLake — open formats end to end, so your data is never held hostage. No cluster to stand up, no orchestration platform, no procurement cycle. Apache-2.0, zero dollars, and the whole thing runs on your laptop before lunch. That's the unit. Not a platform, not a mesh, not a philosophy — a directory you can build, verify, version, and serve, that plugs into the stack you already have. For the deeper reasoning — diagrams included — read the concept doc</a>. Or skip the reading and run datamk init</code>.