Architect BeaconSign in

The ladderRung 7 of 8outline, full version in progress

Rung 7 — Data mesh (domain-owned stores, central metadata/catalog)

One line: Domains own their data as products; the centre owns the standards, the catalog, and the platform — not the pipelines.

Concept. You leave Rung 6 because the central data team became the bottleneck and the least-informed owner of every domain's data at once. Data mesh flips ownership: finance owns finance data products, orders owns orders, each publishing governed, documented, quality-checked datasets on a shared self-serve platform. The centre keeps the catalog, the standards, the identity model, and the platform tooling — a federated governance model. This is an organisational change with a technical footprint, not the reverse. It is also the rung most often adopted for the wrong reason (fashion) and abandoned for the right one (no domain teams to own anything).

Signs you are here. Business: domain teams have data engineers or analytics engineers embedded; a "data product" has an owner, an SLA, and consumers who can find it; the central team's backlog shrank. Technical: multiple domain-owned lake/warehouse areas or accounts; a central catalog with ownership and contracts; data contracts enforced at publish time; a platform team shipping templates; cross-domain joins happen through published products, not shared raw tables.

The decision. Mesh or stay centralized? Centralized wins when there are fewer than roughly four domains with real teams, or when the org has no appetite to fund data-literate people in each domain. Mesh wins when the central team is demonstrably the bottleneck and the domains can staff ownership. Physical separation (accounts, projects) or logical (schemas, catalogs)? Logical first — cheaper, reversible; physical when compliance or cost attribution demands it. How much governance is federated versus central? Central: identity, PII classification, naming, contracts. Federated: models, quality thresholds, refresh cadence.

Tools. Catalog and governance: DataHub, OpenMetadata, Collibra, Atlan, Unity Catalog. Data contracts: Soda, Great Expectations, contract specs in the repo. Platform templates: Terraform modules, dbt packages, Dagster/Airflow project templates. Cross-domain query: Trino/Starburst, Snowflake data sharing, BigQuery Analytics Hub, Iceberg REST catalogs. Access: Okta/Entra groups mapped to domain roles; policy engines (OPA, Immuta, Privacera).

Next trigger. Toward Rung 8: AI agents and LLM applications need answers that are consistent across domains — one definition of "customer", one of "active", one of "margin" — and the mesh's products disagree at the edges; the business wants questions answered in natural language with trustworthy numbers.

Threads. Security: per-domain ownership of access decisions with central identity; PII policy enforced by the platform, not by each team; audit across domains. Compliance: domains become accountable for their regulated data — clear for GDPR controllers, harder for SOC 2 evidence collection unless the platform automates it. Data modeling: domain models published as products with schemas as contracts; shared conformed dimensions (customer, product, time) governed centrally; ubiquitous-language boundaries from domain-driven design. Cost: attributable per domain by design; the new waste is duplication — three domains each building their own customer table.

[k3r war-story slot: the mesh that was really three teams and a wiki]

Kept on this device. Sign in to keep it everywhere.
Review my stack against this rung