Rung 6 — Data lake / warehouse (medallion, Iceberg, MPP)

One line: One governed place for everything, with layers that separate "what arrived" from "what we trust".

Concept. You leave Rung 5 because the reporting database can't hold the history, the sources, or the users. Multiple systems (app, CRM, ads, support, finance) need to be joined; analysts and BI users number in the dozens; ad-hoc questions are expensive and step on each other. The lake/warehouse (today usually a lakehouse: open table formats on object storage, queried by an MPP engine) separates storage from compute and organises data in layers — bronze (raw, as landed), silver (cleaned, conformed), gold (business-ready models). The business gets history, self-serve BI, and a single definition of revenue. The engineers get a platform to run.

Signs you are here. Business: a data team exists; BI is self-serve for many; "single source of truth" is a phrase people use unironically; finance and ops argue with the same numbers. Technical: object storage (S3/GCS/ADLS) with Iceberg/Delta/Hudi tables; a query engine (Snowflake, BigQuery, Databricks, Trino, Redshift); dbt with hundreds of models; Airflow/Dagster DAGs; a data catalog appears; the first 12-hour job timeout.

The decision. Warehouse (Snowflake/BigQuery — simpler, pricier per query) or lakehouse (Iceberg/Delta on object storage + engine — cheaper storage, more to run)? Warehouse when the team is small and SQL-first; lakehouse when volume is large, formats are mixed, or multiple engines need the same tables. Table format: Iceberg is the safe default in 2026 for engine portability; Delta if you are all-in on Databricks; Hudi only with a specific upsert-heavy reason. Medallion or not: yes, but three layers, not seven. Central ownership is correct at this rung — the data team owns the platform and the models.

Tools. Storage: S3, GCS, ADLS. Table formats: Apache Iceberg, Delta Lake, Apache Hudi. Engines: Snowflake, BigQuery, Databricks, Amazon Redshift, Trino / Starburst, DuckDB for the small end. Transformation: dbt, SQLMesh. Orchestration: Airflow, Dagster. Ingestion: Fivetran, Airbyte, AWS Glue, Kafka Connect. Catalog: DataHub, OpenMetadata, Unity Catalog, Glue Catalog. Quality: Great Expectations, Soda, dbt tests.

Next trigger. Toward Rung 7: the central data team is the bottleneck — tickets queue for weeks; domain teams build shadow pipelines; nobody in the data team understands the finance data as well as finance does. Toward Rung 8 (some skip 7): AI/LLM initiatives need governed, semantically-defined answers and the gold layer isn't good enough yet.

Threads. Security: column- and row-level security in the engine; PII tagged in the catalog; separate compute for sensitive workloads; key management for object storage. Compliance: this is where SOC 2, GDPR, and HIPAA controls become platform features — lineage, access logs, retention policies, right-to-be-forgotten as a pipeline. Data modeling: bronze keeps source shape; silver conforms keys and types; gold is dimensional or wide tables per use; Data Vault if many sources and heavy audit needs. Cost: compute-per-query and idle warehouses are the twin killers; separate compute per team so cost has a name; storage is cheap, scanning isn't — partition and cluster.

[k3r war-story slot: the 12-hour Glue timeout / concurrent-write collisions on the lake]