Rung 5 — Batch + streaming (CDC → topics → consumers)

One line: Changes flow out of the database as events; anyone who needs them subscribes instead of asking.

Concept. You leave Rung 4 because "nightly" became too slow for a decision that matters — fraud checks, stock levels across stores, a customer expecting a notification now — and because too many systems are polling the database asking "anything new?". Change data capture (CDC) reads the database's own log and publishes every insert, update, and delete to a topic. Consumers (the warehouse loader, the email service, the inventory sync) subscribe. Batch does not go away; it becomes one consumer among several. The business gains same-day and real-time answers; the engineers gain a decoupled backbone and a whole new class of failure modes.

Signs you are here. Business: real-time dashboards; alerts that fire within minutes; "the warehouse is 15 minutes behind" is a known, acceptable number. Technical: Debezium or a managed CDC; Kafka, Kinesis, or Pub/Sub topics; consumer groups; schema registry; the first duplicate-event bug; the first out-of-order bug; someone owns "the pipeline".

The decision. Do you need streaming at all, or just faster batch (hourly, every 15 minutes)? Faster batch answers most "same-day" needs at a tenth of the operational cost. Managed or self-hosted streaming? Managed (Confluent, MSK, Kinesis, Pub/Sub) unless the team has run Kafka before. Exactly-once or idempotent consumers? Idempotent consumers, always; exactly-once is a marketing term with asterisks. Streaming processing (Flink, ksqlDB) or just streaming transport? Transport first; processing when a question needs windows and joins in flight.

Tools. CDC: Debezium, AWS DMS, Datastream, Fivetran/Airbyte CDC connectors. Transport: Apache Kafka (Confluent Cloud, Amazon MSK, Redpanda), Kinesis, Google Pub/Sub. Processing: Apache Flink, Spark Structured Streaming, ksqlDB, Materialize. Schema management: Confluent Schema Registry, Avro/Protobuf. Orchestration for the batch side: Airflow, Dagster, Prefect.

Next trigger. Toward Rung 6: several source systems feeding topics; consumers need years of history, not a retention window; BI users multiply; ad-hoc questions cost too much on the reporting DB; someone asks for "one place with everything".

Threads. Security: topics carry PII in motion — encrypt, ACL per consumer, mask fields at the producer where possible. Compliance: the event log is a record — GDPR deletion in an immutable log means tombstones and compaction, decide the policy now; HIPAA audit trails come almost for free here if you design for it. Data modeling: event schemas are contracts — version them, never break consumers; model events as facts ("OrderPlaced") not table rows where you can. Cost: brokers run 24/7 whether or not events flow; retention is storage; the first cost blow-up is a consumer re-reading from the beginning of a large topic.

[k3r war-story slot: heartbeat-file noise / the consumer that replayed everything]