What's Change Data Capture?

Imagine you are building a movie-booking system. A user books a seat, and the booking is stored successfully in PostgreSQL. But the booking doesn't matter only to the Booking Service — the Notification Service needs to send a confirmation, the Analytics Service needs to record it.

flowchart TB
  App[Booking<br/>Service]:::svc --> PG[(PostgreSQL<br/>Booking Confirmed)]:::db
  PG --> Notify[Notification<br/>Service]:::svc
  PG --> Analytics[Analytics<br/>Service]:::svc

Let's walk through why each step leads to the next.

Part 1 — The Problem: Detecting Database Changes

Suppose the bookings table contains booking_id = B123, status = PENDING. A user completes payment, and it becomes status = CONFIRMED. Now other systems need to know: Booking B123 was confirmed. There are a few ways to try to solve this — each with a specific failure mode that motivates the next approach.

1.1 Approach: Poll the Database

The first idea is simple: "just keep checking the database."

SELECT *
FROM bookings
WHERE updated_at > :last_checked_time;

flowchart LR
  Notify[Notification<br/>Service]:::svc -->|Anything changed?| App[Booking<br/>Service]:::svc
  Analytics[Analytics<br/>Service]:::svc -->|Anything changed?| App
  App -->|SELECT| PG[(PostgreSQL)]:::db
  PG -.->|No| App

This is called polling, and it has three real problems.

Problem 1 — Unnecessary database load. If nothing changes for 30 seconds, the application still queries every few seconds regardless:

10:00:00 → SELECT → No changes
10:00:05 → SELECT → No changes
10:00:10 → SELECT → No changes
...
10:00:30 → SELECT → 1 change

Now multiply that by every service doing the same polling against the same database, and the database is being repeatedly queried just to discover whether anything changed at all.

Problem 2 — Change tracking becomes complicated. Reliably capturing every change between poll intervals requires careful logic around timestamps, ordering, concurrent updates, retries, clock precision, pagination, and duplicate processing. This only gets harder as the system grows.

Problem 3 — Deletes are difficult. If Booking B123 is deleted with DELETE FROM bookings WHERE booking_id = 'B123', a later SELECT * FROM bookings simply won't include it — there's no row left to discover a change on. You'd need a separate mechanism entirely (an audit table, a tombstone, a soft delete, or a change history table) just to notice deletions.

1.2 Approach: Let the Application Publish the Event

A better idea: the application already knows when it changes something, so why keep checking? On confirming a booking, it can save to PostgreSQL and publish an event to Kafka:

public void confirmBooking(Booking booking) {

    bookingRepository.save(booking);

    kafkaProducer.send(
        "booking-events",
        new BookingConfirmedEvent(booking)
    );
}

flowchart TB
  App[Booking<br/>Service]:::svc -->|write| PG[(PostgreSQL)]:::db
  M3[B125<br/>tail]:::evt
  M2[B124]:::evt
  M1[B123<br/>head]:::evt
  App -->|enqueue| M3

  subgraph Q[Kafka · booking-events]
    direction TB
    M3 -->|next| M2 -->|next| M1
  end

  subgraph Consumers[Consumers]
    direction LR
    Notify[Notification<br/>Service]:::svc
    Analytics[Analytics<br/>Service]:::svc
  end

  M1 -->|dequeue| Consumers

This is far more immediate than polling — but it introduces a different, more subtle problem.

1.3 The Dual-Write Problem

The application is now writing to two different systems, and these writes are not one ordinary local database transaction. Consider saving to PostgreSQL first:

flowchart TB
  App[Booking<br/>Service]:::svc -->|1. Save booking| PG[(PostgreSQL<br/>B123 CONFIRMED)]:::db
  PG -->|2. SUCCESS| App
  M3[B125<br/>tail]:::evt
  M2[B124]:::evt
  M1[B123<br/>head]:::evt
  App -.->|3. Publish fails| M3
  subgraph Q[Kafka · booking-events]
    direction TB
    M3 --> M2 --> M1
  end
  linkStyle 2 stroke:#f87171,stroke-width:1.5px

Now PostgreSQL has B123 = CONFIRMED, but Kafka has no BOOKING_CONFIRMED event at all — the Notification and Analytics services never hear about it.

Reversing the order doesn't fix it either:

flowchart TB
  App[Booking<br/>Service]:::svc -->|1. Publish event| M3
  M3[B125<br/>tail]:::evt
  M2[B124]:::evt
  M1[B123<br/>head]:::evt

  App -.->|2. Save booking · FAIL| PG[(PostgreSQL<br/>nothing stored)]:::db

  subgraph Q[Kafka · booking-events]
    direction TB
    M3 --> M2 --> M1
  end

  linkStyle 1 stroke:#f87171,stroke-width:1.5px

Now Kafka has BOOKING_CONFIRMED, but PostgreSQL never actually stored the booking — a downstream service could send a confirmation for a booking that doesn't exist.

This is the dual-write problem.

1.4 Why Not Put Both in One Transaction?

A natural question: can't we just wrap the PostgreSQL write and the Kafka write in one transaction?

flowchart TB
  App[Booking<br/>Service]:::svc -.->|one atomic commit?| Tx
  subgraph Tx[Distributed Transaction]
    direction LR
    PG[(PostgreSQL)]:::db
    Kafka[Kafka]:::evt
  end

PostgreSQL and Kafka are separate systems — coordinating commits and rollbacks across them requires distributed transaction machinery and adds considerable complexity. There's a simpler path: instead of making PostgreSQL and Kafka participate in one transaction, make the important write happen entirely inside PostgreSQL's own transaction. That's the Outbox Pattern, covered in Part 3 — but first, we need the technology that lets us observe database changes automatically: CDC.

Part 2 — How CDC Works

So far, we saw that polling is inefficient and application → DB + Kafka creates a dual-write problem.

CDC takes a different approach:

Let the database tell us what changed.

Suppose the booking service confirms booking B123:

UPDATE bookings
SET status = 'CONFIRMED'
WHERE booking_id = 'B123';

2.1 PostgreSQL Records the Change

PostgreSQL writes information about database changes to its Write-Ahead Log (WAL). The WAL is not a list of SQL queries. It is low-level information PostgreSQL uses for recovery and replication.

The important idea: The application updates PostgreSQL, and PostgreSQL records the change in its WAL (Write-Ahead Log). Each position in the WAL is identified by an LSN (Log Sequence Number). You can think of an LSN as a position or bookmark in the WAL.

PostgreSQL
    │
    ↓
   WAL
    │
    ├── LSN 100 → UPDATE booking B123
    ├── LSN 120 → INSERT booking B124
    ├── LSN 140 → UPDATE booking B123
    └── LSN 160 → INSERT outbox event

How does a CDC consumer read the WAL?

A CDC consumer cannot simply read PostgreSQL's WAL files directly.

Instead, PostgreSQL provides logical replication for consumers that need to read changes from the WAL.

Each CDC consumer gets its own replication slot. You can think of the slots as a simple map between a consumer and its current position in the WAL:

Consumer        Replication Slot        LSN
   │                   │                 │
Debezium-1        booking_slot        140
Debezium-2        payment_slot        180
Debezium-3        audit_slot          220

The replication slot is a persistent object maintained by PostgreSQL. It tracks the consumer's progress using an LSN (Log Sequence Number).

The important idea is:

The consumer does not directly read the WAL files. PostgreSQL's logical decoding mechanism processes the WAL from the position tracked by the replication slot.

The replication slot allows the consumer to resume from its previous position if it disconnects or crashes.

Now that we know how the consumer gets changes from the WAL, let's see how logical decoding converts those low-level WAL records into meaningful database changes.

2.2 Logical Decoding Makes WAL Useful

The above diagram is a simplified view. PostgreSQL stores database changes in the WAL, but these records are in PostgreSQL's internal, low-level format.

A CDC system needs these changes in a logical form that describes what actually changed:

booking B123
PENDING → CONFIRMED

So, we need a mechanism that can read the WAL and turn these low-level records into meaningful database changes.

That mechanism is called logical decoding.

Resuming after a disconnect is the replication slot's job — §2.4 covers exactly how that works.

sequenceDiagram
    participant App as Application
    participant PG as PostgreSQL
    participant WAL as WAL
    participant Slot as Replication Slot
    participant LD as Logical Decoding
    participant Stream as Logical Replication Stream
    participant Consumer as CDC Consumer<br/>(Debezium)


    App->>PG: UPDATE booking B123
    PG->>WAL: Record database change
    PG->>PG: Commit transaction

    Consumer->>Slot: Connect to replication slot
    Slot-->>Consumer: Consumer position (LSN 140)

    Consumer ->> LD: Read LSN 140
    LD->>WAL: Process WAL from LSN 140
    LD->>LD: Decode WAL using pgoutput
    LD-->>Stream: get logical change
    Stream ->>LD: return logical change

    LD-->>Consumer: booking B123<br/>PENDING → CONFIRMED

2.3 Here Comes Debezium

We need a component that can continuously consume this WAL logs, turn the changes into structured CDC events, and publish them to Kafka.

That is where Debezium comes in.

2.3.1 What is Debezium?

Debezium is an open-source Change Data Capture (CDC) platform. It captures database changes such as INSERT, UPDATE, and DELETE and converts them into structured events.

For PostgreSQL, Debezium reads changes through PostgreSQL's logical replication / logical decoding mechanism.

But there is one more component involved: Kafka Connect.

2.3.2 Where does Debezium run?

In a typical PostgreSQL → Kafka setup, the Debezium PostgreSQL Connector runs inside Kafka Connect.

Kafka Connect is the runtime that runs connectors and moves data between external systems and Kafka.

So the responsibilities are:

The overall architecture looks like this:

flowchart LR
  PG[(PostgreSQL)]:::db
  WAL[WAL]
  LD[Logical Decoding]

  subgraph KC[Kafka Connect]
    DBZ[Debezium<br/>PostgreSQL Connector]
  end

  Kafka[Kafka]:::evt
  Consumer[Downstream Services]:::svc

  PG --> WAL
  WAL --> LD
  LD --> DBZ
  DBZ --> Kafka
  Kafka --> Consumer

So finally we can say, Debezium connects to PostgreSQL's logical replication stream and consumes these changes.

It converts them into structured CDC events:

{
  "operation": "UPDATE",
  "table": "bookings",
  "booking_id": "B123",
  "status": "CONFIRMED"
}

2.4 How Does Debezium Resume?

PostgreSQL uses a replication slot to track the CDC stream position.

The position is represented by an LSN (Log Sequence Number).

Think of it as a bookmark:

WAL
──────────────────────────────→

100   110   120   130   140
                ↑
               LSN

If Debezium restarts, it can continue from the required position instead of starting from the beginning.

The replication slot also tells PostgreSQL which WAL is still needed, so that WAL is retained.

2.5 Finally, Kafka

Debezium can publish the CDC events to Kafka:

flowchart LR
    DB[(PostgreSQL)] --> WAL[WAL]
    WAL --> LD[Logical Decoding]
    LD --> D[Debezium]
    D --> K[Kafka]
    K --> N[Notification]
    K --> A[Analytics]
    K --> C[Cache]

Now one database change can be consumed by many services.

Remember just this:

WAL records the change → logical decoding exposes it → Debezium captures it → Kafka distributes it.

Part 3 — Failure Handling & Operational Concerns

CDC's reliability guarantees only matter once you understand exactly what happens when each piece fails.

3.1 What Happens If Debezium Crashes?

Suppose Debezium has processed through LSN 120 out of a WAL containing 100, 110, 120, 130, 140, and then it crashes. The PostgreSQL replication slot retains the WAL the consumer still needs, so on restart Debezium can continue from LSN 130 onward.

flowchart LR
  PG[(PostgreSQL)]:::db
  WAL[WAL]:::evt
  Deb[Debezium]:::fail

  PG -->|1. WRITE| WAL
  WAL -->|2. READ| Deb
  Deb -.->|3. PUBLISH · FAIL| M3

  subgraph Q[Kafka · booking-events]
    direction TB
    M3[B125<br/>tail]:::evt
    M2[B124]:::evt
    M1[B123<br/>head]:::evt
    M3 --> M2 --> M1
  end

  subgraph L[WAL · Log Sequence Numbers]
    direction LR
    L100["100"] --> L110["110"] --> L120["120"] --> L130["130"] --> L140["140"]
  end

  Deb -.->|Processed through| L120

  linkStyle 1,2 stroke:#ef4444,stroke-width:2px

  classDef fail fill:#fee2e2,stroke:#ef4444,color:#dc2626,stroke-width:2px
  class Deb fail
  class L120 fail

3.2 Can CDC Produce Duplicate Events?

Yes — and this is an important property to design around. If Debezium reads LSN 500, publishes BOOKING_CONFIRMED to Kafka, and then crashes before its latest offset state is safely persisted, it may re-read that same change after restarting. The downstream consumer could then see BOOKING_CONFIRMED twice.

3.3 What If Kafka Is Down?

The database keeps accepting transactions and writing to WAL regardless of Kafka's availability — Debezium just can't successfully publish to Kafka in the meantime:

flowchart LR
  PG[(PostgreSQL)]:::db
  WAL[WAL]:::evt
  Deb[Debezium]:::svc

  PG -->|1. WRITE| WAL
  WAL -->|2. READ| Deb

  Deb -.->|3. Publish event · FAIL| M3

  subgraph Q[Kafka · booking-events]
    direction TB
    M3[B125<br/>tail]:::evt
    M2[B124]:::evt
    M1[B123<br/>head]:::evt
    M3 --> M2 --> M1
  end

  linkStyle 2 stroke:#f87171,stroke-width:1.5px

The replication slot retains the WAL Debezium still needs, so once Kafka comes back, Debezium resumes publishing the accumulated changes:

One operational warning: if a consumer stays down too long, retained WAL keeps growing and can fill PostgreSQL's disk — monitor replication-slot lag.

Part 4 — From Database Rows to Business Events: The Outbox Pattern

4.1 We Have CDC — So Are We Done?

Not quite. CDC solves how do we reliably capture database changes, but there's a second question: what should we actually publish to Kafka? Consider a real booking transaction:

BEGIN;

UPDATE bookings
SET status = 'CONFIRMED';

UPDATE seats
SET status = 'BOOKED';

UPDATE payments
SET status = 'SUCCESS';

COMMIT;

CDC can capture every one of these row-level changes — but a downstream service usually doesn't care about individual updates. It just needs BOOKING_CONFIRMED. That's a business event, and the distinction matters:

Raw CDC gives us database changes; the application understands the business meaning. Combining these two is exactly what the Outbox Pattern does.

4.2 The Outbox Pattern

Recall the dual-write problem: the application wanted to update PostgreSQL and publish to Kafka as two independent writes. The Outbox Pattern changes the design — instead of writing to two systems separately, both the business data and the event are written inside the same PostgreSQL transaction:

BEGIN;

UPDATE bookings
SET status = 'CONFIRMED'
WHERE booking_id = 'B123';

INSERT INTO outbox_events (
    event_id,
    event_type,
    aggregate_id,
    payload
)
VALUES (
    'E100',
    'BOOKING_CONFIRMED',
    'B123',
    '{"bookingId":"B123"}'
);

COMMIT;

Outbox = What business event should be published?
CDC    = How do we reliably move the database change out?
Kafka  = How do we distribute the event to consumers?

4.3 Tracing the WAL for This Transaction

The same transaction from 4.2, as it lands in WAL:

Logical decoding exposes both changes to the replication stream, and Debezium receives them — but we don't want Kafka to receive every raw row change, just the meaningful business event. That's the job of the Outbox Event Router.

4.4 The Outbox Event Router

Debezium can be configured to watch the outbox table specifically and apply its Outbox Event Router transformation:

A typical outbox row carries fields like:

event_id       = E100
aggregate_type = booking
aggregate_id   = B123
event_type     = BOOKING_CONFIRMED
payload        = {"bookingId":"B123"}

The router transforms that database record into the actual Kafka event — so instead of Kafka seeing a raw UPDATE outbox_events, it receives the clean business event BOOKING_CONFIRMED. The event_id doubles as the deduplication key consumers use to detect duplicate delivery. Full option reference: Outbox Event Router.

The second is much closer to what downstream business services actually need.

Part 5 — Putting It All Together

5.1 The Complete Movie Booking Flow

The same journey as a sequence diagram:

sequenceDiagram
  participant User
  participant App as Booking Service
  participant DB as PostgreSQL
  participant Slot as Replication Slot
  participant Deb as Debezium
  participant Kafka

  User->>App: Confirm booking B123

  App->>DB: BEGIN
  App->>DB: UPDATE bookings
  App->>DB: INSERT outbox
  App->>DB: COMMIT

  DB->>DB: Write changes to WAL
  DB->>Slot: WAL available at LSN 0/ABC123

  Deb->>Slot: Read from last acknowledged LSN
  Slot-->>Deb: WAL changes + LSN 0/ABC123

  Deb->>Deb: Decode via pgoutput
  Deb->>Kafka: BOOKING_CONFIRMED

  Deb->>Slot: Acknowledge LSN 0/ABC123
  Slot->>DB: confirmed_flush_lsn = 0/ABC123

5.2 What Problem Does Each Piece Solve?

5.3 How the Architecture Evolved

That is the core of how modern database-to-event pipelines are built.

Frequently Asked Questions

SELECT *
FROM bookings
WHERE updated_at > :last_checked_time;

Application
    ├── Write → PostgreSQL
    └── Publish → Kafka

BEGIN;

UPDATE bookings
SET status = 'CONFIRMED'
WHERE booking_id = 'B123';

INSERT INTO outbox_events (
    event_id,
    event_type,
    aggregate_id,
    payload
)
VALUES (
    'E100',
    'BOOKING_CONFIRMED',
    'B123',
    '{"bookingId":"B123"}'
);

COMMIT;

Outbox → What business event should be stored atomically?
CDC    → How do we reliably capture the database change?
Debezium → Implements the CDC pipeline.
Kafka  → Distributes the resulting events.

WAL
 │
 ↓
Logical Decoding
 │
 ↓
booking B123
PENDING → CONFIRMED

Kafka Connect = Runtime
Debezium      = Connector
Kafka         = Event streaming system

PostgreSQL host (PostgreSQL + WAL)
        ↓
Kafka Connect host (Kafka Connect + Debezium)
        ↓
Kafka

WAL

100 ── 110 ── 120 ── 130 ── 140
              ↑
             LSN

WAL

100 ── 110 ── 120 ── 130 ── 140
              ↑
       Consumer progress

{
  "event_id": "E100",
  "event_type": "BOOKING_CONFIRMED",
  "booking_id": "B123"
}