Rehearse PostgreSQL logical-replication slot recovery before delegating a connector change

A stopped connector can keep filling the publisher disk

A logical replication slot remembers how far a consumer has confirmed. When the consumer stops, PostgreSQL may retain write-ahead log that the slot could still need. Inventory slot name, database, plugin, active process, confirmed position, subscriber, and owner before changing connector code. A generated slot with no accountable subscriber is an operational liability, not a harmless configuration record. Use a disposable publisher and subscriber for the rehearsal.

Insert rows with obvious sequence keys, stop consumption cleanly, and continue writing at a controlled rate. Sample slot positions, retained bytes, filesystem use, and time. Resume before the test volume is threatened, then confirm retained WAL falls and the subscriber receives the final row. This produces an observed growth rate for the fixture. It also proves the monitoring query and recovery action work together.

Crash after the downstream effect but before a durable checkpoint

Arrange for a change to reach the synthetic destination, then terminate the connector before it confirms the position. Restart from the recorded slot. The same change may arrive again. Save the position, destination state, connector log, and result after replay. Sequential happy-path tests cannot expose this boundary because they never separate the effect from the acknowledgement.

Choose a business identity at the destination and test its duplicate behavior. An upsert may be safe for a current row image but wrong for a counter, audit event, or notification. If the destination cannot apply the replay safely, add a deduplication record or a reconciliation state. Do not describe delivery as exactly once unless a transaction spanning source position and destination effect proves that claim.

Let schema compatibility fail in a controlled place

While the subscriber is behind, add a nullable column, then test a renamed or removed field in the connector’s expected shape. Change a value beyond the destination type assumption. Record what the publication emits, what decoding produces, where processing stops, and whether an updated mapping resumes from the same position. Row replication does not make every DDL change compatible.

Plan publisher and connector releases in an order that supports overlap. A tolerant consumer may accept old and new shapes before the source changes. A destructive source migration may require a pause and a verified checkpoint. Data owners choose the compatibility window and loss tradeoff. The delegated engineer should present the observations and an executable rollback point.

Slot removal needs evidence because it releases history

Dropping a slot can free retained WAL and remove the old restart point. Practice only with the disposable slot. Before removal, record active status, application ownership, last confirmed position, retained bytes, and the subscriber’s replacement plan. Match the exact slot name. A prefix search is too risky for an irreversible operation. Confirm storage recovery afterward and show that the retired subscriber cannot simply resume.

Deliver creation commands, versions, the crash trace, backlog measurements, duplicate outcome, schema cases, alert thresholds, resume procedure, and slot-removal checklist. Database operators retain production creation, deletion, storage, and release authority. The programmer can implement metrics and automation around those decisions, but should not turn a connector code ticket into permission to discard replication state.

Prove the slot can be operated during an imperfect incident

Run the recovery while the publisher remains busy. Establish a baseline change rate, disconnect the subscriber, and generate a controlled mixture of inserts, updates, deletes, and a transaction large enough to span several reads. Record the slot restart and confirmed positions alongside retained bytes and disk headroom. Alert on both the absolute retained volume and the time until a tested limit would be reached at the observed rate. Reconnect with an intentionally slow downstream and show whether it gains ground. Then restart the connector during that catch-up. The evidence should reveal whether monitoring reports an inactive slot, a stalled active consumer, and a consumer that is healthy but slower than the publisher. Those states require different responses. Confirm that connection pooling or failover does not start a second consumer with the same identity. If the product supports high availability, promote only inside a disposable topology and document what happens to slot state and connector offsets rather than assuming they move together.

Exercise data meaning as well as transport. Change a table covered by publication, add a nullable column, alter replica identity in the fixture, and send a transaction containing multiple related rows. Verify what the decoding plugin emits and whether the destination applies the transaction atomically under the intended contract. Force a downstream timeout after its write succeeds, then replay from the last durable position. The destination needs a stable event or business key that turns this ambiguity into one result. Inspect tombstones or delete handling, because a connector that only proves inserts can leave stale customer data behind. Finish by abandoning a named test slot and following the documented escalation: identify its subscriber and owner, confirm recovery is unnecessary, capture its final positions, obtain approval, drop it, and verify retained WAL is released. This is a rehearsal, not permission to drop production slots. The handoff should make the irreversible decision conspicuous while giving operators enough measurements to act before disk pressure becomes the incident.

Also test the monitoring and cleanup paths without relying on a developer's terminal history. Restart the publisher, rotate ordinary WAL, and verify dashboards retain enough context to distinguish expected catch-up from indefinite retention. Remove the disposable subscriber without removing its slot, wait for the orphan alert, and follow the ownership lookup. Recreate the subscriber under a different identity and prove it cannot accidentally claim the abandoned slot. Record the least privileges needed to inspect positions, consume changes, and perform approved cleanup. Production credentials, slot deletion, publication changes, and retention limits remain with database operators. The contractor's deliverable is a repeatable drill whose queries, thresholds, replay evidence, and escalation contacts still make sense when the original implementer is unavailable.

Buyer questions

Questions about planning the role