Rehearse Background Job Idempotency With Duplicate Deliveries

Is the handoff ready?

Use these checks before access expands or the first ticket begins.

Name the side effect that must happen exactly once

Idempotency is not a property of a worker in general. It is a property of one side effect: one invoice line, one confirmation email, one uploaded object, one ledger entry. Before the ticket moves, write which effect must occur once, which table records it, and who can accept a duplicate.

A Philippines-based programmer can then test a claim instead of rewriting a queue on faith. If the team cannot name the side effect, the work is not ready. Retry logic that is safe for a log line can be dangerous for a payment.

Write the rehearsal ticket and seed jobs

The ticket names the job type, the payload schema, the queue, the worker image, the database, the retry policy, and the sandbox account. Ask for a seeded job, a forced duplicate, a worker kill at a declared step, and a comparison of the resulting rows.

Seed from a file with fixed identifiers so the reviewer can rerun the same rehearsal. Random identifiers create a new test every time and make a passing result impossible to compare.

Write the expected row counts beside the seed file before the run. A rehearsal without an expected count turns into a discussion about whether the extra row matters.

Bound the first pass to one job type and one worker version. Sequential delivery is the easy case, and it should pass before anyone adds concurrency.

• Seed job with a known identifier and payload hash.
• Duplicate publish of the same job inside one minute.
• Worker termination after the database write and before the acknowledgement.
• Retry after the visibility timeout expires.

Place the guard where the write happens

An in-memory set of processed identifiers disappears when the container restarts. A check followed by an insert is a race. The durable guard is usually a unique constraint on the business key, an upsert with a conditional clause, or a transactional outbox row written in the same transaction as the effect.

Ask the programmer to show the constraint definition and the failing insert, not only the application branch that checks first. If the database cannot enforce uniqueness, say so in the review and treat the worker as best effort.

Record the key scope. A key built from the job identifier protects against duplicate publishes. A key built from the business event protects against two different jobs producing the same effect. Most incidents need the second kind.

Kill the worker at the worst moment

A polite retry test sends the same message twice and watches both succeed quietly. The rehearsal that matters stops the worker between the external call and the acknowledgement, then lets the visibility timeout expire.

The expected result is one effect, one committed transaction, and a visible recovery action. If the effect ran but the acknowledgement did not, the retry must detect the prior write rather than repeat it.

• Kill after commit, before ack: retry finds the existing row and exits cleanly.
• Kill before commit: retry performs the write once.
• Kill during an external call: the worker records a pending state with a reconciliation owner.
• Two workers on the same job: one wins the constraint, the other records a no-op.

Separate sandbox access from production replay

The programmer needs a named sandbox account, queue publish and consume rights on a test queue, read access to the worker logs, and read access to the test database. They do not need production queue purge, dead letter replay, or secret rotation.

Production replay is a company decision with an incident reference. If a rehearsal needs real messages, copy a sanitized shape into the sandbox instead. A duplicate charge created to prove a point is not a test result.

Attach rows, timestamps, and acknowledgement states

Evidence for this rehearsal is a small table: attempt number, publish time, consume time, commit time, acknowledgement time, rows written, and the observed duplicate outcome. Add the worker log line that shows which branch ran.

Ask for the query used to count effects and the count before and after. A reviewer who can rerun the seed and the query has a reproducible result, while a paragraph describing what happened does not survive a staff change.

Note what the sandbox cannot reproduce, such as managed queue behaviour, clock drift, or production isolation levels. Those gaps belong in the handoff and in the risk list.

Review cadence and honest measures

Measure duplicate effects per one thousand accepted jobs, reconciled jobs per one hundred failures, and recovery time from kill to clean retry. Report the denominator and the observation window beside every number.

Run the rehearsal before each change to retry policy, worker image, or key scope, then quarterly on a stable job. A rehearsal that has not run since the last queue upgrade describes an older system.

Report the rehearsal window and the job volume beside the duplicate rate. One duplicate across a small batch and one across a very large batch are not the same signal, and the reader needs the denominator to tell them apart.

Attribute waiting time separately: reviewer queue, sandbox availability, vendor limit, and owner approval. Do not turn a blocked environment into a programmer performance signal.

Close with a rerunnable rehearsal record

The final record holds the seed file revision, the constraint definition, the kill points, the effect counts, the reconciliation owner, and the next review date. It also states which retry behaviours remain unproven.

Accept the work when a second engineer can seed the job, kill the worker at the same point, and reach the same counts. Merge approval and any production replay stay with the company owner.

Include the sandbox queue name and the seed command in the record. A handoff that points at a dashboard instead of a command tends to expire when the environment is rebuilt.

Frequently asked questions

Sources

1. NIST Secure Software Development Framework

   Secure development practices covering code integrity and environment control.

2. OpenTelemetry documentation

   Context propagation and span recording used to trace one job across retries.

3. SLSA specification v1.0

   Build provenance concepts that help separate worker versions in a rehearsal.

4. GitHub pull request reviews

   Review mechanics for requesting changes on the guard implementation.