Review Django and Celery task delivery before delegating retry work

Make the transaction race visible

Create a synthetic order inside a Django transaction and let a worker start quickly. Publish directly, force rollback, and record whether the worker saw a missing row or work that should not exist. Repeat with transaction.on_commit and show that rollback produces no task. This demonstrates why a queue call returning successfully does not mean the business state is ready.

Cover nested savepoints and the project’s test transaction behavior. A callback inside a rolled-back savepoint disappears, while a test wrapper may never commit. Use an integration fixture with the real test broker. Document which unit tests isolate business logic and which integration case proves publication timing.

A Celery task ID is not the business operation

Choose a durable key such as synthetic-order-17 confirmation version 1. Store it with the effect. Dispatch two tasks concurrently for the same key and prove only one becomes eligible. Avoid a check followed by an unguarded write. Use a unique constraint or guarded transition where the application owns the state.

Pass the business key to a sandbox provider if it supports idempotency and record its retention behavior. The same task ID does not necessarily cross every retry or manually reconstructed message. A stable operation key also gives support and reconciliation a way to search without exposing the full customer payload.

Close the connection after the provider accepts the request

Configure a fake or sandbox to commit an effect and then close the connection before replying. The worker receives an error while the external result is unknown. Save its state, retry classification, provider record, and final visible effect. Neither “failed” nor “retry now” is truthful until the application queries by key or sends the item to reconciliation.

Terminate the worker before work, during a durable transition, after the provider effect, and before acknowledgement. Restart it and wait through the broker visibility interval. Observe redelivery and duplicate protection. Test the actual acknowledgement setting and pool type; a copied Celery recipe can behave differently from the deployed broker.

Malformed, temporary, stale, and unknown work need different endings

Exercise invalid payload, missing state, dependency outage, rate limit, business deadline exceeded, unknown provider outcome, and programmer error. Record retry delay, final state, alert, and replay method. Backoff protects a dependency but does not authorize sending a message after its business deadline. Measure queue age separately from the duration of the current attempt.

Hand off transaction traces, broker settings, key schema, concurrency result, unknown-outcome case, worker-loss timeline, retry table, reconciliation query, terminal queue owner, and bounded replay command. Application owners approve deadlines and customer effects. Operators retain broker, worker, and production replay controls.

Trace one business operation across transaction, broker, worker, and provider

Assign one synthetic operation key before the database transaction begins and carry it through the outbox row, task arguments, provider request, acknowledgement, and final record. Save timestamps and state transitions at each boundary. Roll back the originating transaction and prove no worker can observe the abandoned work. Commit another transaction, publish it, and terminate the worker after the provider accepts the request but before acknowledgement. On redelivery, the worker should look up the operation key and reconcile rather than blindly repeat the external effect. If the provider supports idempotency keys, verify their retention window and response when the original request is still in flight. If it does not, design a local reconciliation state whose unknown outcome is visible to an operator. Retries need classified causes and bounded schedules; a permission error should not hammer the provider like a transient timeout.

Test concurrency by dispatching the same operation twice and by running two workers against one ready row. Use a database uniqueness constraint or atomic claim that makes the winner observable. Advance the fixture past its useful time before a delayed retry arrives, then show the worker recognizes stale work without undoing a newer choice. Revoke the synthetic recipient or account between scheduling and execution and confirm the worker rechecks current eligibility. Observe prefetch and acknowledgement settings under worker loss, because configuration changes the size of the replay window. The handoff should include the sequence diagram, operation-key rule, transaction trace, broker identifiers, provider sandbox result, duplicate convergence, stale-work outcome, retry categories, alert threshold, and manual reconciliation steps. Operations owns replay and dead-letter decisions; business owners decide whether a specific external effect may be retried. That separation keeps an outsourced implementation from silently converting an uncertain delivery into either a duplicate action or lost work.

Include broker and result-backend failure as separate cases. Pause broker connectivity after the database commit and show how an outbox dispatcher resumes without creating another business operation. Make the result backend unavailable while the provider succeeds; business correctness must not depend on storing a Celery return value. Expire a task before a worker receives it and reconcile its database state rather than leaving a permanent pending label. Test worker shutdown with a long provider timeout and confirm visibility timeout settings do not create an uncontrolled duplicate storm. Dashboards should group attempts under the stable operation key while preserving individual task identifiers for diagnosis. Remove or redact task arguments that contain personal data. These checks turn retry configuration into an observable delivery policy and give operators a safe way to distinguish queued, running, unknown, reconciled, and terminal work.

Run an operational replay from the documented procedure with someone other than the implementer. Start from a dead-lettered synthetic operation, inspect its current business state and provider evidence, choose reconcile or retry, and record the authorization for that choice. The command must target the stable operation key, refuse already completed work, and support a dry run. Confirm alerts clear only after database state reflects the outcome, not merely when a message leaves the queue. Measure queue age and oldest actionable failure during the exercise. A second operator should be able to explain why no duplicate external effect occurred using the retained ledger alone.

Buyer questions

Questions about planning the role