Assemble a Release Rollback Evidence Pack

Is the handoff ready?

Use these checks before access expands or the first ticket begins.

Name the baseline with an artifact, not a tag

Rollback starts with a known good state. Record the revision, the built artifact digest, the deploy time, the configuration version, and the migration level that produced the current production state. A tag can move; a digest cannot.

A Philippines-based programmer can assemble that record from the pipeline and the registry in an hour. The reviewer should be able to match the running artifact to the record without trusting a dashboard label.

Include the vendor and infrastructure versions that affect behaviour, such as a runtime upgrade or a feature flag default. A code rollback that leaves a new flag default in place is only half a rollback.

Record how the baseline was verified at the time it shipped, not only that it shipped. A revision that passed a smoke check on a quiet Sunday is a weaker baseline than one that ran through a weekday peak, and the pack should say which it was.

Write the pack as a release ticket

The ticket names the release, the change list, the baseline, the target, the operator, the observer, the maintenance window, and the go or no go owner. Ask for the pack in the repository so it versions with the code.

Keep the first pack for one service. A platform wide release note becomes a document nobody reads, while one service pack can be rehearsed and corrected.

State the trigger for using the pack: an error rate, a failed smoke check, a data anomaly, or a customer report. A rollback without a trigger becomes a matter of mood.

List the irreversible steps before the release

Most rollbacks are easy until a data migration runs. Identify every step that drops a column, rewrites a table, sends an external message, charges a card, or changes a file format. Mark whether the application rollback still works after that step.

For each irreversible step, write the forward repair and the decision owner. If the only recovery is a restore, say that plainly and record the acceptable data loss window.

Separate expand and contract migrations. An additive schema change is usually reversible in behaviour, while a destructive change needs its own approval and its own window.

Assign the operator, the observer, and the message

One person runs the commands and one person watches the signals. Both need tested access before the window, including the ability to reach the registry, the deploy tool, and the database console if the plan requires it.

Write the internal update and the customer message in advance, with the owner who approves sending. During an incident, wording should not be invented under pressure.

The programmer can prepare the commands, the checks, and the draft notes. Production execution and the decision to reverse stay with the company owner unless a written policy says otherwise.

Verify recovery with checks that do not need the author

A deploy returning success is not recovery. Run a smoke check on the critical path, a data check on the affected tables, and a queue or job check if the release touched asynchronous work.

Record the expected result for each check before the release. After the rollback, the reviewer compares the observed result with that expectation rather than with a fresh interpretation.

Include one check that would fail if the new code were still running, such as a version endpoint or a response field only the old build returns. That check proves the rollback took effect.

Write the pass condition as a value, not an adjective. A queue that is healthy is not a check, while a queue depth below the pre-release band for ten minutes is. The observer should never have to interpret the result during an incident.

• Version or build endpoint shows the baseline digest.
• Critical user path completes on the old behaviour.
• Affected table count and checksum match the expectation.
• Queue depth returns to the pre-release band.

Rehearse the rollback in a lower environment

Run the pack against a staging copy with the same migration level. Time each step and record where the runbook was wrong. A pack that has never been executed is a hypothesis.

Rehearse the partial failure case: the rollback command succeeds but a smoke check fails. The observer needs a next step, such as restoring a snapshot or disabling a flag, and an owner to approve it.

Note the differences between staging and production, including traffic, data volume, and external dependencies. Those differences belong in the pack as known limits.

Measure time to reverse and checks passed

Measure minutes from the trigger decision to the baseline running, checks passed divided by checks planned, and rollbacks that required a step not in the pack divided by all rollbacks. Report the service and the window.

Track releases with an irreversible step and no rehearsal. That count is the real risk inventory, and it should be reviewed before the next release.

Run a tabletop review quarterly and a live rehearsal before any release with a destructive migration. Update the pack from every real or simulated use.

Close with a pack someone else can execute

The handoff contains the baseline record, the change list, the irreversible steps, the commands, the checks, the operator and observer, the messages, and the rehearsal notes. It also records the known differences from production.

Accept the work when a reviewer who did not write the code can follow the pack in staging and reach the baseline with every check accounted for.

Add the contact list for the release window, including the database owner and the vendor support line. A pack without a name to call is a document, not a plan.

Frequently asked questions

Sources

1. SLSA specification v1.0

   Artifact provenance and build integrity concepts behind the baseline record.

2. NIST Secure Software Development Framework

   Release, environment, and response practices for software changes.

3. GitHub pull request reviews

   Review and approval record for the release change list.

4. OpenTelemetry documentation

   Signals used in smoke checks and recovery verification.

5. NIST Cybersecurity Framework 2.0

   Recovery planning and communication framing.