On this page
Research context
This protocol asks how complete the rollback evidence is for a database migration before release. The unit is one migration release, identified by the migration version and the target environment. The study measures whether the rollback plan exists, whether it was tested, and whether the backup and data-loss consequences are recorded. It does not evaluate whether the migration itself was necessary or well designed.
Set the scope to one service, one database engine, and one release process. Record the migration tool, the backup policy, and the approval roles at the start. If the release process changes during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A new approval step can look like better evidence when it is only a new gate.
Define the population as every migration that reached a production release decision during the window, including migrations that were postponed. The unit is the migration release, not the individual script, because a release can contain several ordered scripts. A migration that is reapplied after a rollback is a new unit with a link to the earlier release. A separate release for a different service is a new unit.
Record the service, the migration type, the data volume, and the release window. These strata matter because a schema-only change and a data transformation carry different rollback risk. Include migrations that were released without a rollback plan, because the base rate is part of the finding. Test the unit definition on two ordinary releases before collection begins.
Define a complete rollback plan as one that names the trigger, the commands, the backup, the expected duration, and the owner. Define a tested rollback as one that was executed in a staging copy with production-like data and a recorded result. Define a destructive change as one that removes or rewrites data in a way that cannot be reversed without the backup. Define a rollback gap as a missing element of the plan.
Do not count a down script as a tested rollback. Do not count a backup that has never been restored as verified. A rollback that requires a manual step by a person who is not available is not complete. Each definition should include one positive example and one boundary example so a second reviewer can apply it consistently.
For each migration, capture the version, the service, the environment, the migration type, and the release date. Add the rollback plan reference, the backup time, the restore test result, the expected duration, and the named owner. Record whether the plan was tested and whether the data-loss consequence was documented. Store the packet with the migration release.
Practical application
Also capture the application version that depends on the migration and the deployment order. These fields help explain a rollback that would fail because the application still expects the new schema. If the backup is stored off-site, record the location and the retrieval time. A second reviewer should be able to reconstruct the release and the rollback path from the record. Store the packet with the migration release.
Draw the sample from the migration system and the release log rather than from a list of incidents. Record the export date, the filters, the row count, and the selection rule. Include migrations from every service and every migration type so the result is not dominated by one team. If the population is ordered by release date, a fixed interval spreads the sample across the window.
Collect the rollback plan as it existed at the release decision, not as it looks after later edits. If the plan was changed after an incident, record the change time and analyze the original separately. Do not infer a tested rollback from the absence of an incident. Note the collection date separately from the release date.
A useful additional measure is the number of migrations that required an unplanned manual step during release. A manual step is not always a failure, but an unrecorded one is a gap. Record the step, the person, the time, and the system where the manual change was applied. That evidence helps the owner decide whether the pipeline, the runbook, or the release approval needs the change.
Report the number of migrations, the share with a rollback plan, the share with a tested rollback, and the share with a verified backup. Show the counts by service and migration type. Present the median expected rollback duration and the share of plans that named an owner. Use the same denominator for every rate.
Inspect the migrations with incomplete evidence and describe the pattern without naming the engineer. A cluster in one service may point to a missing template, while a cluster in one migration type may point to untested data transformations. Compare the result with the data volume, because a large transformation carries more risk. Report the postponed migrations separately.
A useful additional view is the gap between the backup time and the release time. A backup taken days before the release may not reflect the latest data, especially if the service is active. Show the gap and describe the consequence. That view helps the owner decide whether the backup policy or the release window needs to change.
Operating controls
The review should also compare the rollback plan with the application deploy order. A rollback that restores the schema before the old application version is live can still fail. Record the order and the person who confirms each step. That sequence is part of the evidence, not an implementation detail.
The record cannot show whether the rollback would actually work under production load. It can show only what was planned and tested. A tested rollback in staging is not proof of a safe production recovery, and an untested one is not proof of failure. The protocol also cannot capture a manual recovery that was improvised during an incident.
Migration systems can omit manual scripts, data fixes, and changes applied by a database administrator outside the pipeline. A release that looks complete may have an undocumented step. Record those cases and their treatment instead of dropping them silently. If the same person wrote the migration and approved the rollback, note the independence limitation and analyze that subgroup separately.
A useful check is whether the rollback was rehearsed with a realistic data volume. A small staging table may restore in seconds while the production table takes hours. Record the row count and the restore duration. The owner can then judge whether the window is realistic.
Give the result to the database owner and the release manager and ask them to choose one observable change. The change might be a required restore test, a backup freshness rule, or a release gate that blocks a destructive change without a plan. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the backup policy and the release gate at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If rollback plans improved but restore tests did not, the team may be writing documents without practicing recovery. If restore tests improved but release time rose, the new gate may need a faster path for low-risk changes. Close the review when the owner accepts the finding and the next sample is scheduled.
The final view is the backup verification date, not only the backup creation date. A backup that has never been restored is an assumption. Show the last restore test and the result beside the backup time. That evidence turns a backup policy into a recovery capability. Record the next review date with the sample.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
How often do outsourced pull requests arrive without a reproducible test?
Where does review time accumulate in a distributed engineering team?