Table of contents
Research context
This protocol asks how often outsourced releases roll back after deployment and what causes the rollback. The unit is one production deployment of a release artifact, identified by the deploy record and the artifact digest. The study measures the rollback rate, the time from deploy to rollback, the cause, and whether the team recovered service. It does not evaluate the quality of the release plan.
Set the scope to one service, one deployment platform, and one release process. Record the rollback policy, the approval rules, and the definition of a production deployment at the start. If the platform changes during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A new canary step can look like a lower rollback rate when it is only a different gate.
Define the population as every production deployment that completed in the window, including deployments that were later rolled back. The unit is the deployment, not the release, because one release can deploy several times. A redeploy of the same artifact after a platform failure is a new unit. A rollback that is itself rolled forward is a new deployment with a link to the original.
Record the service, release type, change size, deploy window, and deployer. These strata matter because a database migration and a copy change carry different recovery risks. Include deployments from every environment that serves production traffic. Test the unit definition on two ordinary releases before collection begins. Use one release type list for the whole sample.
Define a rollback as a deployment that restores a previous artifact or reverses a production change to recover service. Define a failed deployment as one that never served production traffic and was stopped during rollout. Define a hotfix as a new artifact deployed to correct a production defect. Define a platform fault as a failure in the deploy system rather than the application. Each definition needs an example and a boundary case.
Do not count a feature flag being turned off as a rollback unless it restores the prior production behavior and is recorded as recovery. Do not count a planned blue-green switch as a rollback. A deploy that is stopped before traffic reaches it is a failed deployment, not a rollback. Record those distinctions so the rate reflects actual recovery work.
For each deployment, capture the release id, artifact digest, service, deployer, deploy time, traffic time, and rollback time. Add the cause category, the time to detect, the time to recover, and the person who approved the rollback. Record whether a post-incident review exists. Keep the deploy record with the sample so a second reviewer can repeat the classification.
Practical application
Also capture the test evidence attached before deploy and whether the failure was covered by a test. These fields help explain a rollback that happened because the test checked a nearby behavior. If a test existed but did not run, record the pipeline gap. A second reviewer should be able to tell whether the release evidence supported the deploy.
Draw the sample from the deployment system rather than from a list of known bad releases. Record the export date, the filters, the row count, and the selection rule. Include deployments from every service and deployer so the result is not dominated by one large release. If the history is ordered by deploy time, include every deployment in the window. State any excluded service and why.
Collect the deployment as it existed at the deploy decision, not as it looks after later edits. If the rollback reason was written days later, record the delay. Do not infer a rollback from a later redeploy without a recovery record. Note the collection date separately from the deploy date so the reader can see how much of the record has aged. Keep the raw export with the analysis.
A useful additional measure is the share of rollbacks that happened within the first hour. A fast rollback may indicate good detection, while a late rollback may indicate a weak monitoring signal. Show the detection time beside the rollback time. That view helps the owner decide whether to invest in canaries, smoke tests, or alerting. Report the two times separately.
Report the number of deployments, the rollback rate, and the median time to recovery. Show the counts by service, release type, change size, and deployer. Present the share of rollbacks caused by a code defect, a data problem, a configuration error, and a platform fault. Use the same denominator for every rate and show the number of deployments with incomplete records. Show the raw counts beside each rate.
Inspect the rollbacks and describe the pattern without naming the deployer. A cluster after a particular release type may point to a missing migration check, while a cluster on one platform may point to an infrastructure issue. Compare the result with the test evidence and the deploy window. Report failed deployments separately so they do not inflate the rollback count.
A useful additional view is the share of rollbacks that had a written recovery note before deploy. A team that improvises recovery under pressure is more likely to extend the outage. Show the note status beside the recovery time. That view helps the owner decide whether the gap is planning or execution. Name the approver in the deploy record.
Operating controls
The review should separate a code defect from a release-control failure. A change that was never tested is a control gap, while a tested change that fails in production is a product risk. Record the evidence and the decision. That split prevents the team from adding more tests when the real problem is a skipped check. Keep the test status in the record.
The record cannot show whether the rollback was the best choice or whether a forward fix would have been faster. It can show only the timestamps, cause, and evidence. A rollback is not proof of a bad release, and a smooth deploy is not proof of quality. The protocol also cannot capture a rollback that happened outside the deployment system.
Deployment systems can omit manually triggered changes, and rollback records can be merged with incident tickets. A deploy that was reverted by a platform automation may never appear as a human action. Record those gaps and their treatment instead of dropping them silently. If the deployer and the rollback approver are the same person, note the independence limitation. Keep the original deploy version with the sample.
A useful check is whether the team can restore the previous artifact from a clean environment without the original deployer. Ask for the command and the time it takes. That test measures the recovery path, not the person. Update the runbook with the result. A rollback plan that depends on one engineer is a single point of failure.
Give the result to the release owner and the engineering lead and ask them to choose one observable change. The change might be a canary stage, a migration check, a tested rollback command, or a stricter deploy window. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the test suite and the deploy process at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If rollbacks fell but recovery time rose, the team may be avoiding rollback when it should recover. If recovery time fell but failed deployments rose, the deploy gate may have become too loose. Close the review when the owner accepts the finding and the next sample is scheduled.
The final view is the customer impact, not only the internal rollback. Record how long customers saw the failure and how many were affected. That context helps the owner prioritize the fix that prevents the next outage. Keep the customer timeline with the deploy record. A rollback rate without impact is only half the story.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
How much of an offshore sprint goes to unplanned production support?
What share of offshore handoff notes are usable without a meeting?