On this page
Research context
## Question and scope The scope is one defined population of rollback plan cases observed from first request through verified close in the deployment console. If the population is too small to sample, say so and report the complete set rather than implying a larger study. If a definition changes during the review, keep the earlier classification and apply the new rule only to later cases. Keep the protocol with the rollback plan batch so the next reviewer can see what was measured. Keep the measurement note short enough to read with the case and specific enough to support the reported dependency age. Write the primary question and one supporting question, then stop; a long list of questions usually hides an undefined unit. The population should include cases handled by every owner and channel in scope, not only the queue with the cleanest records. The definitions should be short enough to use at the queue and precise enough to defend in a review with engineering manager. Separate the operational question from the compliance question; this method does not certify either outcome. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved.
## Population and unit A population that changes size during the day should be reported with the start, end, and net change, not only the closing total. If duplicate requests are merged, keep the duplicate count and the merge rule in the dataset. The start event should be the first recorded request, not the first time the team opened the case. Record the extraction date and the queue filter so another reviewer can reproduce the population count. Link the observation to the exact field in the dependency scanner and to the pull request that supports it. A population that changes size during the day should be reported with the start, end, and net change, not only the closing total. Report the number of cases that could not be classified and the reason, rather than forcing them into a category. The stop event should be the verified destination update, not the last message in the thread. Keep a list of requests closed as duplicates and show which case survived. Do not let a summary written later replace the evidence that was available at the first decision.
## Definitions and measures Define the controlled event as the moment a request for rollback plan becomes work, not the moment someone starts typing a note. Record the source system, the record identifier, the extraction time, and the reviewer for every extracted value. A duplicate request is a separate intake event even when it points to the same underlying need; record the link. State how duplicate, cancelled, and reopened cases enter the numerator and denominator. If the case moved between owners, include both owner periods in the review note and name the handoff time. If the definition of completion changes, keep the old and new rules side by side for the transition period. Use a stable case key and a separate identity crosswalk stored in an approved system. Before collection, test the definitions on a missing deployment window, a reopened case, and a handoff. Define whether the measure counts requests, cases, or destination updates, and use that unit consistently. Keep the review note with the case so the next sampler does not have to reconstruct the decision from memory.
Practical application
## Data dictionary and evidence Record the version or date of the deployment window and the issue tracker value so a later reviewer can see what was known at the first action. If a field is derived, record the formula and the source fields used to calculate it. Record the data extraction date and the version of the issue tracker used for the population count. Pilot the dictionary with two reviewers and compare their classifications before the main sample. When a value is missing, record the missing state and the reason instead of leaving an empty field in the extract. Keep free-text notes separate from coded fields so the analysis does not depend on inconsistent wording. Test the dictionary on a case with a missing rollback plan and a case with a corrected value. Keep the source reference and the coded value in the same row, and do not overwrite the original with a later correction. Use a separate field for reviewer notes so the coded values stay comparable. At Outsourced Programmers, the rollback plan review should end with a dated note and a named owner for the next check. Record the version or date of the QA evidence and the CI pipeline value so a later reviewer can see what was known at the first action. Name the required fields for each article type and mark conditional fields in the dictionary. Run a five-case pilot before the full sample and revise any field that two reviewers interpret differently. Pilot the dictionary with two reviewers and compare their classifications before the main sample. If the evidence is sensitive, record the approved reference and access path rather than copying the content.
## Sampling and collection Record the eligible count, the sample size, and every exclusion so another reviewer can reproduce the sample. If a source cannot be accessed, mark it unavailable and record the access problem rather than asking the operator to reconstruct it from memory. Do not expand the sample only because the early result looks weak or strong. Keep a table of planned, eligible, selected, and reviewed counts for each stratum. The note should distinguish what was observed, what was inferred, and what still needs an owner decision. Include adverse cases: a missing QA evidence, a conflicting source, an unavailable engineering manager, and a tight deadline. If a source cannot be accessed, mark it unavailable and record the access problem rather than asking the operator to reconstruct it from memory. Stratify by risk, channel, and owner so the sample is not dominated by the easiest queue. Do not let the reviewer choose the sample after seeing the results. Record this decision in the study log with the case key, the review date, and the person who approved the classification. Include adverse cases: a missing ticket, a conflicting source, an unavailable developer, and a tight deadline. Do not clean the data by filling missing values; preserve the missing state and report it as a result. Record the eligible count and every exclusion before the sample is drawn. If a selected case is out of scope, replace it only with the next case in the same stratum and record the substitution. Keep the evidence reference beside the coded value so a reviewer can test the same case without rebuilding the queue.
## Analysis and reporting Report the eligible count, the sample, the missing-field rate, the exception rate, and the unresolved count at the close of the window. Keep descriptive findings separate from causal claims. Record the sample date, the denominator, the numerator, and the exclusion count for every reported rate. End with the decision the evidence supports and the decision it does not support. If the source changes after the observation, preserve the earlier value and add a dated note instead of replacing it. Keep case-level detail available for review, because a summary chart without records cannot support a decision. If two measures disagree, report both and explain what each one counts. Keep the case-level table available for review and summarize only the measures that answer the question. When a rate changes, check whether the case mix or the evidence standard changed at the same time. At Outsourced Programmers, the reviewer should be able to explain this step to QA lead from the case record alone.
Operating controls
## Limitations Small samples are sensitive to one case; report the instability instead of presenting a percentage as a stable rate. Small samples are unstable; report the range and the cases that create the variation. Keep the limitations with the result so a later reader does not turn a local observation into a general claim. State that the result describes the observed sample, not the whole industry or every team in the service. Use the same note in the case file and the study extract; do not create a second version in a personal document. The study cannot determine whether the professional decision was correct, only whether the record and handoff are reconstructable. Report the unknown population and the evidence that would be needed to describe it. If the reviewer also performed the work, note the independence limitation and use a second reviewer for a subsample. Record the access problems that prevented a case from being reviewed. When a case needs an exception, record the reason, the owner, and the date before continuing the review.
## Review and next action If the action affects excess repository access, route it to developer and record the approval before implementation. If the action changes excess repository access, record the approval from developer before implementation. Link the action to the specific rollback plan cases that motivated it. Record the owner, due date, and exact change that will be tested at the next review. Keep the measurement note short enough to read with the case and specific enough to support the reported escaped defect rate. Record the decision and the next check so the study does not end as an archive. If the result is inconclusive, say so and choose a narrower question rather than claiming a trend. Choose a measure that can move within one cycle and a second measure that checks for unintended effects. If the action needs approval, record the approver and the date before the change is applied. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.