On this page
Research context
The question this protocol addresses is narrow: for a defect found after a release, can a reviewer assign the earliest stage where an existing control should have caught it, using records that a second reviewer can inspect and rebuild? Attribution is often requested in a performance conversation, which is exactly why the method has to be fixed before anyone reads a case. A study designed after the results are visible is a justification exercise, not a measurement.
A defect escape is a defect present in a released artifact that a declared control did not detect before release. That sentence contains three judgements that must be tested separately: the defect existed in the artifact, the artifact reached the released state, and the named control was in scope for that change. Teams that skip the third judgement end up attributing every escape to whichever stage is least popular that quarter, usually review or QA, without checking whether the control was ever designed to detect that class of defect.
The population is every defect reported against a released artifact in a frozen window, limited to one product or service and one severity band. The observation unit is one defect report linked to a release, a change set, and the control evidence that existed at the moment of release. Defects found before release are outside the population even when they are interesting, because the method studies escapes rather than the entire defect flow.
Inclusion rules should be written before extraction begins. Include defects with a reproducible report, a release identifier that resolves to an artifact, and a change set within a declared lookback. Exclude duplicates, reports later withdrawn, and environment failures with no product change. Record every exclusion with the reason and the reviewer who applied the rule, because a shrinking denominator is the simplest way to improve an attribution rate without improving anything real.
The source chain matters more than the final label. Build it from the defect tracker, the release record, the change set, the review record, and the test evidence attached to the change. If any link is missing, the case is incomplete rather than attributable. A label derived from a partial chain is a guess stored in a form field, and it will be treated as fact by whoever reads the summary next quarter.
Use a small closed set of attribution labels. A workable set is introduced in the change set, present before the change set, environment or data condition, requirement ambiguity, and unknown. Keep a separate field for the control that should have caught it, because the introducing change and the missing control are different findings with different repair actions. A defect introduced by an older change but caught by no test at release is not evidence that the current author wrote poor code.
Denominators need the same care as labels. Report eligible defects, attributed defects, incomplete cases, and exclusions. A rate of escapes attributed to one stage divided by all attributed defects is not the same as that stage divided by all eligible defects, and the difference matters whenever the incomplete count is material. Publish both ratios when they diverge, and never present the first as if it were the second.
Practical application
Time states belong in the record even though this is not a speed study. Separate the defect introduction window, the detection point, the triage decision, the fix, and the release that carried the fix. A slow fix and a late detection are different control failures with different owners, and combining them into one interval hides the repair action. Record the timestamps that actually exist rather than reconstructing them from memory later.
Reviewer independence is the central control. Two reviewers read the same case without seeing each other's label, using the same codebook and the same source access. Compare the labels and the evidence each cited. Disagreement is data about the codebook, not a tie to break by seniority, and a case where one reviewer cites a missing test while the other cites a missing requirement is revealing even when both chose the same label.
Blinding has limits in a small team. The reviewer may know who wrote the change, and may have discussed the incident. Ask them to record the label and the evidence before opening the author field, then report that the blinding was partial. Pretending to a double blind design that the team cannot support is worse than stating the constraint plainly, because a reader will otherwise over-trust the agreement statistic.
Build the codebook from examples rather than adjectives. Each label needs two positive examples and one boundary example drawn from closed cases. A label called missed review is useless until the team agrees whether it covers a reviewer who commented on the file but not the failing line, a reviewer who approved without opening the diff, and a change that had no reviewer assigned at all.
Run a known-good and known-bad control before the real sample. Seed two synthetic cases: one where the control evidence clearly exists and one where it clearly does not, both with plausible comments and links. If either reviewer labels the known-bad case as properly controlled, the method is not ready for real records and the codebook needs another pass before extraction.
Recompute a protected subset from source artifacts. Choose cases where the original label is stored, hide it, and rebuild the chain from the tracker and the release record without the first reviewer's notes. A label that cannot be rebuilt from sources is a personal opinion in a database, and it will not survive the first challenge from an engineering manager who disagrees with the conclusion.
Alternative explanations should be listed before results are read. Change mix, release frequency, test selection, reviewer capacity, tool migration, and improved detection can all move the attribution pattern. This method describes where records point under a declared codebook. It does not establish that a stage caused an escape, and it cannot separate a weaker process from a harder quarter without a design that supports that comparison.
Operating controls
Defect records contain customer data, reproduction steps, and sometimes security detail. Use named least-privilege access, stable study keys, redacted extracts, and an approved storage location with a retention date. A study folder copied to a personal drive defeats the access control around the tracker, and a shared spreadsheet with customer names is a second incident waiting for an audience.
The published table should separate observed fields from reviewer classifications. Observed fields are release, artifact digest, change set, detection date, severity, and linked evidence. Classifications are attribution label, missing control, and a confidence note. Each classification cites the codebook rule in force, so a later reader can see whether a change in the rule or a change in the system produced a shift in the numbers.
Sensitivity checks are required. Recalculate with duplicates included, with the lowest severity band removed, and with incomplete cases assigned to the most common label. If the headline changes under a defensible alternative, report both readings and name the decision that depends on the definition. A result that only survives one arrangement of the data is a preference, not a finding.
State a stopping rule. Stop when the window closes, when source integrity fails, or when a security condition requires a protected review path. Extending the window because early cases look inconvenient creates a different study with a different denominator, and it should receive a new version, a new owner, and a written rationale rather than a quiet date change.
Refresh the study when the release process, the review policy, the test suite, or the tracker changes. Compare like with like across periods, and mark a tracker migration as a break in the series. A trend line drawn across a change in how defects are recorded is a drawing, not a measurement, and it will mislead anyone who was not present for the migration.
The practical output is one bounded repair action: add a fixture, change a review checklist item, fix a release gate, or clarify a requirement template. Name the owner, the expected signal, the review date, and the condition for reversing the action if it slows review without improving detection. The method reports no local escape rate, no benchmark, and no causal claim about outsourced work; the sources describe secure development and review practice, and they do not measure any team using this method. Reproducibility is the acceptance test, and a second reviewer should be able to rebuild the chain for a sample, reach the same labels or record a documented disagreement, and identify who retained the final decision.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
Review Turnaround Evidence: A Protocol With Declared Clocks