On this page
Research context
## Question and scope This study asks whether the test evidence attached to a pull request is associated with defects discovered after the change reaches its intended environment. It is a local, prospective question. The study does not claim that a particular test practice prevents defects in every organization, and it does not reduce quality to a single metric. It looks for a pattern that can guide one bounded improvement in an outsourced delivery workflow. Define the release boundary and the follow-up period before data collection. A release may be a production deployment, a customer-facing build, or a batch job run. The follow-up period should be long enough for the affected behavior to be exercised, but short enough that unrelated changes do not dominate the record. Record the start and end timestamps and apply them consistently to every release in the cohort.
## Population and cohort Build a cohort of changes that share a release path and a definition of done. Include changes from more than one contributor or supplier if the workflow is shared, but record the group so the analysis can show whether the pattern holds across them. Exclude emergency changes that bypassed the normal review path unless the study explicitly compares normal and emergency lanes. A mixed cohort can hide the effect of the process being studied. Use the pull request as the unit for evidence and the deployed change as the unit for defect follow-up. Link them with the reviewed commit and the deployment identifier. If one release contains several pull requests, keep the pull request evidence separate and attribute a later defect to the smallest change that could have caused it. If attribution is uncertain, record the uncertainty instead of forcing a single owner. The cohort should be fixed at the start of follow-up. Do not add changes because they have unusually good or bad evidence. If a change is reverted before release, record it as excluded with a reason. If a release is delayed, keep it in the cohort and note the delay so the follow-up window still starts from the actual release time.
## Evidence attributes Code test evidence with attributes that can be observed in the pull request: whether a failing check preceded the fix, whether the changed path has an automated test, whether the test asserts behavior rather than implementation detail, whether edge cases named in the ticket are covered, and whether the evidence identifies the environment and build. Add a flag for manual checks and a flag for known gaps. These attributes describe the packet, not the reviewer's opinion of the developer. Define an escaped defect as a defect found after release that meets the team's severity threshold and can be linked to the released change or its immediate integration surface. Record the detection source, the time to detection, the customer impact, and whether the defect was caught by monitoring, support, a later test, or a user report. A defect found in a pre-release smoke test is not an escape; keep it in a separate category so the two events are not conflated. Use a data dictionary with examples and train the coders on the same small pilot. If two coders disagree about whether a test asserts behavior, simplify the field or keep both classifications until the definition is resolved. The goal is a reproducible description of the evidence, not a perfect ontology.
Practical application
## Collection and linkage Collect the pull request evidence at merge time, before the release outcome is known. That ordering is what makes the study prospective. If evidence is coded after defects are discovered, the coder may unconsciously look for weaknesses in the packets that failed. Store the coded evidence with the merge timestamp and a hash or link to the reviewed commit so it cannot be silently replaced. Follow the release through the observation window using the deployment log, incident system, support queue, and defect tracker. Record every candidate event, then apply the escaped-defect definition. Do not count the same defect twice because it appears in two channels. Keep the source links and the detection time for each event. If a defect cannot be attributed to a specific change, record it against the release and explain the attribution limit. Mask customer data and keep the analysis table separate from the raw evidence. Record the extract dates for both the pull request data and the defect data. A later reader should be able to see which records were available at each stage and which were added after the window closed.
## Analysis and reporting Start with descriptive counts: releases, pull requests, evidence attributes, candidate defects, confirmed escapes, and exclusions. Report the escape rate per release and per pull request only when the denominator is meaningful. Show the evidence attributes by escape status, but avoid presenting a strong causal claim from a small sample. If the cohort is small, use case pairs and describe the pattern without a false precision. Check obvious confounders before interpreting the association: change size, risk area, contributor group, reviewer, release type, and the amount of traffic or usage during the follow-up. A well-tested change in a low-traffic area may have fewer detected defects simply because fewer people used it. A high-risk area may attract more testing and more reporting at the same time. If the team wants a comparison, use the same observation window and the same defect definition for both periods. Report what changed in the workflow and what did not. Do not call an association a cause. The useful conclusion is usually narrower: in this cohort, packets with a reproducible failing check and an explicit edge-case test were less often followed by a confirmed escape, or they were not, and the next step is to test one change.
Operating controls
## Limitations The study can miss defects that are never reported or that appear after the follow-up window. It can also overcount a defect that is attributed to the wrong change in a large release. Detection intensity varies by channel, customer, and season, so the measured escape rate is partly a measure of watching as well as of product quality. A quiet period is not proof that no defect exists. The evidence attributes are proxies. A test that looks behavior-focused may still be weak, and a manual check may be stronger than its label suggests. The coder's judgment is part of the instrument. Report inter-coder agreement or a second review for a subsample, and preserve the original evidence so a later reviewer can challenge the classification. Public secure-development and testing guidance supports the general practices, not a local effect size. This protocol produces evidence for one team's decision. It should not be used to certify a supplier, rank an individual, or claim an industry benchmark.
## Review and next action Share the result with the engineering owner and the quality lead, then choose one test-evidence change to trial: require a failing check for bug fixes, add an edge-case field to the pull request template, or review one high-risk path with a second tester. Define the expected movement and the guardrail before the trial. The guardrail might be review turnaround, test runtime, or the share of changes that ship without a manual check. Run the cohort again after the change with the same definitions. Compare the evidence attributes first, then the escape outcome. If the trial did not work, report it and stop or redesign it. The value of the study is a better decision about one practice, not a permanent dashboard that nobody uses.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
Auditing access removal after contractor offboarding
How much review evidence does an outsourced pull request need?