On this page
Research context
This protocol asks how often an outsourced pull request arrives without a reproducible test that a reviewer can run. The unit is one pull request submitted for review during the observation window. The study measures whether the change includes a test, whether the test fails before the change, and whether the reviewer can reproduce the result from the description. It does not evaluate the quality of the implementation or the severity of the underlying defect.
Set the scope to one repository, one review team, and one test framework version. Record the branch protection rules, the continuous integration configuration, and the pull request template in force at the start. If the template changes during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A new required field can look like an improvement in engineer behavior when it is only a change in the form.
Define the population as every pull request that reached a review decision during the window, including requests that were closed without merging. The unit is the pull request, identified by the repository and the pull request number. A request that is reopened after new commits remains one unit with a revision history. A separate request for a different behavior is a new unit.
Record the author's team, the change type, the size band, and whether the change touches security-sensitive code. These strata matter because a documentation change and an authentication change carry different evidence expectations. Include draft pull requests that were reviewed, because early review is part of the process. Test the unit definition on two ordinary requests before collection begins.
Define a reproducible test as one a reviewer can run from the pull request description using the stated command, environment, and data. Define a failing-first test as one that fails on the parent commit and passes on the change. Define a reviewable test as one that checks the behavior named in the description rather than an unrelated internal detail. Define a missing test as a change with no automated check for the stated behavior.
Do not count a screenshot of a passing test run as a reproducible test unless the command and environment are also stated. Do not count a test that passes before the change as evidence of the fix. A test that only checks that the application starts is not a test of the behavior. Each definition should include one positive example and one boundary example so a second reviewer can apply it consistently.
For each pull request, capture the repository, the number, the author team, the change type, the size band, and the test command. Add whether a test was added, whether it was run in continuous integration, and whether the failing-first result was recorded. Record the reviewer's reproduction result and the time spent trying to reproduce it. Keep the record with the review packet for the next sample.
Practical application
Also capture whether the description names the expected behavior and the environment. These fields help explain a request that has a test but no usable instructions. If the test was run only locally, record the local environment and the reason it was not run in the shared pipeline. A second reviewer should be able to repeat the classification from the pull request record. Keep the record with the review packet for the next sample.
Draw the sample from the review system rather than from a list of memorable bugs. Record the export date, the filters, the row count, and the selection rule. Include requests from every author team and every size band so the result is not dominated by one group. If the population is ordered by review date, a fixed interval spreads the sample across the window.
Collect the pull request as it existed at the review decision, not as it looks after later force-pushes. If the description or test changed after the review, record the edit time and analyze the reviewed version separately. Do not infer a missing test from the eventual fix. Note the collection date separately from the review date so the reader can see how much of the record has aged.
A useful additional measure is the reviewer's confidence in the test. Ask the reviewer to rate whether the test would catch the behavior if it regressed. A test that runs but does not assert the behavior is weak evidence. Record the rating, the reason, and the date of the check. That signal helps the owner target the weakest part of the test suite and compare it with the reproduction rate.
Report the number of pull requests, the share with a test, the share with a failing-first test, and the share the reviewer could reproduce. Show the counts by author team and change type. Present the median reviewer time spent reproducing and the share of requests that required a question. Use the same denominator for every rate and show the number of requests with incomplete evidence.
Inspect the requests without a reproducible test and describe the pattern without naming the author. A cluster in one change type may point to a missing template, while a cluster in one team may point to a different definition of done. Compare the result with the size band, because a large change may be harder to test than a small one. Report the drafts and the closed requests separately.
A useful additional view is the share of tests that failed in continuous integration before the change was merged. A test that exists but never runs is not evidence. Show the pipeline result beside the test presence and describe the gap. That view helps the owner distinguish a missing test from a test that is not enforced.
Operating controls
A second view is the age of the test relative to the code. A test added months after the behavior can still be useful, but it may not prove that the change caused the fix. Record the test creation date and whether it was added in the same pull request. That detail separates a regression test from a retrospective check.
The record cannot show whether the engineer understood the behavior or whether the test covers the real risk. It can show only what was written, run, and observed. A missing test is not proof of a careless engineer, and a present test is not proof of correctness. The protocol also cannot capture a manual verification that was discussed in a call and never written down.
Pull request systems can omit force-pushed commits, deleted branches, and review comments that were resolved. A test that was removed before the review decision may leave no trace. Record those cases and their treatment instead of dropping them silently. If the reviewer also wrote the change, note the independence limitation and analyze that subgroup separately.
The review should also distinguish a missing test from a test that cannot run in the reviewer's environment. A test that requires a private dataset or a production credential is not reproducible for most reviewers. Record the dependency and the workaround. That gap is a tooling problem as much as an engineer problem.
Give the result to the engineering lead and the client reviewer and ask them to choose one observable change. The change might be a required test command field, a failing-first check in the pipeline, or a rule that a change cannot enter review without a reproducible case. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the template and the pipeline at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If test presence improved but reproduction did not, the instructions may still be unclear. If reproduction improved but escaped defects did not, the tests may be checking the wrong behavior. Close the review when the owner accepts the finding and the next sample is scheduled.
A final view is the share of tests that were changed in the same pull request as the behavior. A test that was weakened to pass is different from a test that was added to prove the change. Record whether assertions were removed or skipped. The owner can then decide whether the test policy or the review practice needs attention. Record the next review date with the sample.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
Where does review time accumulate in a distributed engineering team?