On this page
Research context
## Question and scope This protocol asks a narrow question: when a client reviewer approves an outsourced pull request, can a second reviewer reconstruct the basis for that decision from the recorded packet? The study examines the review record and the linked artifacts, not the reviewer's competence, the product outcome, or whether the code was elegant. A packet may be complete and still contain a poor decision; the protocol is designed to detect missing or unusable evidence, not to certify judgment. Set the scope to one repository, one work stream, and a fixed observation window. Record the review workflow version, the required checks, and the branch rules in force at the start. If the team changes its review template during the window, either keep the earlier definition for the full sample or treat the change as a separate period. Mixing two definitions makes a change in evidence practice look like a change in reviewer behavior.
## Population and unit Define the unit as one pull request that reached a merge or reject decision inside the window. A pull request with several commits remains one unit unless the team explicitly reopened it under a new requirement. Record drafts, abandoned branches, and review-only experiments as outside the population, with counts and reasons. If a pull request is reopened after a revert, count the original and the reopened review separately only when the second decision has its own review record. Use the pull request number plus repository as the stable key. Record the authoring organization, reviewer role, changed-file count, and whether the change touched a high-risk area such as authentication, payments, migrations, or deployment configuration. These strata matter because a documentation-only pull request and a permission change should not be judged against the same evidence standard. The unit definition should be testable on one ordinary and one high-risk example before collection begins.
## Evidence fields Build a field list from the decision the reviewer had to make. The packet should identify the linked ticket, the reviewed commit, the changed surface, the checks that ran, the checks that did not run, the observed results, and the named approver. A usable record also says what was assumed and what remains unverified. These fields can be present but empty, so code each one as present-and-usable, present-but-empty, missing, or not applicable. Avoid collapsing a missing screenshot, a failed test, and a test that was never attempted into one negative category. The reviewer needs to know whether the evidence shows a problem or simply does not exist. For each field, record the source location such as a pull request description, a check run, an attachment, or a linked issue. If the evidence lives only in a chat thread that will expire, mark it as fragile and note the retention limit. A second coder should be able to classify the same packet without asking the author what they meant. Pilot the field list on three pull requests, compare classifications, and rewrite any field that produces disagreement. The final dictionary should include one positive example and one boundary example for each code.
Practical application
## Sampling and collection Draw the sample from the repository's review export, not from a list of memorable pull requests. Record the export date, the filter, the row count, and the random or systematic start rule. If the population is ordered by merge time, a fixed interval can spread the sample across the window. Include merged and rejected decisions, because a rejected pull request can reveal whether the review recorded a reason or simply ended without explanation. Collect the packet from the repository and the systems it links to, then stop at the recorded decision. Do not reconstruct evidence that was never captured at the time, because later reconstruction measures the investigator's effort rather than the original review. If a linked attachment has expired, record the expiry as a finding and preserve the link metadata. Note the collection date separately from the review date so the reader can see how much of the record has aged. Use a short pilot to estimate how long each packet takes to classify. If the sample cannot be completed in the available review window, reduce the scope to one work stream rather than cutting corners on the fields. A smaller complete sample is more useful than a large sample with inconsistent coding.
## Analysis and reporting Report the count of pull requests by stratum, the share with a usable ticket link, the share with a reviewed commit, and the share with a result for each required check. Present missing, blocked, and failed evidence separately. A rate should always include its numerator, denominator, and observation window. If the team wants a simple headline, choose the field most likely to change a merge decision and report it beside the overall packet completeness. Do not rank reviewers by the number of comments or the speed of approval. Those measures reward volume and speed, not evidence. Compare packets within the same risk stratum and inspect the cases that lack a required field. A high completeness rate on low-risk documentation changes does not tell you whether high-risk changes are adequately reviewed. Where the data permits, show the distribution of packet completeness rather than only an average. A team with most packets complete and a small tail of missing evidence has a different problem from a team with uniformly thin records. Describe the tail cases and the reason they were difficult to document, then propose one process change that could address that reason.
Operating controls
## Limitations The review record cannot show whether the reviewer actually read the material or understood the risk. It can show only what was available to a later reader. A complete packet may coexist with a bad decision, and an incomplete packet may accompany a sound one. The study should state that distinction plainly rather than implying that documentation equals quality. Repository exports can omit review comments, force-pushed commits, deleted branches, and attachments that expired before collection. A pull request that was approved before the window but merged inside it may belong to a different process version. Record those cases and their treatment instead of silently dropping them. If the reviewer and the author are the same person, note the independence limitation and analyze that subgroup separately. Public guidance on secure development and code review describes practices, not a benchmark for this repository. The protocol therefore produces a local description, not an industry score. Any comparison across teams should be presented as a prompt for inspection, not proof that one team is better.
## Review and next action Give the result to the people who own the review workflow and ask them to select one change: add a required field to the template, link the reviewed commit automatically, retain attachments longer, or split high-risk changes into a separate review path. Choose a change that can be observed in the next sample. Record the owner, the effective date, and the measure that will show whether the packet improved. Repeat the protocol with the same definitions and window after the change. Compare the same fields, not a new dashboard that happens to look better. If the change did not move the result, report that honestly and inspect whether the field was ignored, the template was bypassed, or the underlying workflow made the evidence hard to capture. Close the review when the owner accepts the finding and the next check is scheduled.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.