On this page
Research context
## Research question and scope
This protocol asks which kinds of review comments precede production incidents in outsourced code, and whether those comments point to a fixable gap in the ticket, the tests, or the review process. The unit is one review thread linked to a pull request that later contributed to an incident. The study examines the comment and the incident record; it does not evaluate the reviewer's skill or the programmer's intent. That check belongs in the research question and scope note, with the owner and the next action visible.
Set the scope to one repository, one incident definition, and one observation window. Record the severity scale, the incident review process, and the review workflow version in force at the start. If the team changes its severity scale during the window, keep the earlier definition for the full sample or analyze the change separately. Record the result against the research question and scope evidence before the case is closed.
## Population and unit
Define an incident as a production event that met the team's written severity threshold and required a response beyond routine monitoring. Link the incident to the pull request or change that most likely introduced the condition, using the incident review record. Do not infer causation from timing alone; record the evidence and the uncertainty. The population and unit record should be dated and linked so a second reviewer can reproduce the result.
The unit is the review thread attached to the linked change. A thread with several replies remains one unit, with the resolution recorded separately. Exclude purely social or scheduling comments, but count bot-generated comments separately. Record the file, the risk area, the comment category, and whether the thread blocked the merge.
## Definitions and taxonomy
Start with a small taxonomy: correctness, security or privacy, data integrity, test coverage, operability, performance, accessibility, maintainability, and documentation. Add a separate flag for questions, preferences, and blocking versus non-blocking comments. A comment can carry more than one code only when the reviewer explicitly raises more than one risk. The definitions and taxonomy owner should confirm this at the next review.
Write one example and one non-example for each category. A missing null check is correctness, while an unvalidated permission check is security. A request to rename a variable is maintainability unless the name exposes a secret or changes a public contract. Pilot the taxonomy on a small batch and reconcile disagreements before coding the full sample.
## Evidence and data dictionary
For each thread, capture the comment text, the file and line, the author role, the reviewer role, the resolution, and the incident link. Record whether the comment was resolved before merge, deferred, disputed, or left open. Link the incident review and the exact code version rather than a summary from memory. Record the result against the evidence and data dictionary evidence before the case is closed.
Add a field for the action that followed: code changed, test added, documentation updated, risk accepted, or no action recorded. A pattern of security comments with no action is different from a pattern of security comments that were fixed. The resolution field lets the analysis separate detection from follow-through. The evidence and data dictionary owner should confirm this at the next review.
## Sampling and collection
Practical application
Draw the sample from the incident register and the linked pull requests, not from a list of interesting review comments. Record the incident selection rule and the number of incidents that could not be linked to a change. Include incidents with a known root cause and incidents where the cause remains uncertain, but code the uncertainty separately. That check belongs in the sampling and collection note, with the owner and the next action visible.
Collect the review thread as it existed at merge and preserve later edits. If a comment was deleted, record the limitation. Use the repository export for the code context and the incident system for the outcome. Note the collection date and any retention limits on incident records.
## Analysis and reporting
Report the number of linked incidents, the comment categories that preceded them, and the share of those comments that were unresolved at merge. Separate incidents by risk area and by whether the change had a test covering the affected path. Present counts and rates with the denominator, not a ranked list of reviewers. The analysis and reporting record should be dated and linked so a second reviewer can reproduce the result.
Inspect the cases where a reviewer raised the exact risk and the team merged anyway. Those cases may point to a decision-right problem rather than a review-quality problem. Also inspect the incidents with no related comment, because they may reveal a missing check or a gap in the taxonomy. That check belongs in the analysis and reporting note, with the owner and the next action visible.
## Limitations
The review thread cannot show whether the reviewer anticipated the incident or simply raised a related concern. The incident review may identify a contributing change with uncertainty, and the protocol should preserve that uncertainty. A comment linked to an incident is not proof that the comment would have prevented it. The limitations owner should confirm this at the next review.
Repository exports can omit deleted comments, force-pushed code, and discussions held outside the review. Incident records may be incomplete or written after the fact. If the sample is small, report cases and categories rather than rates. The protocol describes patterns in one environment, not an industry benchmark.
## Review cadence and next action
Ask the review and incident owners to choose one change: add a required check for the recurring category, clarify the authority to block a merge, or add a test that covers the affected path. Record the owner and the effective date. Choose a change that can be observed in the next sample. Record the result against the review cadence and next action evidence before the case is closed.
Repeat the protocol with the same taxonomy and compare the same fields. If the same category recurs after the change, inspect whether the new check is being used or bypassed. Close the review when the owner accepts the finding and the next check is scheduled. The review cadence and next action owner should confirm this at the next review.
## Connecting comments to decisions
Operating controls
A review comment is only one part of the decision record. Link it to the approval, the merge, and the incident outcome so the analysis can see whether the concern was resolved. A comment that was raised and fixed is different from one that was raised and ignored. That check belongs in the connecting comments to decisions note, with the owner and the next action visible.
If the decision was made outside the review thread, record where it happened and who made it. That context prevents the study from treating a silent thread as a missing concern. The report should describe the decision path, not just the comment text. Record the result against the connecting comments to decisions evidence before the case is closed.
## Reading the incident timeline
Place the review comment on the incident timeline so the reader can see how close the concern was to the failure. A comment that named the exact risk is stronger evidence than a general request for more tests. Record the time between the comment and the merge. The reading the incident timeline record should be dated and linked so a second reviewer can reproduce the result.
If the incident happened long after the merge, other changes may have contributed. Record the intervening changes and the uncertainty. The protocol should not claim that an earlier comment would have prevented an incident that depended on later conditions. That check belongs in the reading the incident timeline note, with the owner and the next action visible.
## Using the result without blaming
The result should help the team improve its review and decision rights, not rank reviewers. Present categories and process gaps, and avoid naming individuals in the summary. If a specific case needs follow-up, route it through the normal management process. The using the result without blaming owner should confirm this at the next review.
The owner should choose one change that can be observed in the next sample. A new required check, a clearer blocking rule, or a test for the affected path are all more useful than a general reminder to be careful. The using the result without blaming record should be dated and linked so a second reviewer can reproduce the result. The next report should use the same categories so the owner can compare the result.
## A short reporting template
Report the number of linked incidents, the comment categories, the share unresolved at merge, and the action that followed. Separate high-risk code paths from low-risk changes and show the counts behind every rate. Describe the incident linkage and the uncertainty rather than implying a precise cause. Record the result against the a short reporting template evidence before the case is closed.
Use short case notes to explain the pattern, not to name individuals. A comment that named the exact risk and was overridden points to a different fix from a risk that was never noticed. The owner needs that distinction before choosing a process change. The a short reporting template owner should confirm this at the next review.
End with one change, one owner, and the next sample date. If the change is a new required check, say where it appears and who can waive it. A check without a waiver path tends to be bypassed under deadline pressure. The a short reporting template record should be dated and linked so a second reviewer can reproduce the result.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
How often do outsourced pull requests change files outside the ticket?
Can outsourced dependency upgrade decisions be reconstructed?