On this page
Research context
## Research question and scope
This protocol asks how often an outsourced pull request changes files that the linked ticket did not authorize. The unit is one merged or rejected pull request inside a defined window. The study examines the diff, the ticket, and the review record; it does not evaluate the programmer, the reviewer, or whether the change was technically good. A small out-of-scope edit can be harmless, while a large in-scope edit can still be risky.
Set the scope to one repository, one work stream, and one review workflow version. Record the branch rules, required checks, and ticket template in force at the start. If the team changes its review process during the window, keep the earlier definition for the full sample or analyze the change as a separate period. Record the result against the research question and scope evidence before the case is closed.
## Population and unit
Define the population as pull requests that reached a merge or reject decision in the window. Exclude drafts and abandoned branches, but record their count. A pull request with several commits remains one unit unless the team reopened it under a new requirement. Use the repository and pull request number as the stable key.
Record the authoring organization, reviewer role, changed-file count, and whether the change touched a high-risk area such as authentication, payments, migrations, or deployment configuration. These strata matter because a documentation pull request and a permission change should not be judged against the same scope standard. That check belongs in the population and unit note, with the owner and the next action visible. The same strata should be applied in the next sample so the comparison remains meaningful.
## Definitions that make the measure reproducible
Define the authorized scope as the files, modules, or surfaces named in the linked ticket. Define an out-of-scope file as one that the ticket does not name and that a reviewer cannot reasonably infer as required. Define a review flag as a comment or recorded decision that notices the extra file before merge. Each definition needs a positive example and a boundary example.
Do not treat every formatting change as out of scope if the repository enforces formatting on changed files. Do not treat a generated lockfile as unauthorized if the ticket requires a dependency change. The data dictionary should distinguish necessary side effects from unrelated work, and a second coder should be able to apply the same rule. The definitions that make the measure reproducible record should be dated and linked so a second reviewer can reproduce the result.
## Evidence and data dictionary
For each pull request, capture the ticket link, the reviewed commit, the changed files, the review comments, and the final merge decision. Record the file category, the risk area, and whether the file was named in the ticket. Code the evidence as present and usable, present but ambiguous, missing, or not applicable. Record the result against the evidence and data dictionary evidence before the case is closed.
Add a field for the reviewer response: accepted with a note, split into another branch, removed, or merged without comment. Also record whether the extra file changed after approval. That field matters because a later push can alter the approved scope without a new review. The evidence and data dictionary owner should confirm this at the next review.
## Sampling and collection
Practical application
Draw the sample from the repository review export rather than from a list of memorable pull requests. Record the export date, filter, row count, and selection rule. Include merged and rejected decisions, because a rejected request can show whether scope was noticed before closure. If the repository is ordered by merge time, a fixed interval spreads the sample across the window.
Collect the evidence from the systems that existed at review time. Do not reconstruct a missing review comment later. If a branch was deleted, note the limitation and use the merge diff if it is still available. Record the collection date separately from the review date.
## Analysis and reporting
Report the share of pull requests with at least one out-of-scope file, the categories of those files, and the share where the reviewer noticed the change. Present merged and rejected requests separately, and show the numerator and denominator for every rate. A single percentage without the categories does not tell the owner what to fix. The analysis and reporting record should be dated and linked so a second reviewer can reproduce the result.
Inspect the cases that changed high-risk files without a ticket reference. Describe the specific file and the decision, not the person. If the same category recurs, the ticket template or the review checklist may need a new required field. Avoid ranking authors by scope count without adjusting for the size and type of work.
## Limitations
The protocol cannot show whether an out-of-scope change was beneficial or whether the reviewer understood it. It can show only whether the record linked the file to the authorized work. A complete ticket can still accompany a poor change, and an incomplete one can accompany a harmless cleanup. State that distinction in the report.
Repository exports can omit force-pushed commits, deleted branches, and comments that were edited. A ticket may also be updated after the review to include a file that was originally out of scope. Record the ticket version used and note any later changes. If the reviewer and author are the same person, analyze that subgroup separately.
## Review cadence and next action
Ask the review owner to choose one observable change: require a file inventory in the ticket, add a scope check to the pull request template, or split high-risk changes into a separate review path. Record the owner and the effective date. Choose a change that can be measured in the next sample. Record the result against the review cadence and next action evidence before the case is closed.
Repeat the protocol with the same definitions and compare the same fields. If scope findings fall but review time rises, decide whether the trade-off is acceptable. Close the review when the owner accepts the finding and the next check is scheduled. The review cadence and next action owner should confirm this at the next review.
## How to present a scope finding
Operating controls
Present the finding as a file-category table with the ticket reference, the reviewed commit, and the reviewer response. Use counts rather than a vague description, and show the high-risk categories separately. The owner should be able to see whether the problem is a stray formatting file or an unreviewed permission change. That check belongs in the how to present a scope finding note, with the owner and the next action visible.
If the same author appears repeatedly, do not turn the report into a performance ranking. Describe the pattern in the process and ask whether the ticket template or the review checklist is failing. A scope problem is usually a system problem before it is a people problem. Record the result against the how to present a scope finding evidence before the case is closed.
## Choosing the next sample
The next sample should use the same window and strata so the comparison is meaningful. If the team changes its template, collect a fresh baseline rather than mixing old and new definitions. Record the change date and the number of pull requests affected by it. The choosing the next sample record should be dated and linked so a second reviewer can reproduce the result.
Expand the sample only when the decision requires more precision. A small sample that is complete and well coded is more useful than a large sample with inconsistent categories. The owner should know what the next sample can and cannot answer. That check belongs in the choosing the next sample note, with the owner and the next action visible.
## Keeping the review usable
A scope check should not become a second approval layer for every file. Add it where the consequence is high, such as authentication, payments, migrations, and deployment configuration. For low-risk work, a lightweight ticket reference may be enough. The keeping the review usable owner should confirm this at the next review.
The rule should be written where contributors can see it before they open the pull request. The reviewer can then focus on the exceptions rather than repeating the same explanation. A usable rule is better than a perfect rule that is ignored. The keeping the review usable record should be dated and linked so a second reviewer can reproduce the result.
## A short reporting template
Use a one-page template: observation window, repository, number of pull requests, share with out-of-scope files, high-risk categories, reviewer response, and the chosen next action. Keep the definitions beside the numbers so the reader does not have to guess what counted as out of scope. The template should be stable enough to compare across two samples. Record the result against the a short reporting template evidence before the case is closed.
Add a short limitations section and a list of the systems that could not be checked. If a category has fewer than five cases, show the count and describe the cases instead of presenting a percentage. The owner can still act on a clear pattern without a misleading rate. The a short reporting template owner should confirm this at the next review.
End with the owner, the change, and the date of the next sample. A report without a next action becomes an archive. The follow-up should be small enough to observe and specific enough that a different reviewer could verify it. The a short reporting template record should be dated and linked so a second reviewer can reproduce the result.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
How much rollback evidence does an outsourced release packet contain?
Which review comments precede production incidents in outsourced code?