On this page
Research context
## Question and scope The protocol does not estimate an industry benchmark; it produces a local description that engineering managers can review. State how duplicate requests are identified and whether they are merged, linked, or counted separately. A case is complete only when the code repository shows the approved destination value and the code review supports it. If the study is exploratory, label it exploratory and do not present the result as a stable baseline. If the source changes after the observation, preserve the earlier value and add a dated note instead of replacing it. A useful scope is narrow enough to reproduce and specific enough to show whether excess repository access is being detected. A count without a denominator is not a measurement, especially when the queue changes size during the day. Write the primary measure and one supporting measure; a measure without a denominator or observation window is not ready. Keep the protocol with the code review batch so the next reviewer can see what was measured. At Outsourced Programmers, the reviewer should be able to explain this step to QA lead from the case record alone.
## Population and unit If a reopened case creates a new requirement, count it as a new case and retain a link to the earlier record. Exclude cases opened outside the window only when the exclusion rule was written before collection. Define how a case that waits for product owner is counted: paused, open, or excluded, but not silently dropped. A case that never received an owner still belongs in the population; do not drop it because it is difficult to classify. Use the same note in the case file and the study extract; do not create a second version in a personal document. Include cases opened before the window but closed inside it only when the protocol says they belong to the population. Report the number of cases that could not be classified and the reason, rather than forcing them into a category. A correction should be classified by the original case and by the correction event, with both dates retained. If the queue was archived during the window, report the loss and exclude the affected period rather than assuming the cases were resolved. When a case needs an exception, record the reason, the owner, and the date before continuing the review.
## Definitions and measures Name the owner who can declare a case complete, and record that role in the data dictionary. Record the version of the rollback plan and the deployment console snapshot used for each case. A duplicate request is a separate intake event even when it points to the same underlying need; record the link. A definition that depends on the reviewer's memory is not reproducible enough for this study. Keep the measurement note short enough to read with the case and specific enough to support the reported escaped defect rate. Test each definition on a routine case, a missing-evidence case, and a case that crossed owners. Use a stable case key and a separate identity crosswalk stored in an approved system. A correction is a change to a closed record; a clarification that does not change the record is a note, not a correction. Keep a change log for every definition adjustment and the date it took effect. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved.
Practical application
## Data dictionary and evidence Record the version or date of the QA evidence and the dependency scanner value so a later reviewer can see what was known at the first action. Record the extraction time separately from the event time so a late entry is visible. Use one row per case and one column per required field; do not bury evidence in a long free-text note. Define who may change a coded value and how the change is logged. Link the observation to the exact field in the dependency scanner and to the QA evidence that supports it. Mark missing, unavailable, and confirmed negative values separately; collapsing them hides the difference between no evidence and evidence of absence. Keep the identity crosswalk in the approved system and restrict it to the review owner. Keep the source reference and the coded value in the same row, and do not overwrite the original with a later correction. Record the source location, not just the file name, so a reviewer can reopen the exact evidence. Do not let a summary written later replace the evidence that was available at the first decision. If the source is a document, record the page or field; if the source is a system, record the record identifier and extraction time. Test the dictionary on a case with a missing pull request and a case with a corrected value. Record the data extraction date and the version of the issue tracker used for the population count. A case key should remain stable if the case moves between owners or changes status. If the case moved between owners, include both owner periods in the review note and name the handoff time.
## Sampling and collection Sample across the full observation window, including the days when the queue was busiest and the days when coverage changed. Do not clean the data by filling missing values; preserve the missing state and report it as a result. Do not expand the sample only because the early result looks weak or strong. Do not let the reviewer choose the sample after seeing the results. Keep the review note with the case so the next sampler does not have to reconstruct the decision from memory. Use consecutive code review cases where possible, then stratify by owner, channel, and risk class. Record the time of each observation so a later reviewer can distinguish a late entry from a changed decision. Keep the sample selection reproducible by recording the export date, filter, and random seed or start rule. Record the sampling frame as an export with a date, filter, and row count. When a value is missing, record the missing state and the reason instead of leaving an empty field in the extract. Stop expanding the sample only when the review question is answered, not when the result starts to look favorable. If a source cannot be accessed, mark it unavailable and record the access problem rather than asking the operator to reconstruct it from memory. If a stratum has fewer cases than planned, report the smaller count instead of replacing it with a convenience case. Record the sampling frame as an export with a date, filter, and row count. At Outsourced Programmers, the code review review should end with a dated note and a named owner for the next check.
## Analysis and reporting If the sample is small, report the instability and the cases that drive the result instead of a precise-looking percentage. Keep descriptive findings separate from causal claims. If the result is unstable, say which cases create the variation and what a larger sample would test. Use a case-level table to explain an outlier before treating it as a process signal. If the evidence is sensitive, record the approved reference and access path rather than copying the content. For code review, report the age distribution and the share of cases waiting on a named decision rather than a single average. If two measures disagree, report both and explain what each one counts. Show the unresolved cases by owner, age, and dependency instead of reporting a single completion percentage. End with the decision the evidence supports and the decision it does not support. The note should distinguish what was observed, what was inferred, and what still needs an owner decision.
Operating controls
## Limitations Report what remains unknown and what evidence would be needed to answer the next question. Report the unknown population and the evidence that would be needed to describe it. Record the response or access problems that prevented a case from being reviewed. If the queue changed during the window, explain how the change affects the denominator. Record this decision in the study log with the case key, the review date, and the person who approved the classification. The record may be incomplete, delayed, or maintained outside the team, so the method cannot infer missing work from a clean summary. Report the unknown population and the evidence that would be needed to describe it. Report the limits of the CI pipeline data, including fields that were unavailable or changed during the window. If the reviewer performed the work, use a second reviewer for a subsample and report the disagreement. Keep the evidence reference beside the coded value so a reviewer can test the same case without rebuilding the queue.
## Review and next action A quality review should end with an owner, a date, and one testable change, not a list of intentions. A quality review should end with a testable next step, not a general instruction to improve. Close the review only when the owner has accepted the result and the next check is scheduled. Record the owner, due date, and exact change that will be tested at the next review. If the source changes after the observation, preserve the earlier value and add a dated note instead of replacing it. Choose one next action: fix the source, simplify the rollback plan, change the queue, adjust access, or pause the work until the owner decides. Choose one change and one measure to test it; a long action list is hard to evaluate. Report the action result even if the change did not work; a failed test is still evidence. If the action needs approval, record the approver and the date before the change is applied. At Outsourced Programmers, the reviewer should be able to explain this step to engineering manager from the case record alone.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.