Table of contents
Research context
This protocol asks how much of an offshore review queue delay is caused by missing acceptance criteria rather than reviewer capacity. The unit is one review request in the window, identified by the ticket and the ready-for-review timestamp. The study measures the time to the first reviewer action, the time to approval, and the number of review rounds. It does not judge the quality of the code or the reviewer.
Define ready for review as the moment the author claims the work meets the published criteria, not the moment the branch is opened. A branch can sit for days before the author considers it ready. Record the claim time and the evidence attached at that moment. The queue clock should start when the reviewer could reasonably begin.
Set the scope to one client, one repository, and one review policy. Record the required checks, the reviewer group, and the working hours at the start. If the policy changes during the window, keep the earlier definition for open requests and analyze the later cohort separately. A new required check can look like a slower review when it is only a different gate.
Classify the first pause as missing evidence, reviewer capacity, a failing check, or a product question. A request can have more than one pause, so record the first one and the total time in each category. The classification should use the reviewer comment and the ticket history. A second reviewer should be able to confirm the category.
Define the population as every review request created in the window, including requests that were closed without merge and requests still open at the observation date. The unit is the request, not the commit, because one request can contain several commits. A reopened request after a rejected review is a new unit linked to the original. Test the definition on two recent requests before collection begins.
Record the ticket, the author, the reviewer, the ready time, the first action time, the approval time, the number of rounds, the missing items, and the checks that failed. Add the queue depth on the day of the first action. Keep the review thread with the record. A second reviewer should be able to rebuild the timeline.
Practical application
Separate a missing acceptance criterion from a missing test result. A criterion that was never written is a template problem, while a test that was not run is an execution problem. Report the two categories separately. The fix for one is a clearer checklist, and the fix for the other is a stricter evidence rule.
Check whether the reviewer asked for the same item on more than one request in the same week. A repeated question is a strong signal that the template is incomplete. Record the item and the number of requests. The review should fix the template before asking the team to be more careful.
Report the number of requests, the median time to first action, the median time to approval, and the share with a pause caused by missing evidence. Show the counts by reviewer group and by change type. Present the share that needed more than one round and the median added time per round. Use the same denominator for every rate.
A useful additional view is the share of requests where the reviewer had to ask for information that the ticket template did not request. That share is the clearest measure of a template gap. Record the missing field and the number of occurrences. The owner can then change one field instead of rewriting the whole process.
Inspect the requests with the longest queue time and describe the stage without naming the reviewer. A wait before first action may point to capacity, while a wait after the first action may point to missing evidence. Compare the result with the change size and the queue depth. Report requests closed without merge separately.
Operating controls
Do not compare review times across teams with different risk policies. A team that requires two approvals and a security check is not measuring the same process as a team with one reviewer. Keep the comparison inside one policy or state the difference clearly. A single average can make a careful team look slow.
The record cannot show whether the review was thorough or whether the merged change was correct. It can show only the timestamps, the rounds, and the recorded missing items. A fast approval is not proof of good review, and a slow one is not proof of a blocker. The protocol also cannot capture a review that happened in a call and left no comment.
Check whether the author had access to a staging environment where the criteria could be tested. An author who cannot run the check will attach weaker evidence and trigger a round. Record the environment and the evidence source. The review should fix access before tightening the checklist.
Give the result to the delivery lead and the client owner and ask them to choose one observable change. The change might be a required evidence field, a named reviewer per area, a smaller review batch, or a shared staging environment. Record the owner, the effective date, and the measure that will show whether the change worked. Repeat the protocol with the same definitions.
Close the review with one change and a date to re-measure. The change should address the most common missing item, not the item that is easiest to add to a form. Record the expected effect and the evidence that will prove it. A checklist that is never measured becomes another document the team learns to skip.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
What share of offshore delivery decisions are recorded in writing?
What share of outsourced production work uses a shared admin account?
