Table of contents
Research context
This protocol asks what share of defects in offshore-delivered software are found after release rather than in review or pre-release testing. The unit is one confirmed defect associated with a change delivered during the window. The study measures the stage of discovery, the time to discovery, and the severity. It does not judge the offshore team's skill or assume that every escaped defect was preventable.
A defect found after release is not automatically a testing failure. Some behaviors only appear with real traffic, real data, or a customer's device. The protocol should record the conditions that made discovery possible, such as load, locale, or a third-party integration. That context helps the team decide whether a test could reasonably have caught it.
Set the scope to one product, one release process, and one defect taxonomy. Record the environments, the test gates, and the definition of a confirmed defect at the start. If the release process changes during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A new test gate can look like a quality improvement when it is only a different filter.
Measure the time from deployment to discovery in business hours as well as calendar days. A defect found on a Monday after a Friday release had less exposure than the calendar suggests. Report both and state the release time. The reader should see how long customers were affected, not only the date difference.
Define a confirmed defect as a reported behavior that a named owner reproduced and accepted as unintended. Exclude duplicates and reports that were closed as working as designed. Link each defect to the change that introduced it when the link can be established, and mark the link as unknown when it cannot. Do not force a link that the evidence does not support.
Record whether the defect was reported by a customer, by monitoring, or by an internal review. A customer-reported defect carries a different cost from one caught by an alert before anyone noticed. The source of discovery is part of the severity picture. Report the counts by source and severity together.
Record the discovery stage, the discovery date, the severity, the customer impact, the time from deployment to discovery, and the change or release that introduced the behavior. Add the test that would have caught it, if one can be named. Keep the original report and the resolution together. A second reviewer should be able to classify the stage from the record.
Separate a regression from a defect in new behavior. A regression breaks something that worked before, while a new feature may have a gap that no one specified. The two imply different fixes: regression prevention versus requirement clarity. Record the category and review the pattern separately.
Practical application
Report the number of defects, the share found in review, the share found in pre-release testing, and the share found after release. Show the counts by severity and by product area. Present the median time from deployment to discovery for escaped defects. Use the same denominator for every rate and show the number of defects with unknown introduction points.
Check whether the change had a feature flag and whether the flag limited the exposure. A defect behind a flag that never reached all users is a different event from one that shipped to everyone. Record the rollout stage at discovery. The same bug can have very different customer impact depending on the stage.
Inspect the escaped defects and group the missing checks by type: missing test, missing environment, unclear requirement, or data condition. Compare the pattern with the release size and the test coverage of the changed area. Report low-severity cosmetic issues separately from defects that affected a customer's ability to complete a task. The split keeps the headline rate from being dominated by trivia.
Ask the person who found the defect what they were trying to do. The answer often reveals a missing scenario that the test plan did not consider. Record the user goal in one sentence and link it to the test case if one exists. That link is more useful than a general note to test more.
The record cannot show whether a defect would have been caught by a reasonable test or whether the team chose to ship with a known risk. It can show only where the defect was found and how long it took. A low escape rate is not proof of quality if the reporting channel is weak. The protocol also cannot capture defects that customers never report.
Do not report a single escape rate without the severity mix. A team with many cosmetic escapes may look worse than a team with one severe escape, even though the customer impact is lower. Show the severity distribution beside the rate. The owner should make decisions from the impact, not the headline.
Give the result to the QA lead and the client owner and ask them to choose one observable change. The change might be a pre-release checklist for a risky area, a test-data improvement, a smaller release batch, or a customer-report review. Record the owner, the effective date, and the measure that will show whether the change worked. Repeat the protocol with the same definitions and compare the same fields.
A useful additional view is the share of escaped defects that were caught by the same test after a fix was added. A defect that can be reproduced by an automated test is a candidate for prevention. A defect that cannot be reproduced may need better observability rather than another test. Record the outcome of the fix.
Operating controls
Review the defect taxonomy with the team before collecting the next sample. If two people disagree about whether a missing requirement is a defect or a scope change, the rate will be noisy. Write the examples and revise them once. A stable taxonomy is worth more than a larger sample.
End with one change to the release or test process and a date to re-measure. The change might be a pre-release checklist for a risky area, a smaller release batch, or a monitoring alert. Record the owner and the expected effect. Repeat the protocol with the same definitions and compare the same fields.
Record the deployment method and the rollout percentage at the time of discovery. A canary release that reached five percent of traffic limits the customer impact even when the defect is severe. A full release exposes everyone at once. Report the rollout stage with the defect count so the reader can see the exposure, not only the volume.
Check whether the defect was found in a part of the product with automated tests or only manual checks. A gap in an area with no tests is a different finding from a test that existed but did not cover the condition. Record the test coverage status and the closest existing test. The owner can then decide whether to add coverage or improve the test design.
Ask whether the defect report included enough detail for the team to reproduce it on the first attempt. A report that requires three rounds of questions adds cost after release and delays the fix. Record the number of clarification rounds. That measure shows whether the customer-facing intake form is doing its job.
Record the time between the defect report and the first engineer response, separately from the time to fix. A fast fix with a slow acknowledgement still leaves the customer waiting and may generate duplicate reports. Show both intervals in the report. The owner can then decide whether the gap is triage capacity or communication.
Ask whether the defect was included in the release notes or known issues list. A known issue that is documented gives support a script and gives the customer an explanation. An undocumented issue discovered after release creates a harder conversation. Record the documentation status and use it to improve the release checklist.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
What share of offshore pull requests are abandoned before merge?
How often does an offshore estimate change after refinement?
