Table of contents
Research context
This protocol asks what share of outsourced QA defects are reopened after a fix and why. The unit is one defect that reached a verified-fix state during the observation window. The study measures whether it was reopened, the reason for the reopen, the time between close and reopen, and whether the fix evidence covered the reported behavior. It does not evaluate the severity of the original defect.
Set the scope to one product area, one QA team, and one defect workflow. Record the status definitions, the required fix evidence, and the reopen rules at the start. If the workflow changes during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A new required field can look like a change in defect quality when it is only a change in the form.
Define the population as every defect that moved to verified-fix during the window, including defects that were later reopened and those that stayed closed. The unit is the defect, not the status transition, because one defect can be reopened several times. A duplicate merged into the primary defect remains one unit. A separate report of a different behavior is a new unit.
Record the reporter, the engineer, the QA owner, the severity, and the affected area. These strata matter because a payment failure and a label alignment carry different reopen costs. Include defects that were closed as cannot-reproduce, because that outcome is part of the quality signal. Test the unit definition on two ordinary defects before collection begins. Use the same severity scale as the queue.
Define a verified fix as a change that passed the stated acceptance check and was closed by the QA owner. Define a reopen as a new defect record or status change that cites the original fix and shows the behavior still failing. Define a new symptom as a related but distinct failure that does not reproduce the original steps. Define a changed rule as a reopen caused by a new acceptance criterion rather than a failed fix.
Do not count a reopen caused by a deployment that never included the fix unless the record shows the fix was claimed as deployed. Do not count a customer request for a new feature as a reopen. A test that passes only on the engineer's machine is not verified evidence. Each definition should include one positive example and one boundary example so a second reviewer can classify consistently.
For each defect, capture the id, reporter, engineer, QA owner, severity, area, close date, and reopen date. Add the acceptance check, the evidence attached, and the deployment containing the fix. Record the reason category and whether the original steps still fail. Keep the defect packet with the sample so a second reviewer can repeat the classification. Store the packet with the defect record.
Practical application
Also capture whether the fix changed a test and whether that test fails on the pre-fix commit. These fields help explain a reopen that happened because the original test checked a nearby behavior. If the evidence was a screenshot, record the environment and the build. A second reviewer should be able to tell whether the original close was supported by the record.
Draw the sample from the defect tracker rather than from a list of known bad fixes. Record the export date, the filters, the row count, and the selection rule. Include defects from every severity and every QA owner so the result is not dominated by one noisy area. If the population is ordered by close date, a fixed interval spreads the sample across the window.
Collect the defect as it existed at the close decision, not as it looks after later comments. If the acceptance check changed after the close, record the change time and analyze the review separately. Do not infer a failed fix from a later unrelated incident. Note the collection date separately from the close date so the reader can see how much of the record has aged.
A useful additional measure is the share of fixes that included a regression test. A fix without a test may still be correct, but the team has less evidence that it will stay correct. Show the test status beside the reopen rate and describe the relationship. That view helps the owner decide whether the gap is in testing, deployment, or the acceptance rule.
Report the number of defects, the reopen rate, and the median time to reopen. Show the counts by severity, area, engineer, and QA owner. Present the share of reopens caused by a failed fix, a new symptom, and a changed rule. Use the same denominator for every rate and show the number of defects with incomplete evidence. Show the reopen count beside each rate.
Inspect the reopens and describe the pattern without naming the engineer. A cluster in one area may point to an unstable environment, while a cluster in one QA owner may point to a different definition of verified. Compare the result with the severity and the presence of a regression test. Report cannot-reproduce closures separately because they carry a different risk.
A useful additional view is the time between close and reopen. A reopen within one day may indicate a deployment mismatch, while a reopen after two weeks may indicate a test that missed a slower failure. Show the distribution and describe the pattern. That view helps the owner decide whether to add a monitoring window before closing. Name the deployment in the reopen record.
Operating controls
The review should separate a failed fix from a fix that was never deployed. A defect closed against a branch that did not reach production is a release-control problem, not a QA problem. Record the deployed artifact and the close time. That split prevents the team from rewriting tests when the real gap is in deployment verification. Record the deployed artifact for every close.
The record cannot show whether the engineer understood the root cause or whether the QA owner tested the right path. It can show only the acceptance check, the evidence, and the later status. A reopen is not proof of a careless fix, and a closed defect is not proof of correctness. The protocol also cannot capture a customer who experienced the failure but did not report it.
Defect trackers can omit comments from deleted accounts, merged duplicates, and status changes made by automation. A defect that was reopened and closed again within an hour may leave a thin record. Record those cases and their treatment instead of dropping them silently. If the engineer and QA owner are the same person, note the independence limitation. Keep the original status history with the sample.
A useful check is whether the acceptance evidence names the environment and the build. A fix verified only in a local environment cannot support a production close. Record the build and the deployment. That gap is a release-verification issue that the reopen review can surface before a customer does. Record both fields in the evidence attached to the defect close.
Give the result to the QA lead and the engineering lead and ask them to choose one observable change. The change might be a required regression test, a monitoring window before close, a deployment check, or a clearer acceptance template. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the template and the deployment rule at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If reopens fell but close time rose, the team may be adding a useful monitoring window. If reopens fell but escaped defects rose, the QA check may have become too narrow. Close the review when the owner accepts the finding and the next sample is scheduled.
The final view is the reporter's experience after the fix. Ask whether the customer was told what changed and how to confirm it. A defect can be technically closed while the reporter still cannot use the product. Record the confirmation method and the date. That view keeps the reopen measure connected to the customer outcome rather than the tracker status.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
How often do outsourced pull requests arrive without a reproducible test?
Where does onboarding time go for a new outsourced engineer?