On this page
Research context
This protocol asks whether an accessibility fix closes the original user barrier or only the symptom that was reported. The unit is one accessibility fix that shipped during the observation window. The study compares the original barrier, the code change, the assistive-technology evidence, and the re-test result. It does not evaluate the severity of the barrier or the quality of the design.
Set the scope to one product area, one component library, and one accessibility standard. Record the assistive technologies, browsers, and test scripts in force at the start. If the standard changes during the window, keep the earlier definition for the full sample or analyze the change separately. A new success criterion can look like a new defect when it is only a new rule.
Define the population as all accessibility fixes that were merged or released during the window, including fixes that were later reverted. The unit is the fix, identified by the ticket and the component. A fix that is reopened after a regression remains one unit with a revision history. A separate fix for a different barrier is a new unit.
The re-test should include at least one task that was not in the original report, because a fix can solve the reported element while leaving a related control unusable. Choose a task that uses the same component in a different context, such as a second form or a different page family. Record the result beside the original evidence. The extra task is a small cost compared with a second round of fixes. The task should be chosen before the fix is merged so the re-test has a defined scope.
Record the component, the page family, the assistive technology, and whether the component is shared. These strata matter because a shared component change can affect many flows while a page-specific fix affects one. Include fixes that were closed without a re-test, because the closure reason is part of the evidence. Test the unit definition on two ordinary fixes before collection begins.
Define the original barrier as the user action that failed, the technology used, and the expected result recorded before the fix. Define closure as the condition where the same action succeeds with the same technology and the regression surface passes. Define a symptom-only fix as one where the reported element changed but the task still fails. Each definition should include one positive example and one boundary example.
Do not count a scanner score as closure evidence. Do not count a code review as closure evidence unless the reviewer also ran the interaction. A fix that works with a mouse but not a keyboard is not closed for keyboard users. The coding rules should distinguish a partial improvement from a completed barrier.
Practical application
A shared component fix should update the component's documentation and examples, not only the code. The next engineer who uses the component needs to know the expected keyboard behavior and the focus style. Record the documentation change and the date it was published. A fix that lives only in the code will be undone by the next implementation.
For each fix, capture the ticket, the component, the page family, the technology, the original evidence, and the change. Add the re-test evidence, the regression pages, the result, and the person who accepted it. Record whether the fix changed a shared component and whether the design system was updated. Code each field as present, missing, or not applicable.
Also record whether the original reporter was involved in the re-test and whether the fix introduced a new barrier. These fields help explain a closure that looks complete but leaves a related problem. If the evidence is a recording, record the tool and the timestamp. A second coder should be able to reproduce the classification from the packet.
Draw the sample from the accessibility backlog and the release log rather than from a list of notable fixes. Record the export date, the filters, the row count, and the selection rule. Include fixes from different components and page families so the result is not dominated by one easy case. If the population is ordered by release date, a fixed interval spreads the sample across the window.
The acceptance record should include the user's perspective when the original report came from a customer. A short follow-up with the reporter can confirm that the task now works in their environment. If the reporter is not available, record that limitation. The team should not claim closure on behalf of a user it did not ask.
Collect the evidence as it existed at the re-test, not as it looks after later edits. If the fix was amended after the first re-test, record the amendment and analyze the original result separately. Do not reconstruct a missing recording from memory. Note the collection date separately from the release date so the reader can see how much of the record has aged.
Report the share of fixes with a completed re-test, the share that closed the original barrier, and the share that closed only the symptom. Show the counts by component type and by assistive technology. Present the share of shared-component fixes that updated the design system. Use the same denominator for every rate and show the number of fixes with incomplete evidence.
Operating controls
Inspect the symptom-only fixes and describe the pattern. A cluster in one component may point to a markup pattern, while a cluster in one page family may point to a content problem. Compare the result with the regression surface, because a fix that holds on the reported page but fails on a reused flow is not closed. Report the reverted fixes separately.
Accessibility regressions should be reviewed alongside other quality metrics, not as a separate annual project. Add the fixed flows to the regular regression suite and report the pass rate with the release notes. A metric that is reviewed every release is more likely to stay healthy. The review should name the owner and the next check.
A useful additional view is the time between the fix and the re-test. A re-test performed months later may miss a regression that appeared in the meantime. Plot the timing and describe the cases where the re-test came late. The comparison should guide the release checklist rather than judge the engineer.
The record cannot show whether the fix improved the experience for every user or whether a different barrier remains. It can show only whether the reported task succeeded with the tested technology. A closed ticket is not proof of accessibility, and an open ticket is not proof of failure. The protocol measures a specific barrier, not the whole product.
Backlogs can omit fixes that were made without a ticket, and release logs can omit hotfixes. A component that was patched in a design-system release may not appear in the sample. Record those cases and their treatment instead of dropping them silently. If the same person wrote the fix and accepted it, note the independence limitation.
Give the result to the accessibility lead and the engineering manager and ask them to choose one observable change. The change might be a required re-test recording, a shared-component checklist, or a design-system update. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the standard and the test script at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If closure improved but symptom-only fixes did not, the re-test may be catching the easy cases. If shared-component updates improved but regressions did not, the design system may be lagging the code. Close the review when the owner accepts the finding and the next release is scheduled. A regular review also gives the team a place to record fixes that were made without a ticket.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
How much third-party SDK risk is visible before adoption?
How reproducible are bug reports before outsourced triage begins?