On this page
Research context
A coverage claim usually arrives as one sentence: the change is fully covered. This protocol turns that sentence into a set of records a reviewer can verify, including the tool and version, the configuration, the command, the report, the exclusions, and the behaviour the tests actually assert. Without those records the claim is a statement of confidence rather than a measurement.
The question is not whether a coverage percentage is high. It is whether the claim attached to a specific change is supported by a reproducible measurement, and whether that measurement relates to the behaviour the change introduces. A repository can report a healthy number while the new error path, permission check, or boundary case has never been executed by a test.
Define the claim type first. Line coverage, branch coverage, function coverage, and requirement coverage answer different questions. A claim of full coverage with no type is not falsifiable, and the first verification step is to ask the author to name the type, the tool, and the scope. If the author cannot name them, the claim should be restated before anyone spends time checking it.
The population is the changed files in one pull request, with the tests that ran against that revision. The observation unit is one changed file with its coverage report entry, the test command, and the relevant assertions. A repository wide percentage is context, not the unit, and it can rise while the new code is entirely untested if the change is small relative to the codebase.
Tool configuration belongs in the evidence. Coverage tools exclude files by default pattern, ignore generated code, and sometimes skip lines with special comments. Record the configuration revision and the exclusion list, because an exclusion can turn a red file green without a new test. A configuration change in the same pull request as the claim deserves a direct question about why.
Generated code, migrations, and vendored libraries are common exclusions. Each exclusion needs a reason and an owner. Some exclusions are entirely reasonable, such as a generated client with no branching logic, while others quietly remove the hardest file in the change. The reviewer should see the list and the reason, not just the adjusted percentage.
The command must be reproducible. Record the package manager, the test runner, the filter, the environment variables that affect selection, and the report path. A claim produced by a local command that nobody else can run is not evidence, and it is usually a command that skipped a suite because an environment variable was missing or a database was unavailable.
Practical application
Denominators are the changed executable lines in the declared scope. Report measured lines, covered lines, excluded lines, and files with no report entry. A file missing from the report is not the same as a file with zero coverage, and conflating them hides a configuration gap. When a report is incomplete, the honest claim is partial rather than complete.
Coverage measures execution, not assertion. A test that calls a function and checks nothing can cover every line in it. Ask for the assertions attached to the changed behaviour, and check at least one case where the assertion would fail if the behaviour regressed. A reviewer who reads only the percentage is reading an activity log, not a test result.
Build an adverse fixture. Introduce a deliberate defect in a copy of the changed code, such as an inverted condition or a removed null check, and confirm the test suite fails. If the suite still passes, the coverage number describes execution and not protection, and the claim should be restated as execution coverage with a note about the assertion gap.
Requirement coverage needs its own table. List the acceptance rules from the ticket and the test that demonstrates each one. A rule with no test is an open item even when line coverage is complete, and a test that covers two rules should say so. This table is usually more useful to a product owner than any percentage, because it maps to behaviour they recognise.
Review the boundary cases. Coverage reports do not distinguish an input at the boundary from an input in the middle. Ask which test covers the empty value, the maximum length, the permission denial, and the error path introduced by the change. If the boundary case is covered by accident rather than by a named test, the protection is fragile.
Snapshot and visual tests deserve a note, because they can raise a line count without testing a decision, a permission, or an error message. They can cover rendering paths without asserting behaviour that matters to a user. Count them as evidence for the surface they check and not as evidence for logic in the same file. A snapshot that updates automatically on change is a record of output, not a check of intent.
Flaky tests distort the claim. If a test was skipped, retried until green, or marked pending in the same revision, the effective coverage is lower than the report suggests. Record skipped and retried tests in the evidence table, and treat a retry that only passes on the second attempt as an unresolved signal rather than a clean pass. Record the retry count and the reason in the same row as the result, because a suite that needs two attempts has an unresolved flake and a coverage claim that depends on it is weaker than the report suggests. The reviewer does not need to fix the flake in the same change, but the claim should carry the caveat.
Operating controls
Check the merge result, not only the branch. Coverage measured before the merge can differ after conflict resolution. Rerun on the merged revision when the change touches shared modules, because a conflict resolved by deleting a test is exactly the case a pre-merge report cannot see.
Independence is practical rather than perfect. A second reviewer reruns the command on the same revision and compares the report hash or the per file numbers. They do not need to read every test, but they should sample the assertions for the highest risk file and confirm the adverse fixture fails as expected on their own run.
Report a small table: file, claim type, covered lines, total lines, exclusions, assertion sample, adverse fixture result, and reviewer note. The table is more useful than a badge because it shows where the claim is thin, and it gives the next reviewer a place to start rather than a single number to trust.
Limits must be stated. Coverage does not measure whether the tests are meaningful, does not prove the absence of defects, and does not transfer across languages or frameworks. A high number on one service says nothing about another service or another team. Security and privacy apply to test artifacts as well: reports can include file paths and code fragments, so keep them in the repository or approved storage and avoid customer data in fixtures.
The stopping rule is simple. Stop when the claim type is undefined, when the command cannot be rerun, or when the report is missing files in scope. An unverifiable claim should be restated as unverified rather than accepted with a caveat in a comment. Sensitivity checks are useful before publishing: recompute with exclusions treated as uncovered, with generated files included, and with skipped tests counted as failures. If the conclusion changes, the claim depends on a definition that the reviewer should see.
The repair action is usually small: add a boundary test, remove an unnecessary exclusion, fix a report path, or restate the claim as partial. Name the owner and the date, and recheck the specific file rather than rerunning the whole suite for a headline. The method reports no local coverage percentage and cites no benchmark. The sources describe verification, review, and secure development practice; they do not measure this team. Acceptance is a reviewer who can rerun the command, match the per file numbers, see the exclusion list, and find the assertion that would fail if the behaviour broke.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
Defect Escape Attribution: A Method for Bounded Records