Table of contents
Research context
This protocol measures the share of feature flag kill switches that are tested before a release in an outsourced delivery lane. The unit is one switch on one release at the observation date. A tested switch means the team proved the off state on a production-like copy and measured the time to effect. The study describes release control evidence, not the quality of the feature behind the flag.
Define the releases and the flag systems in scope before collection. Include server flags, client flags, configuration switches, and any emergency control that can stop a customer path. Record the flag provider and the repository that evaluates each switch. Publish the list with a version so a later reviewer can reproduce the scope.
Define a switch by its name, owner, default state, protected path, and removal date. Record the server and client behavior when the flag service is unavailable. Note the background jobs and messages the switch must also stop. Test the definition on the first five switches.
Set the population as every switch attached to a release in the agreed window. Include switches that were created but never enabled because those still carry a default. Record the switches that could not be inspected and the reason. Report them separately from switches that failed a test.
Classify each switch by test evidence. Use categories such as tested with recorded evidence, tested on the client only, not tested, and no named owner. Do not merge the categories after seeing the result. The categories point to different fixes.
Record whether the off state was tested on the server path and in the background jobs. Note the test environment, the accounts, and the time from flip to effect. Record the person who verified the permission to flip the switch. A switch that only hides a button is not fully tested.
Report the number of switches, the number tested with evidence, and the number in each category. Show counts by customer path and by release. Present the switches with no owner and the switches with a failed test as separate rows. Use the same denominator for every rate.
Separate an untested switch from a switch that failed its test. An untested switch has no evidence, while a failed switch has evidence and a defect. Report the two categories separately because the fixes differ. Record the failed step and the test date beside each finding.
Inspect the switches that protect checkout, payments, and access. A cluster of untested switches may mean the release process treats flags as a development convenience rather than a control. Compare the pattern with the release history. A single gap can be a mistake, while a cluster is a process gap.
Practical application
Do not treat a flag that exists as proof that it can stop the risk. The switch may depend on the same provider as the feature or may only affect the browser. Check the fallback path when evaluation fails. Report the dependency beside the result.
The record cannot show whether an incident would be handled well or whether the switch would be flipped in time. It can show only whether the off state was tested and measured before the release. An untested switch is not proof of an outage, and a tested one is not proof of a safe feature. The protocol also cannot see a switch created outside the flag system.
A useful additional view is the time from flipping the switch to the customer-visible effect. Record the cache layer, the polling interval, and the instance count. A switch that takes twenty minutes changes the response plan. That field turns a yes-or-no count into a review of incident speed.
Record the removal date and the owner for each switch. A switch that stays after the risk ends becomes an untested branch. Check the inventory against the code paths each quarter. Report the switches that have no removal date.
Check the default state when the flag service is unavailable. A default that enables the risky path turns a provider outage into a feature outage. Test the fallback with the provider blocked. Report the switches whose safe state depends on the provider being healthy.
Give the result to the client technical owner and the offshore delivery lead with one observable change. The change might be an off-state test, a safe default, a named owner, a permission check, or a removal review. Record the owner, the effective date, and the measure that will show whether the change worked. Repeat the protocol with the same definitions after the change.
Close the review with one change and a date to re-measure. The change should address the largest untested or ownerless group, not the easiest switch to test. Record the expected effect and the evidence that will prove it. A kill switch that is never tested is a configuration setting, not a control.
Build the switch sample from the flag provider and the release records, not from the code alone. The provider shows what is evaluated, while the release record shows what was tested. Compare the two and record the switches that appear in only one place. The mismatches are often the oldest flags.
Use two evidence sources for each switch, such as the release ticket and the off-state test result. The ticket proves the switch was part of the release, while the result proves someone tried the off state. Record the test environment and the account used. A switch without evidence is untested by definition.
Operating controls
Switches on the checkout path are overrepresented in incident reviews because failures are visible. Include a switch that protects a quiet internal path and one that was never enabled. Record the last evaluation date. An unused switch with an unsafe default is still a risk.
Record the default state, the fallback path, and the time to effect for each switch. A switch that depends on the provider being healthy may fail open during the outage it was meant to handle. Test the fallback with the provider blocked. Report the dependency beside the result.
Report the result by customer path as well as by count. A switch around a payment action deserves a stricter test than one around a recommendation widget. Record the protected path beside the category. The reader should see where an untested switch would hurt first.
Repeat the measure after a flag provider change or a release process change. Keep the same categories and the same test definition. Record the switches that moved between categories. A stable count can still hide a new dependency.
Check the removal date and the owner for each switch. A switch that stays after the risk ends becomes an untested branch that future engineers must reason about. Record the last review and the code path it protects. Report the switches with no removal plan.
Treat the protocol as a review of release control, not a judgment of the feature team. An untested switch often means the release checklist treats flags as development tools. Ask which switch would be flipped first in an incident and who has permission. The answer shows whether the control is real.
Record the flag evaluation logs for the release window. The logs show whether the switch was actually evaluated by the path the team expected. A switch that never appears in the logs may be bypassed by a cache or a client build. Report the evidence beside the test result.
Set a quarterly test calendar for the emergency switches and keep it with the release schedule. A calendar makes the test visible before an incident rather than after one. Record the owner and the result for each entry. The calendar is the difference between a control and a document.
Record the person who last reviewed the switch and the evidence they saw. A review that only reads the flag name is not a review of the protected behavior. Ask them to describe the customer effect of the off state. Report the switches whose reviewer could not explain it.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
