Table of contents
- Quick read
- Readiness check
- Define the behavior the switch must stop
- Pick the smallest switch that isolates the risk
- Test the off state on a production-like copy
- Measure how fast the switch takes effect
- Keep the switch until the risk is gone
- Review the switch inventory each release
- Write the flip and rollback note
- Role brief
- Frequently asked questions
- Sources
Is the handoff ready?
Use these checks before access expands or the first ticket begins.
Stop condition
Ready: State the customer behavior the switch must stop before choosing a flag.
Needs work: The lane has no written record for this check.
Safe default
Ready: Keep the default off when the flag service is unavailable or slow.
Needs work: The lane has no written record for this check.
Off-state proof
Ready: Test the server path, jobs, and messages on a production-like copy.
Needs work: The lane has no written record for this check.
Effect time
Ready: Measure and record the delay from flipping the switch to the customer result.
Needs work: The lane has no written record for this check.
Define the behavior the switch must stop
Write the behavior the switch must stop in customer terms before choosing a flag. For example, the switch should stop new checkout attempts while keeping order history available. Record the endpoints, jobs, and messages that must also stop. A switch that hides a button but leaves the background job running has not stopped the risk.
Name the risk the switch addresses, such as a slow vendor, a new pricing rule, or an unfinished migration. Record the release that introduces the risk and the owner who accepts it. Set the date when the flag should be removed. A permanent switch is a configuration branch that nobody reviews.
Record the internal jobs and messages that must stop, not only the web request. A background worker can keep charging cards after the button is hidden. List the queue consumer and the scheduled reconciliation too. The stop condition should cover every path that can repeat the risky action.
Pick the smallest switch that isolates the risk
Choose the narrowest switch that isolates the risky behavior. Prefer one flag around one code path rather than a flag that changes the whole application. Record the default state and the state for each environment. The default must be safe when the flag service is unavailable.
Keep the flag evaluation on the server where the decision affects money or permissions. Record the fallback path when evaluation fails or times out. Test the fallback with the flag service blocked. A kill switch that depends on the same service as the feature is not independent.
Test the off state on a production-like copy
Test the off state on a production-like copy before the release reaches customers. Confirm that the blocked path shows the intended message and that the rest of the journey still works. Check the background workers and the scheduled jobs. A switch that only covers the browser leaves the server path open.
Record the test data, the accounts, and the result. Repeat the test after any change to the flag or the protected code. Store the evidence with the release ticket. The off state is the part customers will see during an incident.
Measure how fast the switch takes effect
Measure the time from flipping the switch to the customer-visible effect. Record the cache layer, the polling interval, and the number of instances. Test with the same traffic pattern the release expects. A switch that takes twenty minutes is a different tool from one that takes seconds.
Write the command or console path the on-call engineer will use. Confirm that the engineer has permission before an incident. Record the person who verified the permission. A runbook step that fails on access is not a runbook.
Record the result in the release ticket and compare it with the incident target. If the switch is slower than the target, change the mechanism or the target before the release. A measured delay is easier to defend than an assumed one. The on-call engineer should know the number before the incident.
Keep the switch until the risk is gone
Keep the switch while the risk is live and remove it when the risk ends. Record the removal criteria and the owner. Check that the protected code still behaves correctly after removal. A flag left in place becomes an untested branch.
Review the flags that affect checkout, payments, and access each release. Remove the ones whose risk has passed and record the removal. Keep a short list of emergency switches with a named owner. The shorter the list, the easier it is to test.
Record the test that proves the code works without the flag. Remove the flag in a small change that can be reviewed on its own. Watch the protected path for one release before closing the removal ticket. A flag removed without a check can hide a second dependency.
Review the switch inventory each release
Keep an inventory with the flag name, the owner, the risk, the default, and the removal date. Review it before each release and after each incident. Remove flags that no longer have a code path. An inventory that disagrees with the code is worse than no inventory.
Sample one kill switch each quarter and run the off-state test end to end. Record the result and the slowest step. Give the finding to the release owner. A tested switch is the only kind that can be trusted during an incident.
Write the flip and rollback note
Write the exact steps to flip the switch, the person who can do it, and the expected customer message. Include the steps to turn the feature back on after the risk passes. Record the evidence that confirms each state. A switch with no rollback note invites a second outage when the team restores the feature.
Keep the note beside the on-call runbook and link the dashboard that shows the protected path. Test the link and the permission before the release. A runbook that points to a deleted dashboard is worse than no link. Review the note whenever the flag provider or the protected code changes.
Copy-ready brief
Put the operating agreement in writing.
State the customer behavior the switch must stop before choosing a flag.
Keep the default off when the flag service is unavailable or slow.
Test the server path, jobs, and messages on a production-like copy.
Measure and record the delay from flipping the switch to the customer result.
Frequently asked questions
What if the flag service is down?
The default must be safe, and the fallback path must be tested with the service blocked.
Who should be able to flip the switch?
The named on-call engineer, with permission verified before an incident rather than during one.
When should the flag be removed?
When the risk ends and the protected code is verified without the flag.
Sources
- Martin Fowler: feature toggles
Toggle categories, lifecycle, and operational tradeoffs.
- OpenFeature specification
Feature flag evaluation concepts and provider behavior.
- Google SRE Book: release engineering
Release control and rollback practice.
- NIST Secure Software Development Framework
Secure release and configuration management practice.
