On this page
- Quick read
- Readiness check
- Write the off state before the on state
- Assign the switch to a person, not a team
- Rehearse with cohorts, cache, and in-flight work
- Capture evaluation and behaviour together
- Plan for the flag service being down
- Keep production authority with the company
- Set the rehearsal cadence and useful measures
- Close with a runbook line a stranger can follow
- Role brief
- Frequently asked questions
- Sources
Is the handoff ready?
Use these checks before access expands or the first ticket begins.
Default
Ready: Off state is safe and documented per cohort
Needs work: Off state was never exercised
Owner
Ready: Named operator and backup with console access
Needs work: Anyone with repository access
Rehearsal
Ready: Sandbox flip with cohort and cache checks
Needs work: A local unit test only
Record
Ready: Timestamp, operator, cohort, observed behaviour
Needs work: A message saying it worked
Write the off state before the on state
A kill switch is a product promise about what happens when a capability disappears. Write the promise per surface: does the button hide, does the request fail with a specific message, does the queue drain, does the old code path still exist and still work?
Many launches test the enabled path and leave the disabled path to a one line condition. That condition becomes the incident response. A Philippines-based programmer should be able to read the off state, find the code path, and show it running in a sandbox.
Write the off state in user terms as well as code terms. If the button disappears, does the page explain why, or does it look broken? A kill switch that stops the feature and confuses the customer has moved the incident rather than ended it.
If the fallback needs a manual step, name the person who performs it and the time it takes. If the fallback is impossible, say that the flag stops writes but leaves reads on the new path, and record who accepted that risk.
Assign the switch to a person, not a team
The ticket names the flag key, the default value, the cohorts, the surfaces it controls, the operator, the backup operator, and the escalation channel. It also records who can change the default in code and who can change it in the flag service.
Two systems can hold the same decision: a configuration service and a hard coded default. When they disagree, behaviour becomes unpredictable. Declare the source of truth and what happens when the service is unreachable.
Give the programmer sandbox flag rights and read access to the evaluation logs. Production flag changes stay with the company operator, even when the change is a rollback.
Rehearse with cohorts, cache, and in-flight work
A flip in a local test proves the conditional. A rehearsal proves the system. Use a sandbox with two cohorts, warm the cache, start a request, flip the flag, and observe both the request already in flight and the next one.
Flags are often cached per process or at the edge. Record the cache lifetime and the observed delay before the new value takes effect. If the delay is five minutes, the incident runbook must say five minutes rather than imply an instant stop.
Include a job or message already queued under the enabled path. Decide whether it completes, is discarded, or is processed by the new path, and record the decision.
- Flip on to off for one cohort while a second cohort stays on.
- Flip with a warm cache and measure the propagation delay.
- Flip during an in-flight request and record the response.
- Flip twice in one minute and check for duplicate side effects.
Capture evaluation and behaviour together
The evaluation log shows which variant a request received. The application log shows which branch ran. The database or queue shows whether the side effect happened. Keep all three for the same request identifier.
Ask for a short table: time, actor, flag value, cohort, request identifier, branch, side effect, and observed user-visible result. This table is what an incident reviewer will ask for later.
A passing rehearsal also needs a negative case. Show a request that should not have received the new path and confirm it did not. Otherwise the test only proves that the flag exists.
Plan for the flag service being down
The kill switch depends on a service that can fail. Test what happens when the flag service times out: does the application use the last known value, a cached value, or the local default? Each choice has a different risk and needs an owner decision.
Test a stale value as well. If the service returns the last known good configuration after an outage, a recent rollback may be undone silently. Record the maximum staleness the product can tolerate.
Test permission loss. If the operator loses console access during an incident, the backup must be able to act without a new approval chain. Write that path before launch.
- Flag service timeout with a cached value available.
- Flag service returns a stale value after recovery.
- Operator account locked during an incident.
- Default changed in code while the service holds a different value.
Set the rehearsal cadence and useful measures
Measure flip to effect time in seconds, successful fallbacks divided by all fallback attempts, and incidents where the off state was unavailable divided by all flips. Report the cohort size and the observation window.
Rehearse before each launch that adds a flag, after any change to the flag service or cache layer, and quarterly for flags that guard money, access, or customer data.
Review the flag inventory each quarter and delete flags that no longer guard anything. A stale flag is a hidden branch that nobody tests.
Close with a runbook line a stranger can follow
The handoff holds the flag key, the owner, the backup, the console path, the expected propagation delay, the off state per surface, the evidence table, and the flags still needing removal.
Accept the work when an operator who did not build the feature can flip the flag, see the fallback, and verify recovery using only the written record.
Note the flags that share a dependency with this one. Flipping two switches in sequence can produce a state that neither flag owner tested alone.
Copy-ready brief
Put the operating agreement in writing.
One key, declared default, cohorts, and controlled surfaces.
Cohort flip, warm cache, in-flight request, queued work.
Evaluation log, branch log, side effect state, user visible result.
Company operator flips production; programmer works in sandbox.
Frequently asked questions
Who is allowed to flip a production flag?
A named company operator and a backup, listed in the runbook with console access reviewed on a schedule.
How fast should a kill switch take effect?
Fast enough for the risk, and no faster than the cache and propagation delay allow. Record the real measured delay.
What if the flag service is unavailable?
The written rule decides between last known value, cached value, and local default. Test that rule before launch.
Sources
- CISA Secure by Design
Product design guidance that places security decisions with the owning organisation.
- SLSA specification v1.0
Provenance concepts for matching a running build to a reviewed revision.
- NIST Secure Software Development Framework
Environment separation and change control practices.
- GitHub pull request reviews
Review and approval mechanics for the flag guarded change.