Table of contents
Research context
This protocol asks how much stale feature-flag debt accumulates in an outsourced codebase and what kind of risk it carries. The unit is one feature flag that exists at the observation date. The study measures age, ownership, removal condition, current default, and whether the flag still changes a production path. It does not evaluate whether the feature behind the flag was successful.
Set the scope to one repository, one deployment platform, and one flag service. Record the flag SDK, the evaluation model, and the inventory source at the start. If the team changes flag providers during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A migration can look like a reduction in debt when it is only a different inventory.
Define the population as every flag that can be evaluated in the production application during the window, including flags that are always on or always off. The unit is the flag key, not the code reference, because one flag can appear in several services. A flag that is removed and later re-added is a new unit. A renamed flag with the same behavior is a new unit.
Record the flag type, owner, creation date, default value, and affected services. These strata matter because a release flag and a permanent kill switch carry different lifetimes. Include flags with no owner, because the absence is part of the finding. Test the unit definition on two ordinary flags before collection begins. Use the same categories for every single inventory row.
Define a removal condition as a written event or date after which the flag should be deleted. Define stale as a flag past its removal condition or older than the team's maximum release-flag age with no documented reason to keep it. Define active exposure as a flag whose value can still differ between environments or customers. Define orphaned as a flag with no named owner in the inventory or the repository.
Do not count a configuration value as a feature flag unless it changes application behavior through a flag evaluator. Do not count a database column that records a user preference as a flag. A flag that is permanently on but still read at runtime is not automatically stale if it supports an operational control. Each definition should include one positive example and one boundary example.
For each flag, capture the key, type, owner, creation date, default, affected services, and last change date. Add the removal condition, the current evaluation state, and the number of code references. Record whether the flag is documented in the runbook. Keep the inventory extract with the sample so a second reviewer can repeat the count. Store the extract date with the inventory.
Practical application
Also capture whether the flag was used in the last quarter and whether its off state has a test. These fields help explain a flag that is old but still valuable. If usage is unknown, record the gap rather than assuming the flag is dead. A second reviewer should be able to classify the flag from the inventory and the repository.
Draw the sample from the flag service and the repository rather than from a list of known cleanup candidates. Record the export date, the filters, the row count, and the selection rule. Include flags from every service and every owner so the result is not dominated by one neglected area. If the inventory is ordered by creation date, include every flag in the window.
Collect the flag as it existed at the observation date, not as it looks after a later cleanup. If a flag was removed during the study, record the removal date and analyze it as a separate event. Do not infer staleness from the feature name alone. Note the collection date separately from the flag creation date so the reader can see the age distribution.
A useful additional measure is the share of stale flags that still have a production code path. A stale flag that is always on may be harmless, while one that can still route traffic is a live control with no owner. Show the code-path count beside the age. That view helps the owner prioritize removal by exposure rather than by count.
Report the number of flags, the share stale, the share orphaned, and the share with active exposure. Show the counts by type, service, and owner. Present the median age and the number of flags past their removal condition. Use the same denominator for every rate and show the number of flags with missing metadata. Show the raw flag count beside every rate.
Inspect the stale flags and describe the pattern without naming the engineer. A cluster in one service may point to a missing cleanup step in the release template, while a cluster in one owner may point to a team that changed. Compare the result with the flag type and the last change date. Report permanent controls separately so they do not inflate the debt count.
A useful additional view is the time between the last code change and the flag creation date. A flag added during a large migration may be intentionally long-lived, while one added for a one-week rollout should be gone. Show the age and the original release note. That context prevents a simple age threshold from misclassifying a legitimate control. Quote that note in the inventory row.
Operating controls
The review should separate flag debt from configuration drift. A flag that is always on because the default changed may be dead code, while a flag that is always off because a customer segment never launched may still be a product decision. Record the reason and the owner. That split prevents the team from deleting a path that a contract still requires.
The record cannot show whether a flag is safe to remove without reading the code and testing the off state. It can show only the metadata, references, and stated removal condition. A stale flag is not proof of a bug, and a young flag is not proof of good hygiene. The protocol also cannot capture flags evaluated outside the application, such as in a CDN or a vendor console.
Flag inventories can omit flags created directly in the vendor console, and repository searches can miss dynamically constructed keys. A flag may also be referenced in a mobile build that the server inventory does not see. Record those gaps and their treatment instead of dropping them silently. If the same person owns the flag and approves its removal, note the independence limitation.
A useful check is whether the off state is tested in the same environment where the flag is evaluated. A unit test with a mocked evaluator does not prove that the production fallback works. Record the test level and the environment. That gap is a control problem that the cleanup review can surface before a release. Record the test level and the environment.
Give the result to the engineering lead and the service owners and ask them to choose one observable change. The change might be a required removal date on every release flag, a quarterly cleanup review, an owner field in the flag template, or a dashboard for stale flags. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the template and the cleanup cadence at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If stale flags fell but orphaned flags did not, the team may be deleting code without assigning ownership. If orphaned flags fell but active exposure did not, the inventory may be improving without the runtime risk. Close the review when the owner accepts the finding and the next sample is scheduled.
The final view is the cost of keeping the flag, not only the risk of removing it. A flag that requires a test matrix, a dashboard, and a runbook entry consumes attention even when it is always on. Record the maintenance work attached to each stale flag. That view helps the owner decide whether the next cleanup should start with the oldest flag or the most expensive one.
Sources
Frequently asked questions
What should happen first?
Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.
Who approves production changes?
The client's technical owner keeps final merge and release authority.
Related research
Where does review time accumulate in a distributed engineering team?