Test Third Party API Outage Fallbacks With Named Owners

Is the handoff ready?

Use these checks before access expands or the first ticket begins.

Classify every vendor call by consequence

Start with an inventory, not a strategy. List each call, the data it returns, the screen or job that depends on it, and what the user sees when it fails. A payment authorisation, a shipping quote, and a profile avatar have nothing in common during an outage.

Group the calls into four classes: blocking, degradable, deferrable, and cosmetic. The class decides whether the product waits, shows a cached value, queues the work, or hides the feature.

A Philippines-based programmer can build that inventory from the code in a day. The owner then reviews the classes, because the classification is a product decision disguised as a technical list.

Include the calls made by background jobs and admin tools, not only the customer facing screens. An outage that stops a nightly reconciliation is a different problem from one that hides an avatar, and both belong on the same list.

Write one ticket per failure class

The ticket names the calls in the class, the timeout, the retry policy, the fallback state, the user visible message, and the person who owns the vendor relationship. Ask for a test command that can force each failure mode.

Do not mix a blocking payment path with a cosmetic avatar in one change. The review depth and the blast radius are different, and a shared helper can accidentally apply a lenient fallback to a strict path.

Record the vendor contract terms that matter: rate limits, retry guidance, status page, support channel, and notice period. The programmer should not interpret the contract for the company.

Inject the failures the vendor will actually produce

A real outage is rarely a clean connection refusal. It is a slow response that holds a worker, a gateway timeout, a five hundred error on one endpoint, and partial data for a subset of identifiers.

Build a stub or proxy that can delay, fail, and truncate responses. Run the same user action against each mode and record the state, the elapsed time, and what the user saw.

Test the recovery as well. When the vendor returns, the queued work should drain in a bounded way rather than all at once.

• Timeout at the client threshold with no response body.
• Slow response beyond the worker or request budget.
• Five hundred on one endpoint while others succeed.
• Truncated payload with a success status.

Design the fallback and the recovery state

A circuit breaker is useful only if the open state has a defined behaviour. Write what happens for the first failure, after the threshold, and during the half open probe. Each state needs a log line and an owner.

Cached data needs a maximum age. A shipping quote from yesterday may be acceptable while a stock count is not. State the maximum staleness per class and show the timestamp to the user when the value is old.

Queued work needs a bound. If the vendor is down for six hours, the queue must not grow without limit or replay into a rate limit. Set a cap and a discard policy with the owner.

Record the state, the message, and the recovery

Evidence is a table per scenario: call, injected fault, breaker state, response time, user visible result, queued items, and recovery outcome. Add the exact stub configuration so the reviewer can repeat it.

Include the negative case. A cosmetic call should degrade quietly, while a blocking call should stop with a clear message. If both behave the same, the classification was not implemented.

Record the customer message and who approved it. Wording during an outage is a company responsibility, even when the programmer implements the banner.

Check retry storms, webhook backlog, and partial writes

Retries without jitter and a cap turn a vendor slowdown into a self inflicted load problem. Test the retry path with a hundred queued items and confirm the pacing.

Vendor webhooks can arrive late and out of order after an outage. The handler needs an idempotency rule and a reconciliation job with an owner.

A partial write is the worst case: the vendor accepted the request but the response was lost. Test the reconciliation query for the identifier rather than a blind retry.

• One hundred queued requests during a two minute outage.
• Late webhook after the local state already moved on.
• Vendor accepted the write but the client saw a timeout.
• Vendor rate limit response during recovery.

Measure fallback use and recovery time

Measure requests served by fallback divided by requests attempted during an outage, queued items drained divided by items queued, and recovery time from vendor return to normal state. Report the window and the affected class.

Track the count of outages discovered by customers rather than by monitoring. That number says more about detection than any average uptime figure.

Report the fallback quality beside the fallback rate. A cached price shown without its age is not a graceful degradation, and the number should not count it as a success.

Review after every outage that touched a vendor call and on a quarterly cadence for the inventory. A vendor change or a new endpoint invalidates the old rehearsal.

Close with a rehearsal packet per failure class

The packet holds the call inventory, the class, the timeout and retry settings, the fallback state, the injected scenarios, the user messages, and the unresolved dependencies. It names the vendor owner and the next review date.

Accept the work when an on call engineer can read the packet during an incident and know which path is active without opening the code.

Add the vendor status page and support contact to the packet. During an outage the on call reader should not have to search for a phone number.

Frequently asked questions

Sources

1. OpenTelemetry documentation

   Tracing and metrics for vendor latency, errors, and breaker state.

2. NIST Cybersecurity Framework 2.0

   Response and recovery framing for dependency outages.

3. SLSA specification v1.0

   Provenance and build integrity concepts for the deployed fallback.

4. CISA Secure by Design

   Guidance for resilient defaults and clear failure behaviour.