Table of contents
- Quick read
- Readiness check
- Name the vendors that can stop the customer path
- Write the customer-visible symptom first
- Set the degraded mode the team can actually run
- Decide who declares an outage
- Record what to tell customers
- Test the note with a tabletop drill
- Keep the note beside the support queue
- Role brief
- Frequently asked questions
- Sources
Is the handoff ready?
Use these checks before access expands or the first ticket begins.
Vendor list
Ready: Name every vendor on the customer path and the action it can block.
Needs work: The lane has no written record for this check.
Symptom note
Ready: Write what the customer sees before describing the internal cause.
Needs work: The lane has no written record for this check.
Degraded mode
Ready: Record runnable steps, permissions, and the data the mode cannot accept.
Needs work: The lane has no written record for this check.
Decision owner
Ready: Name who declares the outage and the backup who acts without them.
Needs work: The lane has no written record for this check.
Name the vendors that can stop the customer path
Start with the vendors that can stop the customer path when they fail. Include payment, messaging, identity, mapping, shipping, and any API that sits inside checkout or login. Record the owner of each contract and the support channel. A fallback note for an unnamed vendor cannot be used during an outage.
Rate each vendor by the customer action it blocks and the time of day the failure would hurt most. Record whether the vendor publishes a status page and an incident history. Note the contract terms for support and credits. The rating tells the team where to spend rehearsal time.
Write the customer-visible symptom first
Describe the outage from the customer side before describing the internal cause. Write what the customer sees, such as a failed payment or a missing confirmation. Record the support language that matches the symptom. The support team should not have to translate engineering terms during a live event.
List the internal signals that confirm the vendor is the cause. Include the error codes, the status page state, and the timestamp of the first report. Separate a vendor outage from a local configuration error. A fallback that starts for the wrong cause can make the incident larger.
Record the exact words support should avoid, such as blaming the vendor before the cause is confirmed. Give them a neutral phrase that describes the customer impact. A careless message creates a second incident. Put the approved phrase at the top of the support note.
Set the degraded mode the team can actually run
Write a degraded mode that the team can actually run with the people on shift. Examples include queuing payments for later, showing a pickup option, or disabling a nonessential widget. Record the steps, the permissions, and the expected customer message. A fallback that needs an executive approval at midnight is not a fallback.
Test the degraded mode on a staging copy with the vendor blocked. Confirm that the rest of the customer path still works. Record the data the mode cannot accept and how the team will recover it. A degraded mode that loses orders is not an improvement.
Write the exit from the degraded mode as clearly as the entry. Record how queued work is replayed, how duplicates are detected, and who confirms the backlog is clear. Test the exit on the same staging copy. A mode that cannot be reversed safely is not a fallback.
Decide who declares an outage
Name the person who can declare a vendor outage and the backup who acts when they are unavailable. Record the evidence required before the declaration. Set a time limit for a decision when the status page is silent. The team should not wait for perfect proof while customers see errors.
Separate the declaration from the customer communication and from the technical recovery. Give each role a named owner and a checklist. Record the handoff between roles. Clear roles shorten the period when nobody is deciding.
Record what to tell customers
Write a short message that states what is affected, what still works, and when the team will update again. Avoid promising a fix time that the vendor has not confirmed. Include the support channel and the reference number. Keep the message in the same place as the fallback steps.
Record who approves a public status update and who can change the message. Preserve the timeline of what was sent and when. Update the message when the state changes rather than leaving an old promise visible. Customers forgive a clear delay more than a silent one.
Test the note with a tabletop drill
Run a tabletop drill with support, engineering, and the client owner in the same room or call. Walk through the first thirty minutes using a realistic vendor failure. Ask each person to state their first action and the evidence they need. Record the gaps without blaming the people who found them.
Choose one gap to fix before the next quarter. Update the contact card, the degraded mode, or the customer message. Schedule the next drill and keep the record with the vendor list. A fallback note that is never rehearsed is a document, not a capability.
Keep the drill short enough that the team will repeat it. Ninety minutes with a written outcome is more useful than a half-day exercise with no record. Invite the person who writes the customer message. A short drill that happens quarterly beats a perfect plan that never runs.
Keep the note beside the support queue
Store the fallback note where support already works, not in an engineering wiki that requires a separate login. Link it from the queue, the status page draft, and the on-call runbook. Keep the vendor contacts and the current degraded mode in the same page. A note that takes three clicks to find will be skipped during a rush.
Assign a review date and a reviewer for each vendor section. Update the contact when a contract changes or a vendor merges. Remove a vendor that no longer sits on the customer path. The note should be shorter after each review, not longer.
Copy-ready brief
Put the operating agreement in writing.
Name every vendor on the customer path and the action it can block.
Write what the customer sees before describing the internal cause.
Record runnable steps, permissions, and the data the mode cannot accept.
Name who declares the outage and the backup who acts without them.
Frequently asked questions
Can the offshore pod declare a vendor outage?
Only if the agreement names that person and the evidence required for the decision.
Should the team promise a fix time?
No. Give the next update time until the vendor confirms a recovery estimate.
What should the drill cover?
The first thirty minutes, the customer message, the degraded mode, and the handoff between roles.
Sources
- AWS Well-Architected reliability pillar
Dependency failure and recovery planning guidance.
- Google SRE Book
Incident response and operational practice.
- NIST SP 800-53 security and privacy controls
Contingency planning and incident handling controls.
- CISA Secure by Design
Resilience and customer communication principles.
