Review a remote engineer's on-call runbook before handoff

Is the handoff ready?

Use these checks before access expands or the first ticket begins.

Start with the alerts the engineer will receive

A runbook is only as good as the alerts it covers, so begin by listing the alerts the remote engineer will actually receive. Include the alert name, the service, the threshold, and the expected first action. If an alert can fire without a runbook entry, the on-call handoff is incomplete. The list should come from the monitoring system, not from memory.

For each alert, record what a normal reading looks like and what would make it urgent. A threshold without context invites the engineer to guess. Include the time window and the aggregation method so the engineer knows whether a spike is real. The goal is a first decision, not a full diagnosis.

Test the first diagnostic step

Ask the remote engineer to run the first diagnostic step in a staging environment while the author observes. The step might be a dashboard query, a log search, or a health check. Confirm that the engineer has access and that the output matches the runbook. A step that cannot be run is a documentation gap, not a training gap.

Record the time it takes to find the relevant signal. If the engineer spends most of the time navigating tools, the runbook needs a direct link. If the engineer cannot interpret the output, the runbook needs an example. Fix the document before the next practice session.

Separate diagnosis from recovery authority

The remote engineer may be able to diagnose an incident without being allowed to change production. Write the boundary explicitly: which actions are read-only, which require approval, and which are forbidden. Name the person who can approve a restart, a rollback, or a traffic shift. A runbook that implies authority the engineer does not have will lead to a blocked incident.

For each recovery action, include the evidence the engineer must collect before and after. A restart without a before-and-after log is hard to review. A rollback without the deployed version and the rollback target is dangerous. The runbook should make the evidence part of the action, not an optional extra.

Write the escalation path with names

Replace the general instruction to escalate with a named path for each severity. Include the primary contact, the backup, the channel, and the expected response window. State what information the engineer should send in the first message. A clear escalation template saves time when the incident is already stressful.

Check the path against the actual on-call schedule for the next month. A name that is on vacation is not a path. Record the date the schedule was checked and the person who confirmed it. Update the runbook when the schedule changes, not after an escalation fails.

Practice one realistic incident

Run a tabletop exercise using a recent incident with the customer data removed. Ask the engineer to follow the runbook and narrate each decision. Pause when the runbook is ambiguous and record the question. The exercise should end with the engineer proposing one change to the document.

Do not use the exercise to test whether the engineer can solve the incident alone. Test whether the runbook gives them a safe next step. If the engineer needed to ask the author for information that should have been written down, add it. The goal is independence with evidence, not heroics.

Keep the runbook current

Assign an owner and a review date to every runbook section. A runbook that is reviewed once and never again will drift from the system. Add a short change log so the engineer can see what changed and why. Link the runbook to the service repository so a code change can trigger a review.

When an incident reveals a gap, update the runbook before closing the incident review. Record the change and the person who approved it. If the gap was caused by an architecture change, add the architecture dependency to the review. A runbook is a living control, not a one-time deliverable.

Hand off with a shadow shift

Before the remote engineer takes on-call alone, schedule a shadow shift where they follow the runbook while the author is available. The author should observe without taking over. Record the questions the engineer asked and the steps that were slow. Use the notes to improve the runbook and the training.

End the shadow shift with a written confirmation that the engineer can find the alerts, run the diagnostics, and escalate to the right person. Name the date the engineer becomes primary and the backup for the first week. A shadow shift is not a formality, and the confirmation should be based on observed work. Keep the confirmation with the on-call schedule so the next rotation can see it.

Frequently asked questions

Sources

1. CISA Secure by Design

   Authoritative guidance used to frame this field guide.

2. NIST Secure Software Development Framework

   Authoritative guidance used to frame this field guide.

3. OpenTelemetry documentation

   Authoritative guidance used to frame this field guide.