Table of contents
- Quick read
- Readiness check
- Pick the restore target before the drill
- Copy the backup away from the source
- Restore into an isolated environment
- Check the data and the application path
- Time the recovery and name the decision
- Keep the drill record and fix the slow step
- Write the recovery note the on-call will use
- Role brief
- Frequently asked questions
- Sources
Is the handoff ready?
Use these checks before access expands or the first ticket begins.
Restore target
Ready: Name the database, point in time, environment, and application version.
Needs work: The lane has no written record for this check.
Backup copy
Ready: Use a complete copy in a separate account and record its checksum.
Needs work: The lane has no written record for this check.
Verification
Ready: Open the real customer path and record read, write, and query evidence.
Needs work: The lane has no written record for this check.
Timing
Ready: Separate copy, load, and verification time against the recovery target.
Needs work: The lane has no written record for this check.
Pick the restore target before the drill
A restore drill begins with a written target rather than a vague wish to test backups. Decide which database, which point in time, and which environment the team will rebuild. Name the application version that must run against the restored data. Without a target, the team can spend a day restoring files and still not know whether the service works.
Choose a backup that is old enough to prove retention works but recent enough to exercise the current schema. Record the backup identifier, its timestamp, and the storage location before the drill starts. Confirm that the backup was produced by the same system the team plans to use. A backup chosen after the drill begins cannot prove that the normal schedule is reliable.
Record what the client would lose if the restore stopped at the chosen point in time. A target that ignores the last hour of orders may pass the drill and still fail the business. Write the acceptable data loss in the agreement. The drill should test the target the client actually accepted.
Copy the backup away from the source
Restore from a copy in a separate account or bucket so a mistake cannot touch the production backup. Confirm that the copy is complete by comparing its checksum with the recorded value. Record who has permission to read the copy and who can delete it. A restore that depends on production credentials is not an independent test.
Move the copy through the same path the team would use during a real recovery. If the recovery plan assumes a staging account, use that account rather than an engineer laptop. Note every manual step and the time it took. The path matters as much as the restored data.
Restore into an isolated environment
Build the target database in a network that cannot reach production customers. Apply the same engine version, extensions, and connection limits that production uses. Load the backup with the normal restore command rather than a hand-edited script. Record the start time, the finish time, and any error that required a retry.
Keep the restored database private until the team confirms it contains no unexpected live data. Check the row counts for the largest tables and compare them with the backup record. Run one read query that the application actually uses. A restore that cannot answer a real query is only a file copy.
Check the data and the application path
Point a test copy of the application at the restored database and open the customer path. Check login, a read screen, a write action, and the reporting query that matters most. Confirm that timestamps, identifiers, and file references still resolve. Record the version of the application and the configuration used for the test.
Inspect the data for signs of truncation, duplicate keys, or missing sequences. Compare a sample of records with the source system at the backup time. Note any table that needed a manual repair and why. A clean restore should not require undocumented fixes.
Time the recovery and name the decision
Measure the time from the decision to restore until the application serves a verified request. Separate the copy time, the load time, and the application check so the slow step is visible. Compare the total with the recovery target in the client agreement. Record who would make the go-live decision in a real event.
Write the point at which the team would stop and escalate instead of continuing. Include the conditions that would make the restore unsafe, such as a corrupted backup or a missing encryption key. Name the person who can accept that risk. A drill without a stop rule can turn into an unplanned production change.
Keep the drill record and fix the slow step
Store the drill record with the backup identifier, the commands, the timings, and the verification evidence. Give the record to the client technical owner and keep a copy in the delivery workspace. List every step that required tribal knowledge. The next engineer should be able to repeat the drill from the record alone.
Choose one improvement for the next drill, such as automating the copy, shortening the load, or adding a missing check. Assign an owner and a date. Schedule the next drill before the current record is forgotten. A restore process improves only when someone tests the change.
Share the record with the next on-call engineer before the next drill, not after it. Ask them to follow it without help and note where they hesitate. Fix the first point of confusion rather than adding more prose. A record that only its author can follow has not transferred the capability.
Write the recovery note the on-call will use
Write a recovery note that the on-call engineer can follow without the person who built the backup system. Start with the decision point, the backup identifier, and the account that holds the copy. List the commands in order and the expected output after each one. A note that assumes the reader knows the storage layout is not usable at night.
Include the contact for the client owner, the database vendor, and the network team. Record the conditions that require an escalation rather than a longer attempt. Keep the note beside the restore record and review it after every drill. The recovery note and the drill record should tell the same story.
Copy-ready brief
Put the operating agreement in writing.
Name the database, point in time, environment, and application version.
Use a complete copy in a separate account and record its checksum.
Open the real customer path and record read, write, and query evidence.
Separate copy, load, and verification time against the recovery target.
Frequently asked questions
How often should the lane run a restore drill?
At least quarterly for the primary database, and sooner after a platform or backup change.
Can the team restore into the production account?
No. Use an isolated account so the test cannot touch live data or the production backup chain.
What proves the restore worked?
A recorded backup identifier, a successful load, and a verified application request against the restored copy.
Sources
- PostgreSQL backup and restore
Backup methods, restore procedures, and point-in-time recovery.
- NIST SP 800-34 contingency planning
Contingency planning and testing expectations for information systems.
- AWS Well-Architected reliability pillar
Backup, recovery, and failure testing guidance.
- Google SRE Workbook
Reliability practice and disaster testing context.
