Outsourced ProgrammersPowered by StealthAgents

What share of outsourced data backfills are rehearsed before production?

A protocol for counting backfill scripts, dating each rehearsal, and separating an unrehearsed backfill from one that failed its rehearsal.

What share of outsourced data backfills are rehearsed before production?
WORK_LANE / RESEARCH BRIEF / DATAREVIEW QUEUEACCEPTANCEPASSowner review requiredEVIDENCE TRAILTICKET → BRANCH → REVIEW → CLIENT RELEASE
Table of contents
  1. Research context
  2. Practical application
  3. Operating controls
  4. Sources
  5. Frequently asked questions

Research context

This protocol measures the share of data backfills that are rehearsed before a production run in an outsourced delivery lane. The unit is one backfill script attached to one release at the observation date. A rehearsal means the real script ran against a production-like copy and the result was verified. The study describes change evidence, not the quality of the data model.

Define the releases and repositories in scope before collection. Include schema migrations that change values, one-off repair scripts, and jobs that rewrite rows in bulk. Record the ticket and the environment for each. Publish the list with a version so a later reviewer can reproduce the scope.

Define a backfill by the row keys, the tables, the end value, the application version, and the expected count. Record the exclusions and the time boundary for inclusion. Note the rollback method and the point of no return. Test the definition on the first three scripts.

Set the population as every backfill attached to a release in the agreed window. Include scripts that were run manually and scripts that are part of a deploy. Record the scripts that could not be inspected and the reason. Report them separately from scripts that failed a rehearsal.

Classify each backfill by rehearsal evidence. Use categories such as rehearsed with recorded counts, rehearsed without customer-path checks, tested only on a sample, and not rehearsed. Do not merge the categories after seeing the result. The categories point to different fixes.

Record whether the input was frozen with a snapshot and a timestamp. Note the schema version, the row count, and the batch size used in the rehearsal. Record the named owner and the person who verified the result. A rehearsal on a moving input cannot support a count comparison.

Report the number of backfills, the number rehearsed with evidence, and the number in each category. Show counts by release type and by data sensitivity. Present the unrehearsed scripts and the failed rehearsals as separate rows. Use the same denominator for every rate.

Separate an unrehearsed backfill from a failed rehearsal. An unrehearsed script has no evidence, while a failed rehearsal has evidence and a defect. Report the two categories separately because the fixes differ. Record the failed step and the date beside each finding.

Inspect the scripts written around a former contractor or a compressed release. A cluster of unrehearsed backfills may mean the checklist never covered data repair. Compare the pattern with the repository history and the release calendar. A single omission can be an oversight, while a cluster is a process gap.

Practical application

Do not treat a successful syntax check as a rehearsal. A script can parse and still lock a table or rewrite the wrong rows. Check that the run produced counts and opened the screens that read the field. Report the difference between a parse and a rehearsal.

The record cannot show whether a real run would meet the maintenance window or whether a customer saw the change. It can show only whether the script ran on a comparable copy and the result was verified. An unrehearsed backfill is not proof of an incident, and a rehearsed one is not proof of a safe change. The protocol also cannot see a repair run directly in a console.

A useful additional view is the time from the first batch to the last batch at production row counts. Record the batch size, the pause, and the peak lock wait. That field turns a yes-or-no count into a review of whether the window is realistic. Report the slowest table beside the total.

Record whether the rehearsal measured customer latency while the script ran. A backfill can finish inside the window and still block writes for the whole time. Note the read and write latency at the peak. Report the scripts that were never observed under application load.

Check the rollback evidence for each script. A rollback that restores the snapshot is different from one that rewrites the old values. Record the tested method and the time it took. Report a script whose rollback has never been run.

Give the result to the client technical owner and the offshore delivery lead with one observable change. The change might be a frozen snapshot, a production-size rehearsal, a pause threshold, or a rollback test. Record the owner, the effective date, and the measure that will show whether the change worked. Repeat the protocol with the same definitions after the change.

Close the review with one change and a date to re-measure. The change should address the largest unrehearsed or failed group, not the easiest script to document. Record the expected effect and the evidence that will prove it. A backfill plan that is never rehearsed is a document, not a change control.

Build the sample from migrations and repository history, not from the tickets the team remembers. Search for scripts that update or insert in bulk, including one-off files outside the migration folder. Record how each script entered the sample. The forgotten script is often the one with no ticket.

Use two evidence sources for each backfill, such as the rehearsal record and the database query count. The record proves the plan, while the query proves the rows changed as expected. Compare them on the same frozen input. A record without a count is only an intention.

Operating controls

Scripts from recent incidents are overrepresented because they were discussed. Include a script that ran quietly last quarter and one that is still pending. Record the last run date. A quiet script can still hold elevated access or an outdated query.

Record the database role used for the run and whether it can change schema as well as data. A backfill that needs schema permission is a broader change than the ticket suggests. Note the approval for that role. Report the scripts that used a broader role than the task required.

Report the result by data sensitivity as well as by count. A backfill that rewrites payment records deserves a stricter rehearsal than one that updates a display label. Record the classification beside the finding. The reader should see which gap carries the most risk.

Repeat the measure after the first rehearsed run and keep the definitions unchanged. A move from no rehearsal to one is progress, but it is not yet a habit. Compare the next window with the same categories. The trend matters more than a single release.

Check the evidence that the rehearsal used the same application version as production. A script verified against an older screen can miss a new read path. Record the version and the commit. Report the backfills whose verification used a different build.

Treat the protocol as a review of change planning, not a judgment of the engineer who wrote the script. An unrehearsed backfill often means the release calendar had no room for a copy. Ask what would have made the rehearsal possible. The answer points to the constraint to remove.

Keep the rehearsal record with the release and audit evidence so a reviewer can connect the script to a real row count. Record the snapshot, the query, and the verification screenshot or output. A rehearsal that cannot be tied to a frozen input is only a demonstration. The record should survive a change of engineer.

Set a rule that no backfill touching customer data runs without a rehearsal record and a named pause owner. Write the rule into the delivery agreement and the release checklist. Record the exception process and the approver. The rule turns a good intention into a gate that can be checked.

Sources

  1. PostgreSQL COPY
  2. Flyway database migrations
  3. Google SRE Workbook
  4. NIST SP 800-34 contingency planning

Frequently asked questions

What should happen first?

Begin with a bounded ticket, approved access, explicit acceptance evidence, and a named reviewer.

Who approves production changes?

The client's technical owner keeps final merge and release authority.

Related research

How often do outsourced maintenance lanes test a database restore?

Build a programming lane your technical owner can review.

Bring the stack, backlog, acceptance checks, and release boundaries. We will help scope the role around the work.

Discuss your programming needs