Validate fixes before release

A repair can look correct in code review and still fail on the historical shape that caused the incident. Validation needs the failing evidence and a decision rule that does not depend on anyone's confidence, including a model's. That is what replay provides: a deterministic pass or fail against the incident's own data, before the proposal reaches review.

The scenario

The contract-drift incident is open: the provider renamed candidate.email_address to candidate.email, and your mapping reads the old path. A proposal is generated from the incident evidence:

Proposal rep_7f3k2m  Operations:    - Copy $.candidate.email to $.candidate.email_address  Explanation: the provider renamed email_address to email in the webhook    payload; restoring the legacy path preserves the existing mapping.

The operation vocabulary is deliberately tiny (copy a path, coerce a type) because every operation in it can be verified against evidence. A fluent explanation is not part of what gets verified.

The dataset comes first

Before the candidate runs, an immutable replay dataset is built from approved incident evidence: representative values materialized from the structural shapes (a string field gets a representative string; no customer value exists to leak), ordered consistently, manifest hashed. Immutability is the point: the fixture and expected result are fixed before evaluation, so the target cannot drift to match whatever the model produced.

The verdict

The candidate runs in a bounded adapter with no network access, and every fixture produces a comparable result:

Replay run for rep_7f3k2m        15 fixtures  Schema check          pass  Evidence references   pass (all 15 belong to this incident)  Operation policy      pass (1 operation, within bounds)  Replay comparison     pass: 15/15 fixtures produce the expected shape

And when a candidate is wrong, the failure is just as concrete. A bad candidate that coerced the wrong field fails with a stable mismatch path, not a mystery:

  Replay comparison     FAIL: 15/15 mismatch at $.candidate.salary                        expected string, produced number

A failed or unsafe candidate stays unapprovable, permanently. There is no override, and regenerating against unchanged evidence produces the same verdict.

Review validation

The approval decision comes after the deterministic result, with the checklist the approval workflow formalizes:

1. Confirm the dataset belongs to the current incident and has not changed.
2. Review the schema, operation, evidence-reference, and replay checks.
3. Compare the original failure, candidate output, and expected result.
4. Approve or reject, with rationale; the approved artifact is fingerprinted so delivery ships exactly what was approved.

Check your own handler

The verdict above is about Seamward's mapping. To see whether your own code handles the changed shape, run the same fixtures through your handler with the replay runner: once on the code deployed now, where you expect a failure, and once on your fix.

Your own handler                 Reported by your runner  Current code   main @ 4c1e9a2   FAIL: 15/15 mismatch at $.accepted  Your fix       fix  @ 0b3f5a9   pass: 15/15

Both runs appear in the Repair tab next to the validation, marked as reported by your runner. They inform the approval decision; they do not make it.

Operating boundaries

• The model proposes and explains; deterministic validation decides pass or fail.
• Seamward's replay sandbox has no provider or production network access. A run in your own environment is denied network access only when you pass --deny-network.
• A run reported by your runner is labelled as reported, never as verified by Seamward, and never approves a proposal.
• A passing replay does not merge, deploy, or promise every customer test will pass.
• Recorded approval is required before export or GitHub delivery.

Related: Replay for fixtures and isolation, and Validate with replay for the dashboard workflow.