Docs

Validate fixes before release

A repair can look correct in code review and still fail on the historical shape that caused the incident. Validation needs the failing evidence and a decision rule that does not depend on anyone's confidence, including a model's. That is what replay provides: a deterministic pass or fail against the incident's own data, before the proposal reaches review.

Invitation-only access

Replay and repair validation are available to invited workspaces. Deterministic validation decides whether a candidate passes; an authorised reviewer decides whether it may be delivered.

The scenario

The contract-drift incident is open: the provider renamed candidate.email_address to candidate.email, and your mapping reads the old path. A proposal is generated from the incident evidence:

Code example

Proposal rep_7f3k2m  Operations:    - Copy $.candidate.email to $.candidate.email_address  Explanation: the provider renamed email_address to email in the webhook    payload; restoring the legacy path preserves the existing mapping.

The operation vocabulary is deliberately tiny (copy a path, coerce a type) because every operation in it can be verified against evidence. A fluent explanation is not part of what gets verified.

The dataset comes first

Before the candidate runs, an immutable replay dataset is built from approved incident evidence: representative values materialized from the structural shapes (a string field gets a representative string; no customer value exists to leak), ordered consistently, manifest hashed. Immutability is the point: the fixture and expected result are fixed before evaluation, so the target cannot drift to match whatever the model produced.

The verdict

The candidate runs in a bounded adapter with no network access, and every fixture produces a comparable result:

Code example

Replay run for rep_7f3k2m        15 fixtures  Schema check          pass  Evidence references   pass (all 15 belong to this incident)  Operation policy      pass (1 operation, within bounds)  Replay comparison     pass: 15/15 fixtures produce the expected shape

And when a candidate is wrong, the failure is just as concrete. A bad candidate that coerced the wrong field fails with a stable mismatch path, not a mystery:

Code example

  Replay comparison     FAIL: 15/15 mismatch at $.candidate.salary                        expected string, produced number

A failed or unsafe candidate stays unapprovable, permanently. There is no override, and regenerating against unchanged evidence produces the same verdict.

Review validation

The approval decision comes after the deterministic result, with the checklist the approval workflow formalizes:

  1. Confirm the dataset belongs to the current incident and has not changed.
  2. Review the schema, operation, evidence-reference, and replay checks.
  3. Compare the original failure, candidate output, and expected result.
  4. Approve or reject, with rationale; the approved artifact is fingerprinted so delivery ships exactly what was approved.

Operating boundaries

  • The model proposes and explains; deterministic validation decides pass or fail.
  • The replay runner has no provider or production network access.
  • A passing replay does not merge, deploy, or promise every customer test will pass.
  • Recorded approval is required before export or GitHub delivery.

Related: Replay for fixtures and isolation, and Validate with replay for the dashboard workflow.