Use cases
Validate fixes before release
Validate a bounded repair against approved evidence in the controlled pilot.
A repair can look correct in a code review and still fail on the historical shape, edge case, or expected result that produced the incident. Validation needs representative evidence and a decision rule that does not depend on a model’s confidence.
This use case builds an immutable replay dataset from approved incident evidence, runs a supported candidate without network access, and compares the result deterministically before a reviewer records any approval.
Controlled pilot
Before you begin
- A supported incident
- Approved redacted evidence
- An expected replay result
- Pilot access
Why repair validation matters
Understand why a plausible mapping change still needs deterministic evidence before delivery.
Plausible transformations can change unrelated fields, rely on live provider behaviour, or fit one example while failing another. A fluent explanation cannot establish that the candidate preserves the expected output.
Repair validation is a distinct use case because its goal is not to generate more code. It creates a reviewable pass or fail result from a bounded input, expected output, and deterministic checks.
When to use this workflow
Choose replay validation for supported incidents with an approved fixture and expected result.
Use replay validation when a supported incident has enough approved evidence to represent the failure and the expected result can be stated before the candidate runs.
- Validate a supported field rename or primitive type conversion.
- Compare a proposed mapping with the incident fixture before repository delivery.
- Reject a candidate that references evidence outside the approved dataset.
- Preserve the validation result with the human review decision.
Pilot boundary
Use repair validation only for an agreed integration and approved evidence set.
The pilot is limited to an agreed integration, evidence set, and supported transformation. The validation runner has resource limits and no network access.
Do not treat arbitrary application code, live provider calls, database access, or deployment behaviour as part of the replay contract. Those checks remain in the customer repository and release process.
Build the replay dataset
Create an immutable fixture with the observed input and expected result.
Create the fixture before evaluating a candidate so the input and expected result do not shift to match generated output. Preserve the incident and observation references that justify every included value.
- Include only evidence approved for replay.
- Store the expected result with the immutable fixture.
- Remove secrets and unnecessary payload fields before the dataset is created.
- Preserve the incident and evidence references used to build the fixture.
Review deterministic validation
Compare the candidate result before recording a human approval decision.
Compare the candidate output with the expected result and review every deterministic check. Do not approve a proposal with a failed check, unsupported operation, or unexplained evidence reference.
- Confirm that the dataset belongs to the current incident and has not changed.
- Review schema, operation, evidence-reference, and replay checks.
- Compare the original failure, candidate output, and expected result.
- Approve or reject only after the deterministic result is available.
Operating boundaries
Keep the runner isolated, the model advisory, and every delivery behind recorded approval.
- The model proposes and explains; deterministic validation decides pass or fail.
- The replay runner has no provider or production network access.
- A passing replay does not merge, deploy, or prove every customer test will pass.
- A human approval is required before export or GitHub delivery.