CHINZY / FDE and robotics acceptance fieldbook / 05
FDE pilot release and handover
Freeze samples, versions and ownership before the rehearsal. Keep before/after evidence for an independent reviewer. Budget caps and thresholds in examples are fixtures; agree actual gates with the project owner.
Acceptance checks
Holdout set isolation
Lock the holdout IDs and hashes before tuning. Group duplicates by source and check that versions of one ticket do not cross the split.
02Paired manual baseline
Run the same sanitized tasks manually and with assistance. Pair by task ID and include review time, not just generation time.
03Canary stop conditions
Agree a stop condition for isolated traffic and inject a matching test failure. Confirm new traffic returns to the old version and prior writes remain visible.
04Restore and replay reconciliation
Restore only into an isolated copy. Reconcile incremental events after the snapshot by business ID, recording gaps, duplicates and the last proven point.
05Log redaction regression
Send a searchable fictitious marker in successful and failing requests. Inspect logs, traces and exports without using real customer data as the probe.
06Per-task budget exhaustion
Use a mock meter and a small task cap. At the cap check queued steps and the final explanation; no paid calls are needed for this test.
07Configuration drift reproduction
Run the same input against two test configurations with one variable changed. Save config versions and output differences before blaming the model.
08Independent operator handover
Ask an operator outside the build team to recover a test service from the runbook. Record blocked steps, credential ownership and the verification result, then rerun.