<!-- informative -->

> **RETRACTED (2026-09).** Retired as active guidance: the automation targets it named were executed in the remediation batches, and its durable content (the three standing rules) is carried by `report-recurring-failures.md` and the permanent regression tests. This document is retained only as the historical recommendation record; do not treat it as the current test strategy.

# Testing Recommendation 2: Automation Gaps Exposed by the SPEND-Gate Review

## Purpose

This is a follow-on to `docs/research/testing-recommendation.md`, for the team that owns test automation. The first
recommendation's mechanisms were built and did find real defects. An independent adversarial review of the SPEND gate
then found eleven further issues that the delivered suite did not detect. This document explains why the mechanisms
missed them, names the small number of genuinely new automation targets, and specifies the additional work with
acceptance criteria.

It is a work specification, not an implementation. The evidence is `docs/research/spend-gate-adversarial-review.md`
and the commits that fixed its findings (`9e657e5`); the current suite is described in
`docs/report-testing-hardening.md`.

## The problem to solve

The delivered mechanisms were biased in four ways:

- single-threaded and single-process;
- categorical malformed inputs (NaN, infinity, boolean, wrong type) rather than continuous boundaries or sentinel
  defaults;
- invariants over reduced `RunState` rather than over arbitrary ledger operation sequences;
- verifier mutation applied to some verifiers but not all, and to numeric shape rather than semantic meaning.

A guard that is correct but unreachable, or an invariant that never sees the concurrent interleaving, passes every
mechanism above while remaining broken.

## Findings the suite missed, and why

| Finding | Existing class | Mechanism gap | Automation target |
| --- | --- | --- | --- |
| F1 double billable call on malformed completion | 4 | No interleaving or duplicate-delivery harness | A5 concurrency |
| F2 cancellation launders a reported actual | 4/5 | No accounting invariant for cancellation; crash matrix omits the workflow | A3 invariants, A4 crash matrix |
| F3 budget transfer moves spent budget | 2/8 | Invariants only over `RunState`, not ledger operations | A3 ledger fuzzer |
| F4 fabricated inline acceptance evidence | 5/1 | Verifier mutation did not cover the acceptance guard | A2 verifier matrix |
| F5 `run_trial` starts without a reservation | 9/6 | Spend-path enumeration lived in prose, not a test | A7 spend-path meta-test |
| F6 `(0, 0)` disables token enforcement | 1/9 | Boundary table tested malformed values, not defaults that disable a guard | A1 guard-disable mutation |
| F7 token units billed as money | 1/8 | Categorical fields not constrained to a catalogue | A8 categorical completeness |
| F8 ledger loses a reservation across writers | 4 | No multi-writer or multi-process test | A6 multi-writer |
| F9 settle fails within tolerance | 1/5 | Boundary values categorical, not epsilon-distance | A3 property tests |
| F10 freeze hash drifts on sets and bytearrays | 1/2 | No generative type coverage for normalisation | A3 property tests |
| F11 no ledger tamper-evidence | 3 | Not a defect under the accepted threat model | none |

## New sub-classes worth naming

1. **Interleaving and durability across writers.** The property is not "this function is correct" but "this invariant
   holds under two concurrent deliveries or two writers". This is where F1 and F8 live.
2. **Sentinel or default disables a guard.** A parameter defaulting to `0`, `None`, or "optional" silently turns an
   enforcement off. The guard is correct but unreachable. This is where F5 and F6 live.
3. **Semantic validity of a shape-valid value.** The shape is accepted but the meaning is wrong: truthy check objects,
   non-monetary units. This is where F4 and F7 live.

## Rule of thumb

For every enforcement add a bypass test; for every invariant add a concurrent and multi-writer variant; for every
shape check add a semantic check. Applying these three rules to the existing mechanisms would have caught F1 through
F8.

## New mechanisms

Each mechanism names its target, the property it asserts, how to build it, its test tier, and its anti-vacuity guard
(how the mechanism proves it is not passing by doing nothing).

### A1 Guard-disable mutation

- Target: F5, F6, and any enforcement parameter with a default.
- Property: an enforcement cannot be disabled by supplying its sentinel or default value; it either remains enforced
  or refuses explicitly.
- Build: enumerate enforcement parameters (`commitment`, `max_input_tokens`, `max_output_tokens`, `timeout_seconds`,
  `spend_ledger`, `limits`, and similar). For each, construct the valid case, then substitute `0`, `None`, `{}`, `[]`,
  and the empty string, and assert the invariant still holds or the call is refused before any side effect.
- Tier: fast.
- Anti-vacuity: assert the valid case succeeds first, so a rejection is attributable to the mutation.

### A2 Verifier-mutation coverage matrix

- Target: F4, and every verifier.
- Property: each verifier rejects a mutation of the authoritative object it trusts.
- Build: define a registry of verifiers and their authoritative inputs — import verification, manifest binding,
  acceptance reducer guard, oracle classifier, model-output admission, spend ledger authorization, freeze
  normalisation. A meta-test fails if any registered verifier has no mutation test. Each verifier gets at least a
  content mutation and an identity/version mutation.
- Tier: fast.
- Anti-vacuity: the meta-test enumerates verifiers from source or a registry, so a new verifier without a mutation test
  fails the build.

### A3 Ledger operation fuzzer and property tests

- Target: F3, F9, F10, F2.
- Property: any sequence of ledger operations preserves the no-overcommitment invariant and monotonic accounting;
  numeric operations are correct at tolerance boundaries; normalisation is hash-preserving.
- Build: a generator over operation sequences (`reserve`, `commit_reservation`, `settle`, `reclaim_reservation`,
  `transfer`) checking after every step that total charged across any ancestor path never exceeds the authorized
  limit, that `transfer` never moves more than `available`, and that settled attempts never lose an admitted actual.
  Add Hypothesis properties for deltas in `[-TOLERANCE, TOLERANCE]` around commitments and for generative nested types
  (sets, frozensets, bytearrays, bytes, tuples) round-tripped through `_deep_freeze` / `canonical`.
- Tier: fast (ledger fuzzer), fast (Hypothesis).
- Anti-vacuity: require each generated sequence to contain at least one `settle` and one `transfer`; for properties,
  assert a seeded failing example is minimised and stored.

### A4 Crash-matrix workflow coverage

- Target: F2, and the workflows the first matrix did not run.
- Property: crashing before commit at every event index of every representative workflow leaves recoverable, invariant
  consistent, non-laundering state.
- Build: extend the existing matrix to include cancellation, budget-exhaustion cancellation of in-flight attempts,
  root replacement with budget transfer, and observation-then-crash. Assert the accounting invariant "an attempt with
  a durable reported actual is finally settled at that actual or the commitment, never zero".
- Tier: slow.
- Anti-vacuity: assert each workflow produces the expected non-crash control run first.

### A5 Concurrency harness for single-flight

- Target: F1.
- Property: one `(attempt_id, token)` invokes the priced client exactly once, and all callers observe the same
  outcome, regardless of whether the completion is valid.
- Build: `threading.Barrier` to rendezvous two callers on the same attempt; count client invocations; run the case
  with a valid completion and with a completion that fails post-call validation. Assert exactly one call and identical
  results. Guard with a per-test timeout.
- Tier: fast.
- Anti-vacuity: assert the barrier was reached by both parties and that one call happened; a test that never
  interleaved would otherwise pass.

### A6 Multi-writer ledger durability

- Target: F8.
- Property: concurrent writers on one workdir never lose a reservation and never exceed the ceiling.
- Build: a multiprocessing test with a barrier where two workers each reserve against a ceiling smaller than their sum;
  assert exactly one reservation is represented and the ledger file parses. Prefer a spawn context with a module-level
  worker to avoid fork-in-threads hazards.
- Tier: slow.
- Anti-vacuity: assert the file exists and is parseable after the run, and that the accepted reservation equals the
  ceiling remainder.

### A7 Spend-path meta-test

- Target: F5.
- Property: every code path that can spend reserves against both the run ledger and the experiment ledger.
- Build: a curated registry of spend paths with the reason each is safe. A meta-test enumerates the public entry points
  that reach a `ModelClient`, `plan_attempt`, or `reserve`, and fails if a path is not registered. Keep the enumeration
  curated rather than attempting brittle whole-program call-graph analysis.
- Tier: fast.
- Anti-vacuity: the registry must contain at least the known paths; deleting one fails the meta-test.

### A8 Categorical field completeness

- Target: F7.
- Property: every authority-bearing categorical field (`units`, `check_class`, `relation_type`, budget dimensions,
  outcome names) is a member of a catalogue and rejects anything else.
- Build: extend the boundary table's field enumeration to categorical response fields, and assert the catalogue is
  non-empty and used as the sole source of allowed values.
- Tier: fast.
- Anti-vacuity: assert each catalogue has at least one value and that a known member is accepted.

## Tooling

- Hypothesis is already installed: use it for A3 properties and for generative type/normalisation coverage. Pin seeds
  or set `derandomize` so CI stays reproducible.
- Concurrency uses `threading.Barrier` for A5 and `multiprocessing` (spawn context, module-level worker) for A6.
- No new runtime dependency is required; these are development-only additions and stay out of runtime requirements.

## Tiering and integration

- Fast tier (pre-commit): A1, A2, A3 properties, A5, A7, A8. These are deterministic and should run in seconds.
- Slow tier (pre-push and nightly): A4, A6, and the ledger fuzzer under a larger step budget.
- Metrics and applicability guards already exist and are unchanged.

## Cost and benefit

| Mechanism | Cost | Findings it would have caught | Verdict |
| --- | --- | --- | --- |
| A1 guard-disable mutation | ~1 day | F5, F6 | Do first |
| A2 verifier-mutation matrix | ~1 day | F4 | Do first |
| A3 ledger fuzzer plus property tests | ~1.5 days | F2, F3, F9, F10 | Do |
| A5 concurrency harness | ~1 day | F1 | Do |
| A8 categorical completeness | ~0.5 day | F7 | Do |
| A7 spend-path meta-test | ~1 day | F5 | Do |
| A6 multi-writer ledger | ~0.5 day | F8 | Do, slow tier |
| A4 crash-matrix extension | ~1 day | F2 | Do, slow tier |

Total roughly 7 to 8 days, each item with at least one P1/P2 behind it in this review. Exhaustive concurrency fuzzing
and whole-system state-machine property testing remain not worth it: high cost, flaky, and weak invariants give false
confidence.

## Acceptance criteria

- Every mechanism lands with a test that fails on the pre-fix revision and passes on the fixed revision.
- The verifier-mutation and spend-path meta-tests fail if a verifier or spend path is added without coverage.
- A1 proves each enforcement refuses or holds under its sentinel value.
- A3's fuzzer asserts no-overcommitment after every step and rejects a sequence that would violate it.
- A5 asserts exactly one client call and identical outcomes for both callers.
- A6 asserts one accepted reservation and a parseable ledger after concurrent writers.
- The fast tier stays under thirty seconds; the slow tier is time-bounded and non-flaky across repeated runs.
- No test is weakened to green; defects the mechanisms expose are reported as failing tests, as in the previous
  programme.

## Non-recommendations

- Do not build exhaustive randomized concurrency fuzzing or a whole-system state-machine property test.
- Do not add ledger tamper-evidence (F11) unless the threat model changes to a shared or multi-tenant filesystem.
- Do not widen the fast tier with multiprocess tests; keep those slow.
- Do not treat a green mechanism as a substitute for the independent adversarial review; the review found these
  precisely because it was independent and mandated to falsify.

## Open questions for the automation team

1. Which mechanisms are in scope for the next cycle, and which are deferred to the batch that owns the corresponding
   production change?
2. Is Hypothesis with pinned seeds acceptable for the fast tier?
3. Which verifiers and spend paths belong in the registries, and who reviews additions?
4. What is the time budget for the slow tier before it stops being run?
