<!-- informative -->
# Recurring-Failure Analysis and Test Recommendations

Corpus analysed: 125 commits, 8 review rounds (~85 findings), 53 test modules, `docs/research/*` adversarial
reviews, `tests/counterexamples.yaml`, `docs/report-testing-hardening.md`, `docs/report-milestone8-readiness.md`.

This document is informative. It recommends tests with a high probability of detecting future bugs; it does not
define requirements.

## Recurring error classes (ranked by recurrence)

| # | Class | ~Count | Canonical evidence |
|---|---|---|---|
| **A** | **Inverted/stale predicate + accounting laundering** — a `dispatched`/`late` snapshot taken at the wrong time zeroes real liability or publishes cancelled artefacts | 8 (incl. 3 P1) | The *same* defect shape found 3×: R04 (`report-milestone8-readiness.md`), SPEND F2 (`controller.py:453-457` computes `dispatched` *after* the cancel transition → always False), Exec F3 (same predicate stale before `executor.cancel()`). Survived because cancellation was only ever tested **sequentially** |
| **B** | **Coercion / "looks close enough"** — `True==1`, `1.0==1`, NaN/inf, None-as-absent, scalar-as-list, token units as money | 16 | 7 of 13 hardening defects (`report-testing-hardening.md`); R02; R17 scalar `path` iterated char-by-char; SPEND F7 `"bananas"` billed; SPEND F9 epsilon window `(c, c+TOLERANCE]` rejected |
| **C** | **Sentinel/default disables a guard** — `spend_ledger=None`, `(0,0)` ceiling, missing `limits` → enforcement silently off | 8 | SPEND F5/F6; "the guard is correct but unreachable" (`testing-recommendation-2.md:51`) |
| **D** | **Fail-open / swallowed failure** — cleanup/teardown errors dropped, `except: pass`, deadline not wrapping the blocking call | 16 | R15, Exec F1 (workload survives teardown; `_teardown` reports success), Exec F4 (exceptional path), Hardening #8/#9, Axis2 defect 2 (recovery-path `TypeError`) |
| **E** | **Recovery/restart/cancel reconciliation** — derived consequences, stragglers, accounting inconsistent after crash/cancel/restart | 14 | R14, R11 (`rm -rf` trial dir), Axis2 F4/F5/defect 4/6, MEAS F3, Axis2 F2 (`--resume` races live run) |
| **F** | **Verifier/oracle binds the wrong object** — hash the working copy not the commit, label-vs-label echo, hash omits what executes, fabricated inline evidence | 14 | R07, R08 + R08-rev F1 (collector `python -c` imports candidate `/work/json.py` → broken candidate graded PASS), SPEND F3 (`transfer` checks original limit not capacity — 18.0 charged vs 10.0 authorised), SPEND F4, Hardening #4 |
| **G** | **Identity/aliasing not bound** — per-node vs per-cycle keys, `(arm,seed)` trial identity, `call_id` vs `attempt_id` key aliasing | 10 | R10, R11, R12, Reply F5b (`"attempt_id": null`), MEAS residual cycle-id collision |
| **H** | **Concurrency/interleaving** — single-flight broken, multi-writer `.tmp` loss, admission races | 11 | SPEND F1/F8/N1, Exec F2/F3, R16, Axis2 F2 |
| **I** | **Vacuous truth / claim–behaviour drift** — checks that cannot fail, metrics hiding spend, claims not held on the real path | 14 | Axis2 F1 (deleting one store makes *all eleven* stress checks pass), Axis2 F3 (check impossible to fail), R20, R19, Reply F4 |

### Cross-cutting meta-class (the enabler): tests that blessed the bug

11+ documented instances of the suite encoding implementation assumptions as expected behaviour:

- `tests/test_milestone4_conformance.py:228-247` "endorses clipped accounting" (protected R02).
- Shipped MEAS F1 regression reused one attempt for both cycles → P1 invisible.
- `test_total_deadline_bounds_a_dripping_reply` used a fake `read()` returning 1 byte ignoring the size
  argument — "the shipped regression test is non-faithful" (`model-reply-adversarial-review.md:291-296`).
- Delivered cancellation tests were sequential → both laundering races invisible (SPEND F2, Exec F3).
- Axis2 F8: assertion pinned to scenario shape, not semantics.
- Every review opens with "a green suite is not a verdict"; baselines of 236→573 passing tests were
  repeatedly falsified by 3–11 new findings each round.

## Why the suite kept missing these (root causes)

1. **Single-threaded bias** — no interleaving harness until A5/A6; cancel-vs-dispatch races structurally invisible.
2. **Categorical but not semantic boundary coverage** — mutation tables inject wrong *types*, not
   defaults-that-disable (`None`, `(0,0)`), epsilon windows, or nested exotic types (`set`/`bytearray` in
   `_deep_freeze`).
3. **Normal-path failure injection** — teardown tested only the non-exceptional path; stress checks tested
   tampering but not *deletion* of evidence.
4. **Vacuous checks never self-attacked** — no "this check must be able to fail" proof until invariants got
   tamper-proofs (and only there).
5. **Incomplete verifier-mutation matrix** — 7 verifiers covered; acceptance guard, hidden-suite hash,
   `oracle_version` binding, `path_in_scope` normalisation were not.
6. **Unfaithful fixtures** — fakes that do not honour the real API contract.
7. **Success-only milestone demonstrations** — capability probes standing in for negative/race/crash proofs.

## Recommended tests (ranked by probability of catching future bugs)

The three standing rules from `testing-recommendation-2.md:55-59` that "would have caught F1 through F8" and
should be policy, not per-batch remediation:

> **(1) for every enforcement a bypass test; (2) for every invariant a concurrent and multi-writer variant;
> (3) for every shape check a semantic check.**

### Tier 1 — directly attack the 3×-recurring classes (do first)

1. **Interleaved cancellation harness (kills class A).** Extend `test_single_flight_concurrency.py`'s barrier
   pattern to `cancel()` × `dispatch` × `admit_result`/`admit_artefact` with `threading.Barrier` at the exact
   windows in `controller.py` (`_cancel_attempt` snapshot, `late`-flag read near `controller.py:1241`,
   post-`CANCELLATION_CONFIRMED` accounting). Assert: durable `actual` is never laundered to 0 after dispatch;
   `ARTEFACT_ADMITTED`/`ARTEFACT_INTEGRATED` refuse cancelled attempts (currently only `RESULT_ADMITTED` is
   reducer-guarded — Exec F2 asymmetric guards). *Why high-yield: this exact shape escaped three independent
   adversarial reviews.*
2. **Guard-disable matrix completion (kills class C).** `test_guard_disable_mutation.py` covers some parameters;
   extend the registry to *every* enforcement parameter derived from `RunConfiguration`/`ModelClient`/`Ledger`
   APIs (token ceilings, `limits`, `spend_ledger`, deadlines, `max_output_tokens`, retry caps), each with
   sentinels `None/{}/[]/""/0/(0,0)/NaN/-1`. Add a meta-test: a new enforcement parameter without a sentinel
   row fails (same pattern as `spend_paths.yaml`). Include the deferred SPEND F6 residual — the `(0,0)`
   one-call overshoot currently pinned as acceptable at `test_milestone8_readiness.py:818`.
3. **Coercion + semantic-value completion (kills class B).**
   (a) fix-track the known red `test_bug_*` pins (`True==1`, nested-enum `None`);
   (b) add **event-payload numeric fuzzing** (currently only schema documents and ledger amounts are fuzzed —
   `commitment`, `deadline`, `timeout_seconds` in events are not);
   (c) epsilon/ulp boundaries for every tolerance (`settle` at `commitment+TOLERANCE`, ±1ulp);
   (d) generative nested types through `_deep_freeze`/hash paths (`set`, `bytearray`, custom `__hash__`) — the
   SPEND F10 class;
   (e) authority-bearing categoricals via A8 (partially delivered — close remaining catalogues such as
   completion `units`).
4. **Verifier-mutation matrix completion (kills class F).** Bring `verifier_registry.yaml` to *every* verifier:
   acceptance-evidence guard (SPEND F4), `hidden_suite_manifest_hash` binding of drivers + expected values
   (R08-rev F3), `oracle_version` pin (Hardening #4), `path_in_scope` `..`/absolute/symlink normalisation
   (Exec F7), import-verification committed-tree binding (R07), manifest label ≠ echoed label (R12). The
   registry meta-test already fails on uncovered verifiers — extend `REQUIRED_VERIFIERS`.

### Tier 2 — high-yield against classes E, G, H, I

5. **Non-faithful-fixture audit + fidelity meta-check.** For every fake/monkeypatch used in regression tests
   (especially `_FakeResponse`, scripted clients, executors), assert the fake honours the real API contract
   (e.g. `read(n)` returns ≤n and *blocks* like a socket; `subprocess` caps). Regression tests for past fixes
   must be re-run against the pre-fix revision to prove they fail ("test fails on pre-fix, passes on fixed" is
   already an acceptance criterion in the recommendation docs — enforce it mechanically for every
   `counterexamples.yaml` entry).
6. **Crash-matrix workflow completion (class E).** Add to `test_crash_matrix.py`: the approval-grant → derived
   acceptance-resume chain, the model-review/admission path, the conflict-resolution path, the freshness
   supersession chain (`RESULT_REVALIDATED` → dependent invalidation — currently only table/Markdown pins, no
   behavioural crash test). Slow tier is fine.
7. **Identity-collision tests (class G).** Two validation cycles on one node must not collide on idempotency
   keys, cycle indices, or evidence ids (R10, MEAS residual, SPEND inline-evidence residual); a RETRY with
   changed context must re-bind `prompt_hash` (`controller._prompt_hashes` is keyed by node, not attempt —
   untested); transcript key names must match the recorder's (`call_id` vs `attempt_id` — Reply F5b). Add an
   **effect-catalogue meta-test** (mirroring `counterexamples.yaml`) enumerating every reducer effect kind
   (`freshness_transitioned`, `child_budget_registered`, `dispatch_command`, `reservation_reclaimed`, …) with
   at least one asserting test each.
8. **Multi-writer / >2-party store concurrency (class H).** Concurrent `store.admit` threads on one run
   (single-writer contract untested under contention); recovery racing a live writer at the event-store level
   (only the smoke_run workdir lock is tested); timer-fire racing observation admission. Keep multi-process
   cases in the slow tier per the existing policy.
9. **Anti-vacuity self-attack suite (class I).** Every check that gates a decision gets a "can this fail?"
   mutation proof (the `TAMPERINGS` pattern from `test_invariants.py`, generalised): stress checks must fail
   when their evidence store is *deleted* (Axis2 F1), `check_reachability` must catch a disconnected cycle
   (synthesised fixture — `report-research-synthesis.md` already names this gap), metric denominators must
   include failed trials (extend the golden metrics), `oracle_grades()` must not collapse
   `ORACLE_INFRASTRUCTURE_ERROR` to `FAIL`. Also: an assertion-numbers-vs-actual sweep (Axis2 F7 "64 hostile
   trials" was 56).

### Tier 3 — closing the thinner surfaces

10. **Real-provider reply-shape diversity** (currently one happy JSON shape + hostile size/class checks):
    missing/partial/malformed `usage` fields and their token-accounting consequences; `finish_reason=length`;
    multi-choice `n>1`; non-UTF8 bytes; streaming/SSE; multiple JSON objects and the prose-tolerance limits of
    `parse_provider_completion` left unpinned. Move the hostile-stub behaviours (`bad_json_body`, `deep_body`,
    `overrun_unparseable`, `bad_verdict_huge`, `drip_forever`) out of slow-tier-only so `--fast` exercises them.
11. **Sampling-parameter binding**: frozen manifest → HTTP body for `temperature`, `top_p`,
    `max_output_tokens` (only `seed` is asserted on the wire today); multi-seed independence/aggregation;
    RETRY re-sampling semantics.
12. **Clock-driven timer tests**: `time_machine` is proven available (`test_static_gates.py:118`) but unused.
    Pin deadlines, advance the clock, observe fires — deadline arithmetic, wall-vs-monotonic, multi-timer
    ordering, disarming semantics beyond the ack timer.
13. **`state_fingerprint` completeness**: mutation-test the comparator over relations, timers, freshness,
    model_outputs, credentials, approvals, failure fingerprints (currently 5 fields — a fingerprinter bug
    silently weakens every replay-equivalence assertion).
14. **Executor-profile binding**: `executor_profile` in `RunConfiguration` is decorative — test that the
    dispatched executor matches the declared profile/version; refusal path for REVIEW with no bound model
    executor.
15. **Redaction in failure surfaces**: secrets inside rejection reasons and exception text, imported artefact
    file contents and git bundles, URL-encoded/JSON-escaped echoes in provider replies; overlapping/substring
    secrets.
16. **Sandbox-gate consistency**: R08 isolation / fail-closed / cgroup tests `pytest.skip` without
    `NAMESPACE_V1`, while `test_milestone3_sandbox.py:133` *fails* — pick one policy (fail-closed preferred)
    so trust-boundary coverage cannot silently vanish on an unprovisioned host.
17. **Open-defect registry**: the ~8 red `test_bug_*` pins are enforced only by docstring convention. Add them
    to `counterexamples.yaml` (or a sibling `open_defects.yaml`) with `status: open` and a meta-test forbidding
    silent weakening — the same review-visible floor pattern as `min_covered: 23`.

## Priority summary

Highest probability of detecting future bugs, given what actually recurred:

1. Interleaved cancel/dispatch/admit harness (kills the 3× laundering class).
2. Guard-disable sentinel completion + meta-test (kills the "default disables enforcement" class).
3. Semantic/coercion completion including epsilon and exotic nested types (kills the largest class by count).
4. Verifier-mutation matrix completion (kills wrong-object oracle binding).
5. Fixture-fidelity proof + pre-fix-fails acceptance for every counterexample (stops the meta-class that
   enabled everything else).

Everything in tiers 2–3 is worth doing, but these five map one-to-one onto the defect shapes that survived
multiple named adversarial reviews despite already being documented after their first occurrence.
