<!-- informative -->

> **RETRACTED (2026-09).** Retired as active guidance: its mechanisms were built and did find real defects, and its durable content (the recurring-defect taxonomy and the standing detection rules) now lives in
> `report-recurring-failures.md` and in the permanent regression tests added with each remediation batch. This document is retained only as the historical recommendation record; do not treat it as the current test strategy.

# Testing Recommendation: Recurring Defect Classes and Detection Mechanisms

## Purpose

This document is a handoff to the team that will own and adjudicate the test strategy. It records, from an
independent readiness review of the orchestrator (`docs/report-milestone8-readiness.md`) and two subsequent
remediation passes, a taxonomy of the defect classes that keep recurring, a ranked set of detection mechanisms that
attack each class rather than a single instance, an assessment of off-the-shelf FOSS tooling, and a cost-benefit
assessment of adopting a schema library. It contains no implementation work; it is a recommendation for the
adjudicating team to accept, amend, or reject.

## Scope

The subject system is the single-machine Autonomous Project Orchestrator: an event-sourced controller with a
deterministic reducer over generated transition tables, a sandboxed command executor, a model adapter with real cost
accounting, an acceptance/evidence layer, a frozen hidden oracle, and an experiment harness. The readiness review
examined revision `c26a0ec` and produced findings grouped in that document as R01 through R23. Batches 1 and 2 of the
remediation are committed at `97c1891` and `9e94357`; Batches 3 through 6 remain.

## Defect classes that recur

The classes below are the observed clusters from roughly two dozen findings. Counts are approximate; the point is
recurrence, not precision.

| Class | Description | Representative findings | Severity | Frequency |
| --- | --- | --- | --- | --- |
| 1 | Boundary validation incompleteness: untrusted or external values coerced instead of rejected (bool, NaN, infinity, negative, scalar-versus-list, unknown reference) | R01, R02, R05, R07, R17, R22 | High | Very high |
| 2 | Verify the wrong object: a verifier checks a working copy, a label, or re-derived state instead of the authoritative frozen object | R01, R07, R12, R13 | High | High |
| 3 | Fail-open resource handling: failures swallowed, cleanup reported as successful without verification, missing coverage continues | R01, R04, R05, R15 | High when hit | Medium |
| 4 | Non-atomic multi-event workflows: a crash between two events leaves inconsistent state or loses derived effects | R03, R04, R11, R14, R22 | High | High |
| 5 | Silent clipping and vacuous truth: reported values capped, empty sets treated as passing, absent checks treated as success | R02, R08, R19, R23 | High | High |
| 6 | Missing applicability guard: an event whose precondition has passed is still applied | R04, R10, R22 | Medium | Medium |
| 7 | Metric semantics: numerator or denominator is wrong, or a cohort is not comparable | R20 | Low to medium | Low |
| 8 | Identity and version not bound: an executed binding is not persisted, or a mismatch is not rejected | R02, R07, R11, R12 | Medium | Medium |
| 9 | Reconstruct-from-current instead of freeze-at-dispatch: a decision context is re-derived after execution rather than frozen before it | R03, R06, R12, R17, R20 | Medium to high | High |
| 10 | Tests bless the bug: a test encodes the implementation's assumption as correct behavior, so the bug is protected | meta | High leverage | High |

Classes 1, 2, 4, 5, and 9 account for most findings. Class 10 is the reason the others survived: each individual fix
is cheap, but without a mechanism that attacks the class, the next instance appears in the next milestone.

## Detection mechanisms, ranked

Ranked by expected impact per unit of effort. Each mechanism targets a class, not a bug.

| Rank | Mechanism | Target classes | Cost | Impact | Notes |
| --- | --- | --- | --- | --- | --- |
| 1 | Counterexample registry | 10 | Hours | High | A manifest mapping every review finding id to the permanent test that pins it, plus a meta-test that fails if a mapping or its test disappears. Cheapest guard against silent regression deletion. Build first. |
| 2 | Boundary mutation table | 1, 5 | 1 to 2 days | Very high | Table-driven: for every field of every request, response, event, and ledger boundary, inject `true`, `NaN`, `inf`, `-1`, `null`, scalar-where-list, list-where-scalar, unknown reference, and empty set; assert rejection and zero side effects. Generate the field list from the schema and event specs so it extends itself as the surface grows. |
| 3 | Invariant library plus seeded state-machine fuzzer | 2, 4, 5, 6 | 2 to 3 days | Very high | Small predicate library over the reduced state and ledger, then a seeded fuzzer driving legal transitions and asserting invariants after each step. Each predicate must be mutation-proven before use: tamper the state directly and assert the predicate fires. Weak invariants give false confidence. |
| 4 | Crash-injection matrix | 4, 9 | 1 to 2 days | Very high | Generalize the existing pre-commit crash hook: for a representative workflow, raise before commit at event index i for all i, reopen the store, recover, and assert the invariant library plus "no redispatch". Cost is quadratic in workflow length, so bound the workflows. |
| 5 | Verifier mutation tests | 2 | 1 day | Moderate to high | For each verifier (import verification, manifest binding, acceptance evidence, oracle grade), tamper the underlying object and assert rejection. Precedent already exists for the replay-equivalence comparator. |
| 6 | Applicability-guard table | 6 | 1 day | Moderate | Drive "an event not declared from this state is retained or rejected" from the generated transition tables, so the guard set cannot silently shrink. |
| 7 | Golden metric definitions | 7 | Hours | Low to moderate | Hand-computed numerator and denominator cases, for example two trials costing ten each with one acceptance must report twenty, and speed-up is computed only over a comparable successful cohort. |
| 8 | Fail-closed harness | 3 | 1 day | Moderate | Monkeypatch the resource operations that actually leaked (cgroup writes, directory removal, file writes, status pipes) to raise, then assert the caller reports failure rather than success. Keep it narrow; broad monkeypatching becomes brittle. |

Not recommended: exhaustive randomized concurrency fuzzing or full event-sequence property fuzzing. The cost is high,
the signal is low without a scheduler model, and weak invariants create flakiness and false confidence. The crash
matrix plus a few targeted two-thread barrier tests cover the realistic cases. A standalone identity-binding suite is
also not recommended: that class is fully subsumed by mechanisms 2, 4, and 5.

## FOSS tooling

The project currently keeps a deliberately small dependency set; `requirements-dev.txt` contains only a Markdown
linter and PyYAML. Treat the following as development-only additions and leave runtime requirements unchanged.

| Rank | Tool | License | Serves mechanism | Notes |
| --- | --- | --- | --- | --- |
| 1 | Hypothesis | MPL-2.0 | 2, 3 | Strategies for malformed numeric and shape inputs, and `RuleBasedStateMachine` with `invariant` decorators for the reducer. Highest-value single addition. Configure deterministic seeds so CI stays reproducible. |
| 2 | mutmut or cosmic-ray | BSD-3, MIT | 5, 10 | Mutation testing: surviving mutants are tests that do not actually assert. mutmut is simpler; cosmic-ray is more configurable and can distribute. Run on changed modules with a "no new surviving mutants" gate; full-tree runs are slow. |
| 3 | ruff | MIT | 3, 5 | Enable bugbear (`B`), blind-except (`BLE001`), try-except-pass (`S110`), and simplify (`SIM`). Cheap static gate; the codebase already contains the swallowed-exception pattern that leaked. |
| 4 | Semgrep OSS | LGPL-2.1 | 1, 3, 5 | Local rules for the repo-specific anti-patterns: `except ...: pass`, `min(actual, commitment)`, `float(payload[...])` without finiteness, and iterating `body.get(field) or []`. Catches the pattern, not the instance. Fine to use as a dev tool without copyleft impact on project code. |
| 5 | time-machine | MIT | 6 | Deterministic clock. The reducer must not read the clock and timers use absolute deadlines; a controllable clock makes those tests exact instead of sleep-based. |
| 6 | pytest-randomly plus pytest-timeout | MIT | 4 | Order randomization surfaces hidden shared-state coupling; timeout prevents crash and race tests from hanging CI. |
| 7 | pytest-cov (coverage.py) | Apache-2.0 | corpus gate | Branch coverage as a floor over the frozen conformance corpus and a guard against silently deleting coverage. It is a floor, not a correctness proof. |
| 8 | pytest-run-parallel | Apache-2.0 | 4 | Raises interleaving on targeted concurrency probes only, not the whole suite. |
| 9 | deal or icontract | MIT | 3, 5 | Runtime contracts with Hypothesis integration. Optional; overlaps the hand-written invariant library. |

Not applicable or not worth it: schemathesis and hypothesis-jsonschema (there is no HTTP API and no JSON Schema
source); atheris and coverage-guided fuzzing of the event space (heavy setup, worse signal than the state machine);
toxiproxy and chaos-toolkit (network and infrastructure layer, wrong abstraction); pytest-fault (unmaintained); race
detectors such as Helgrind and ThreadSanitizer (they target C, C++, and Rust; there is no reliable Python race
detector, so rely on locks plus targeted barrier tests).

## Schema-library assessment: pydantic v2

The question was raised whether replacing the hand-rolled validation in `schemas.py` and the request/response
validation in `executor_protocol.py` with pydantic v2 would give a more solid foundation. Assessment follows.

### What pydantic v2 would provide

- Strict, finite numeric types (`FiniteFloat`, strict integers, non-empty strings) that reject boolean, `NaN`,
  infinity, and negative values by construction rather than by per-field hand-written validators. This is class 1,
  the most frequent class, addressed structurally.
- Unknown-field rejection and required-field enforcement made declarative and uniform, removing the "a validator was
  forgotten for this field or this shape" failure mode.
- Discriminated unions for artefact, observation, cost, and event types, with unknown variants failing closed.
- JSON Schema export, which can drive generated documentation and, via hypothesis-jsonschema, the boundary mutation
  table (mechanism 2).
- Structured machine-readable validation errors, useful for redaction and audit diagnostics.

### What it would not provide

- Cross-field semantics remain custom: evidence ownership, result existence, hash matching the imported content,
  commitment dimension against budget, and catalogue membership of limit keys.
- It does not help verifier correctness (class 2), fail-closed behaviour (class 3), atomicity and crash consistency
  (class 4), applicability guards (class 6), or metric semantics (class 7). Those are the more severe findings.
- `frozen=True` freezes model attributes, not nested plain dictionaries and lists, so the existing deep-freeze helper
  is still required for manifests.
- Pydantic defaults are weaker than the current code: `extra` defaults to ignore and lax mode coerces strings to
  numbers, integers to floats, and tuples to lists. Without explicit `extra="forbid"` and strict types everywhere,
  a migration would reintroduce the coercion class it is meant to remove.

### Migration options and blast radius

| Option | Scope | Call-site churn | Effort | Risk |
| --- | --- | --- | --- | --- |
| Facade migration (recommended) | Keep the public functions `validate`, `validate_request`, `validate_response`, `build_request`, and the `SchemaValidationError` and `ProtocolViolation` exception types; implement the internals with pydantic models and wrap `ValidationError` | Near zero | 3 to 5 days | Low |
| Full replacement | Models become the API and callers pass models | High | 2 to 4 weeks | Medium to high |
| Event-layer extension | Discriminated union over EventSpec in `events.py` | Zero if admission keeps taking dictionaries | 4 to 6 days | Medium to high, hot path and conformance corpus |

The facade option is viable because only a handful of call sites exist, no test asserts validation message text
(only exception types), and models can remain transient validation wrappers so the persisted event and database
format does not change. One existing test iterates schema and field descriptors; retaining a descriptor view of the
models keeps it working.

### Costs

- The project's first heavy runtime dependency: pydantic plus pydantic-core (compiled) plus annotated-types and
  typing-extensions. All permissively licensed and compatible with the project license, but it is a real departure
  from the current minimal-dependency posture.
- Strictness discipline: the coercion matrix, especially integer-versus-float and boolean handling, differs between
  strict and lax mode and has changed across pydantic 2.x releases. It must be pinned and covered by a dedicated
  matrix test.
- A fourth potential source of truth for enums and catalogues. Normative values must stay in `glossary.py` and the
  generated tables; models must reference catalogues through validators, never duplicate them as literals.
- Hot-path overhead on every event admission and request/response validation. Benchmark `model_validate` against the
  current linear loop before extending to the event layer.

### Verdict

Conditionally favourable, scoped to the shape and number layer, behind the facade. Adopt if the provider and
configuration surface is expected to grow (a real adapter or external configuration) or if JSON Schema becomes a
documentation source of truth. Otherwise, a small dependency-free strict-types module plus mechanism 2 captures most
of the numeric and shape benefit without the supply-chain, wheel, and strictness-discipline costs. Do not migrate
`events.py` in the same change.

## Proposed implementation order

1. Counterexample registry (mechanism 1), seeded with every reproducer from the readiness review and from the current
   gate review, before anything else changes.
2. Mutation-proven invariant library, then the seeded fuzzer and the crash-injection matrix (mechanisms 3 and 4).
3. Boundary mutation table (mechanism 2), generated from the schema and event specs.
4. Verifier mutation tests and frozen-binding checks (mechanism 5).
5. Applicability-guard table and golden metric definitions (mechanisms 6 and 7).
6. Fail-closed harness (mechanism 8), narrowly scoped.
7. Schema-library decision (facade migration or strict-types module), gated on the benchmark and the coercion-matrix
   test.

## Acceptance criteria for the test program

- Every finding in `docs/report-milestone8-readiness.md` is mapped to a permanent test in the registry, and removing a
  mapping fails a meta-test.
- The boundary mutation table covers every field of every boundary and asserts zero side effects on rejection, not
  only an exception.
- The invariant library is mutation-proven: each predicate is shown to fire when the state is tampered.
- The crash-injection matrix covers every event boundary of the representative workflows and confirms recovery leaves
  no open commitment without an attempt, no blocked parent without children, and no reset of consumed capacity.
- Mutation testing reports no new surviving mutants in changed modules.
- The corpus coverage floor does not decrease.

## Non-recommendations

- Do not build exhaustive randomized concurrency fuzzing or full event-sequence property fuzzing.
- Do not add a separate identity-binding suite; fold it into mechanisms 2, 4, and 5.
- Do not adopt a schema library wholesale, and do not migrate the event layer, without the benchmark and the
  coercion-matrix test.
- Do not let the new mechanisms become a coverage target; an invariant that is never mutation-proven is worse than no
  invariant because it creates false confidence.

## Open questions for the adjudicating team

1. Which mechanisms are in scope for the next cycle, and which are deferred?
2. Is Hypothesis, and a deterministic seed configuration, acceptable given the project's determinism requirements?
3. Is mutation testing affordable at the desired cadence, and on which module set?
4. Should JSON Schema become a documentation source of truth, which is the main condition that would justify adopting
   pydantic?
5. Who owns the counterexample registry, and what is the rule for deleting a mapping?
