<!-- informative -->

# Testing Hardening Report

## Summary

This report covers a test-only hardening programme derived from the testing
recommendation in `docs/research/testing-recommendation.md`. It executed the
entire proposed implementation order (mechanisms 1 through 9) and delivered it
as nine phases.

The programme was deliberately constrained: **no production code was changed.**
Every check that found a defect was left failing so the failure itself is the
report. The work therefore ends with a suite that is red on purpose, and that
red is the deliverable.

Outcome at a glance:

- 433 tests total (423 fast, 10 slow). The fast subset runs in about 16 seconds.
- 22 failing tests, each one an independently reproducible defect report.
- 13 distinct defects across boundary validation, verifier binding, metrics,
  fail-closed resource handling, static patterns, and numeric coercion.
- Existing regression suite remained green throughout; no previously passing
  test was weakened, skipped, or deleted.

## Objective and ground rules

The recommendation argued that the project's failures were not individual bugs
but whole classes, and that the way to stop the next instance is a mechanism
that attacks the class. The agreed ground rules for this work were:

1. Write tests that encode the intended behaviour; never weaken a test to green.
2. Do not fix the product code. Report the defect through a failing test.
3. Keep the delivered suite runnable on a developer machine without a forge.
4. Make the checks self-extending, so a new field, event, machine, or metric is
   covered without editing the test.

## What was delivered, by mechanism

| Mechanism | Purpose | Delivered as | Status |
| --- | --- | --- | --- |
| 1. Counterexample registry | No finding can be silently unmapped or deleted | `tests/counterexamples.yaml`, `tests/test_counterexample_registry.py` | Complete: all 23 findings mapped, 16 covered, 7 explicitly pending with a remediation batch |
| 2. Boundary mutation table | Malformed values rejected with no side effects | `tests/test_boundary_mutations.py` | Complete; reports 3 defect classes |
| 3. Invariant library plus fuzzer | Inconsistent state fails immediately | `tests/invariants.py`, `tests/test_invariants.py`, `tests/test_state_fuzzer.py` | Complete; every invariant is mutation-proven |
| 4. Crash-injection matrix | Recovery cannot lose or duplicate work | `tests/test_crash_matrix.py` | Complete; 35 crash points pass |
| 5. Verifier mutation tests | A verifier cannot certify tampered input | `tests/test_verifier_mutations.py` | Complete; reports 2 defects |
| 6. Applicability-guard table | An undeclared transition is rejected; guards cannot shrink | `tests/test_applicability_guards.py` | Complete and passing |
| 7. Golden metric definitions | Metrics keep their meaning and denominators | `tests/test_golden_metrics.py` | Complete; reports 2 defects |
| 8. Fail-closed harness | Resource failures are reported, never swallowed | `tests/test_fail_closed_harness.py` | Complete; reports 1 defect |
| 9. Static and schema gates | Swallowed exceptions, unsafe coercions, coverage floor, migration gate | `tests/test_static_gates.py`, `tests/test_schema_library_gate.py`, `ruff.toml` | Complete; reports 5 findings |

Supporting infrastructure was added so the suite is enforced rather than
optional:

- `scripts/setup-dev`, `scripts/check-tests`, `scripts/check-static`,
  `scripts/check`, and tracked hooks under `scripts/hooks/`.
- A fast/full split: slow-marked end-to-end experiment tests are skipped by
  `scripts/check-tests --fast`; commit hooks run the fast subset, push runs the
  full suite.
- A forge-portable CI wrapper at `.github/workflows/tests.yml` that calls the
  same scripts, so a future forge migration changes the wrapper only.

## Defects reported

Each row is reproducible with the named test. "Class" refers to the failure
taxonomy in the recommendation.

| # | Class | Defect | Where | Test |
| --- | --- | --- | --- | --- |
| 1 | 1 | `schema_version=True` is accepted because `True == 1` | `schemas.HarnessSchema.validate` | `test_bug_schema_version_accepts_boolean` |
| 2 | 1 | Boolean `schema_version`/`policy_version` accepted | `events.validate_event` | `test_bug_event_version_accepts_boolean` |
| 3 | 1 | Explicit `None` nested enum treated as absent and skipped | `events.validate_event` | `test_bug_nested_enum_accepts_none` |
| 4 | 2, 8 | Oracle grade trusted even when `oracle_version` differs from the manifest | `oracle.classify_trial` | `test_bug_classifier_ignores_oracle_version_binding` |
| 5 | 2 | Incomplete grade raises `KeyError` instead of a verification error | `oracle.OracleGrade.effective` | `test_bug_classifier_does_not_reject_an_incomplete_grade` |
| 6 | 7 | Cost per accepted root reports `0.0` when spend occurred and nothing was accepted | `experiment._summarise` | `test_bug_cost_per_accepted_root_hides_spend_when_nothing_is_accepted` |
| 7 | 7 | Mean realised speed-up averages failed trials' fallback `1.0` | `experiment._summarise` | `test_bug_mean_realised_speedup_includes_failed_trials` |
| 8 | 3 | A failed cgroup limit write leaks the created cgroup directory | `sandbox.SandboxRunner.run` | `test_bug_failed_cgroup_limits_leave_no_orphaned_cgroup` |
| 9 | 3 | Blind/swallowed exceptions (`except Exception: pass`) | `conformance.py`, `experiment.py` | `test_no_blind_or_swallowed_exceptions` |
| 10 | 1 | Unguarded `float(payload[...])` without a finiteness check | `reducer.py` (multiple sites) | `test_event_payload_numbers_are_finiteness_checked_before_float` |
| 11 | 1 | Iterating a mapping value defaulted with `or []` accepts a scalar | `decomposition.py`, `reducer.py` | `test_no_iteration_over_a_mapping_value_defaulting_to_an_empty_list` |
| 12 | 1 | `schema_version=1.0` accepted because `1.0 == 1` | `schemas.HarnessSchema.validate` | `test_schema_coercion_matrix_is_unchanged` |
| 13 | 1 | `graph_version=True` accepted because `bool` is an `int` | `schemas` `graph_version` predicate | `test_schema_coercion_matrix_is_unchanged` |

Defects 1, 2, 3, 9, 10 and 11 cluster into one theme: a value or exception that
"looks close enough" is accepted or swallowed. Defects 4 and 5 are the same
theme at the verifier layer: the verifier trusts its input rather than binding
it to the frozen authority. Defects 6, 7 and 13 are metric or type semantics
that overstate or hide what happened.

## Evidence that the new mechanisms work

The mechanisms were also checked for the failure mode they are meant to prevent:
a check that always passes because it never exercises anything.

- Invariant library: each predicate has a direct state tamper that must make it
  fire, and every predicate is asserted absent on a healthy state and present on
  its tamper.
- Seeded fuzzer: anti-vacuity guards require each seed to admit at least four
  events and the seed set at least eighty in aggregate.
- Crash matrix: three representative workflows (accept, decomposition, timeout)
  are re-run crashing before commit at every one of 17, 8 and 10 event indices,
  with recovery asserted against invariants, no redispatch, and idempotence.
- Applicability guard table: per-machine declared-pair counts are pinned
  (for example 120 for the node machine), so deleting a guard fails the test.
- Boundary table: field lists are generated from the schema and event specs; a
  new field with an uncategorised validator fails the coverage assertion.
- Coverage floor: branch coverage over the frozen corpus replay is held at a
  pinned 48% floor (measured about 50%), so corpus coverage cannot be deleted.

## Verification

- Fast suite: 22 failed, 401 passed, 10 deselected, about 16 seconds.
- Full suite: adds the 10 slow end-to-end experiment tests, about 90 seconds.
- Documentation checks (`scripts/check-docs`) pass, including generated-table
  verification and link checking.
- No source file under `src/` was modified by this work.

All 22 failures are intentional. A reviewer can confirm this by inspection: every
failing test name contains `bug_` or is one of the three static gates.

## Risks and caveats

- The suite is red by design. Until the defects are fixed, `pre-commit` and
  `pre-push` will fail. The work was committed with hooks bypassed, which should
  not become the normal workflow.
- Two recommended tools were deferred: mutation testing (mutmut or cosmic-ray)
  and randomised test ordering (pytest-randomly). Both add signal but are slow
  or destabilising relative to the intentionally red suite. Static patterns are
  instead covered by a local AST scanner and ruff.
- The 7 pending readiness findings (R08, R10, R11, R15, R18, R20, R23) still
  lack permanent regression tests; the registry tracks them with a remediation
  batch each and prevents silent removal.
- Coverage is a floor, not a correctness proof, and the benchmark in the schema
  gate is a coarse budget rather than a pydantic comparison; it exists to make a
  future migration decision evidence-based.

## Recommended next steps

1. Triage the 13 defects by severity and assign owners. Defects 1, 2, 3, 10 and
   11 are input-validation gaps that are cheap to close and should go first.
2. Fix a defect together with its reported test; the test turning green is the
   definition of done.
3. Promote pending readiness findings to covered as remediation batches land,
   raising `min_covered` in the registry each time.
4. Only then re-enable the commit and push hooks as the default workflow, and
   turn on the forge pipeline.
5. Revisit the deferred tooling (mutation testing, order randomisation) once the
   suite is green, so a new failure is unambiguously a new defect.
