<!-- informative -->

# Axis 2 — Robustness/Load Stress: Adversarial Review

Review of the Axis 2 change set described in
[axis2-robustness-review-brief.md](axis2-robustness-review-brief.md) against the
implementer's [axis2-robustness-report.md](axis2-robustness-report.md).

**Frozen revision reviewed:** `0bfe48c542234d8032375bd2a39b87f4708473e1` — a
worktree snapshot at `/tmp/kilo/axis2-review` of the main tree `34d6ada` plus
the uncommitted Axis 2 change set (blob-verified identical to the main working
tree for every in-scope file; the stray untracked `home/` directory is
excluded; git-ignored `evidence/` stays in the main repo). Every claim below
was re-run against that worktree. No source files were edited.

Baseline confirmed: **573 passed** (`.venv/bin/python -m pytest -q`, 151 s) and
`scripts/stress_run.py --scale tiny --skip-conformance` → exit 0, all eleven
checks pass (wave root kept at `/tmp/kilo/axis2-base`, report at
`/tmp/kilo/axis2-base.json`). The implementer's two evidence waves were
re-derived from their JSON reports (see F7).

## 1. Verdict

**GO** — the client, funnel, cause derivation, and settlement invariants held
under every falsification attempt; the two P1 findings live in the stress
driver's check soundness and the new `--resume` settlement's concurrency
surface, both fixable without touching the fail-closed core, and per the brief
P1s must be fixed before the axis is called done.

## 2. Findings

### F1 (P1) — Deleting one trial store defeats every trial-derived stress check (vacuous pass)

- **Claim violated:** stress-check soundness ("each corresponding check must
  fail"; "a vacuous pass is a P1 finding").
- **Reproducer:** baseline tiny wave root `/tmp/kilo/axis2-base`; copy it, then
  `rm -rf <root>/hostile-truncated/trials/DIRECT-1003`; run the real check
  functions from `scripts/stress_run.py` against the mutated root (probe:
  `/tmp/kilo/axis2-probes/probe_soundness.py`).
- **Observed:** **all eleven checks pass.** `check_pass_and_failures`,
  `check_failures_settled` (and the retry/rate-limit checks) iterate
  `_trial_records(workdir)` from disk; the missing trial is simply skipped.
  `check_ledger_consistent` counts only records inside
  `experiment-spend.json`, which still holds all four trial records.
- **Expected:** at least `no_pass_from_hostile`/`failures_recorded` must fail
  for the workload whose hostile evidence was removed.
- **Related:** deleting a whole workdir fails only `ledger_consistent` +
  `transcripts_sound`; the hostile-evidence checks are still vacuous. Tampering
  attacks are caught (classification swap → `no_pass_from_hostile`; cloned
  accepted store → three checks fire; torn transcript line, in-flight
  reservation, and planted secret each fail their check), so the gap is
  specifically *missing* stores.
- **Permanent test:** a stress-run self-check that reconciles, per workdir, the
  set of `experiment-spend.json` trial records against the set of
  `trials/<arm>-<seed>/events.db` stores present and openable (name and count),
  failing on any recorded trial without a matching store (and on any store that
  fails to open, which today crashes the driver with a raw `ReplayError`).

### F2 (P1) — `--resume` racing a live `run()` on one workdir corrupts the live trial and escapes with an unhandled exception

- **Claim violated:** "No unhandled exception escapes a trial" at the runner
  boundary; the interrupted-trial workflow (resume is advertised under the same
  recorded authorization).
- **Reproducer:** start `scripts/smoke_run.py --json` in real mode against the
  stub (`/s/valid?delay=10`, `ORCHESTRATOR_PROVIDER_TIMEOUT=25`); poll until
  `trials/DIRECT-1001/events.db` exists and the first review is in flight
  (~9 s); then start `scripts/smoke_run.py --resume` on the same workdir and
  env. Probe: inline in
  `/tmp/kilo/axis2-probes` (race variants; final form `resume-race4-*`).
- **Observed:** the resume opened the **live** trial's store, failed its
  in-flight attempt (`ATTEMPT_FAILED_ON_RECOVERY`), settled the live trial's
  spend record mid-flight (`status: RECOVERED`, 5.000018), and the still-running
  comparison then crashed when its own `run_trial` tried to settle:
  `spend.SpendLedgerError: trial 'EXP-RECURSIVE_LAZY-1001' was never reserved`,
  as a full traceback out of `smoke_run.py` with exit code 1 (not in
  `DOCUMENTED_EXIT_CODES`). Second run of the same race reproduced the crash at
  the same settlement point.
- **Expected:** the runner must refuse to resume a workdir with a live writer
  (or the recovery must refuse trials whose run is not provably dead), and no
  exception may escape the runner.
- **Note:** killing the run first (SIGKILL) and then resuming works correctly in
  every variant tried: reservations settle at reconciled actuals, double resume
  is a no-op, resume after completion is a no-op, and a wrong-authorization
  resume leaves the reservation in flight (see F9's wording note). The defect
  is only the missing mutual exclusion with a *live* writer.
- **Permanent test:** hold an exclusive lock file per workdir for the duration
  of `run()` (created before the first reservation, released on exit); `--resume`
  must refuse with a documented exit code when the lock is held; regression test
  races a real `run()` and a `--resume` in two processes and asserts the run
  finishes with its own classification and the resume exits refused.

### F3 (P2) — `check_deadlines_bound` cannot fail by construction

- **Claim violated:** "a check that survives none of them is a finding".
- **Observed:** `run_child` calls `subprocess.run(..., timeout=workload.wall_budget)`;
  a child that exceeds the budget raises `TimeoutExpired` and is recorded with
  `wall_seconds == workload.wall_budget`, so `wall > budget` is never true; a
  normally returning child necessarily has `wall < budget`. The check's findings
  list is unreachable; over-budget children are only caught indirectly by
  `no_unhandled_exception` (exit −1).
- **Expected:** the deadline check should own this signal — e.g. measure a
  slightly smaller kill budget than the workload budget, or record and compare
  the timeout event explicitly.
- **Permanent test:** construct a `ChildResult` with `wall_seconds` above the
  budget and assert the check reports it (it does — the finding is that the
  driver can never produce that input), or restructure the timeout/budget pair.

### F4 (P2) — Batch degrade drops reviewed-but-unexecuted siblings without a recorded terminal node state

- **Claim attacked:** trial-stop integrity; the brief's expectation that
  "sibling nodes reach recorded terminal state" after a mid-batch kill.
- **Reproducer:** `run_trial` on `ARM_RECURSIVE_CONCURRENT` with a client whose
  second review call fails (probe `/tmp/kilo/axis2-probes/probe_batch.py`,
  `fail_index=2`, seeds 1001 and 1003).
- **Observed:** the first node's review succeeded; the batch degraded to
  per-leaf execution for `node_ids[index:]` only. The earlier-reviewed sibling
  is never executed and stays `READY` (root `BLOCKED`, run outcome `None`,
  classification `CORRECTLY_UNRESOLVED`). Everything is settled (all attempts
  terminal + `FINALLY_SETTLED` with actuals), the failed review is re-issued as
  `RETRY`, the failure fingerprint is recorded exactly once, and the ledger
  closes clean — but the sibling's review work is silently discarded and no
  node-level terminal outcome is recorded for it.
- **Expected (per the brief's attack):** the abort path should drive dropped
  siblings to a recorded terminal state (as `_drive_repair_terminal` does for
  the failed node in the single-leaf path), or the design should state that
  dropped siblings are intentionally left live.
- **Not falsified:** claim 1's letter (recorded classification, settled
  accounting, comparison continues) holds; the gap is the node-graph residue.

### F5 (P2) — `recover()` leaves a `RESERVED`/`PLANNED` attempt accounting-OPEN when the run has no terminal outcome

- **Claim attacked:** "the end-of-trial sweep must finish with every attempt
  terminal and `FINALLY_SETTLED` with a recorded actual".
- **Reproducer:** controller-level probe
  `/tmp/kilo/axis2-probes/probe_sweep.py`: plan+dispatch a `RESERVED`-state
  attempt by an injected error after `ACTION_RESERVED`, then `controller.recover`.
- **Observed:** attempts left `DISPATCHED`/`RUNNING` are failed-on-recovery and
  settled (conservatively or at the recorded actual; exactly one settlement
  event; second sweep is a no-op; a fresh process's view matches). But a
  `RESERVED` attempt with `OPEN` accounting survives both sweeps unless the run
  outcome is already terminal (the `PLANNED/RESERVED` cancel branch requires
  `run_terminal`). State: `RESERVED`, accounting `OPEN`, actual `None`.
- **Expected:** the sweep should cancel PLANNED/RESERVED stragglers regardless
  of run outcome (the cancellation transitions exist in
  `docs/tables/attempt-lifecycle.yaml`).
- **Reachability:** only via an injected error between reservation and
  dispatch today; the stress checks would flag the shape if it ever occurred
  (`failures_settled`). Low likelihood, but it is exactly the invariant the
  sweep exists to enforce.

### F6 (P2) — `plan_attempt` still accepts a caller-manufactured cause

- **Claim attacked:** "the cause is derived from authoritative state and cannot
  be manufactured by a caller or a reply".
- **Reproducer:** `/tmp/kilo/axis2-probes/probe_cause.py`, `manufacture-initial`:
  `controller.plan_attempt(run, node, 5.0, action_type="REVIEW", cause="INITIAL")`
  after a terminal attempt.
- **Observed:** admitted. The cause enum is validated at admission
  (`("INITIAL", "RETRY")` only), but nothing cross-checks the asserted cause
  against the terminal-attempt state, so a privileged caller can mislabel a
  retry as `INITIAL`. A reply cannot reach this path (`plan_attempt` is never
  called from model output), and every in-repo caller omits `cause`, so the
  derivation is authoritative for all real call sites.
- **Expected:** drop the parameter, or raise when the supplied cause disagrees
  with the state-derived one.

### F7 (P2) — Report number "64 hostile trials" does not reproduce

- **Observed:** the named clean wave (`evidence/axis2-stress-20260923T174144Z/stress-report.json`)
  contains 14 hostile workloads × 4 trials = **56** hostile trials (all
  `CORRECTLY_UNRESOLVED`, zero `ACCEPTANCE_CLAIMED` — the qualitative claim
  holds). No grouping of that wave yields 64. The other quantitative claims
  re-derived cleanly: 22 children, all checks pass, real child 423.97 s,
  "6 of 9 real calls timed out" (transcript: 9 entries, 6 × "The read operation
  timed out", 3 ok), recovery story in the first wave (real child exit −1,
  `EXP-RECURSIVE_CONCURRENT-1001` settled `RECOVERED`, `in_flight == {}`).

### F8 (P2) — The milestone-6 `retries` assertion is pinned to the scenario shape, not the semantics

- **Observed:** `assert trial.retries == (1 if trial.validation_failures else 0)`
  (tests/test_milestone6_acceptance.py) is accurate for the frozen experiment
  only because the deterministic build-failure aborts after a single lineage
  fails twice (probe: `FIXED_ONE_LEVEL/1003` → vf=2, retries=1; the concurrent
  arm's fallback aborts before the second failing node would be retried). Two
  validation-failing nodes in one trial (vf=4, retries=2) would break the
  assertion even though the metric would then be *more* honest. The metric
  itself is cause-derived and its consumers (`run_pilots` calibration,
  comparison rows, smoke JSON) carry no structural-zero assumption.
- **Expected:** assert the metric from attempt causes directly (count planned
  `RETRY` attempts in the store) and keep the validation-failure cross-check as
  a scenario-specific comment, not an equality.

### F9 (P3) — "Refuses" overstates the wrong-authorization resume behaviour

- **Observed:** `--resume` under a different authorization exits **0**,
  recovers the trial stores ("recovered 1 attempt(s)"), prints a NOTE, and only
  declines the *settlement* (reservation stays in flight). Defensible
  (recovery is idempotent and settlement is the authorized action), but the
  report's "a resume under a different authorization refuses and says so" does
  not match the implemented split; the smoke run header should document it.

## 3. Claims verified (all eight guarantees)

1. **Trial stop integrity.** Attacked with nine exception types injected at the
   model boundary inside whole trials (`VersionConflict`, `WorkflowError`,
   `ProtocolViolation`, `CredentialRefused`, `BudgetExhausted`,
   `RunLimitExhausted`, `KeyError`, `AssertionError`, `KeyboardInterrupt`).
   Caught kinds: trial returns graded `CORRECTLY_UNRESOLVED` with
   `INFEASIBLE` outcome, all attempts settled, ledger closed. Bug-shaped
   exceptions are converted at the dispatch boundary to `ExecutorFailed` with
   the reason recorded in `DISPATCH_FAILED` (`'bug-shaped'`) — visible, not
   silent. `KeyboardInterrupt` escapes `run_trial` (funnel does not catch
   `BaseException`) and leaves the reservation honestly in flight. Attacked,
   not falsified (see F4 for the node-graph residue; F2 for the one shape that
   does escape, via the resume race).
2. **No pass from hostile replies.** Whole-wave evidence re-derivation (56/56
   hostile trials `CORRECTLY_UNRESOLVED`, zero claims), plus a cloned-store
   injection attack (caught by three checks) and a payload swap (caught).
   `admitted_bodies_clean` verified on the wave. Attacked, not falsified.
3. **Deadlines bound every call.** Probe matrix: `Retry-After` −5/0/huge/
   HTTP-date/missing all bounded inside the deadline; 429 in the last
   microseconds fails closed (1 request, wall 4.97 s of a 5 s deadline); drip
   fails at exactly the deadline (1.00 s of 1.0); 5xx single-shot; 400 re-send
   shares the deadline with the documented ~0.1 s socket-budget floor observed
   within the deadline. The stress check itself is structurally vacuous (F3).
4. **Ledger `committed == Σ settled actuals`, zero outstanding.** Verified on
   the baseline wave, all funnel workdirs, batch-degrade workdirs, kill+resume
   workdirs, and the stub-mid-trial-kill workdir (40.0 committed, nothing in
   flight). Overrun liability: `settle(actual 9.0)` over a 5.0 reservation
   raises `committed` past `available`, and the next reserve is refused
   (`SpendCapExceeded`) — divergence is visible, never silent. Attacked, not
   falsified (F2's race can settle a live record mid-flight — see accounting).
5. **Failures recorded and settled.** Sweep probe: reconciled-with-actual,
   provisional/timed-out, and conservative shapes all reach terminal +
   `FINALLY_SETTLED` with a recorded actual; one settlement event per attempt;
   `total_charged == Σ actuals`; sweep idempotent; a fresh process's view
   matches. Gap: `RESERVED`-state stragglers (F5).
6. **Retries fresh, billed, settled, cause-derived.** `INITIAL→RETRY` derived
   after terminal predecessors of every shape (failed dispatch, `RESULT_PRODUCED`,
   conservative settlement, `CANCELLED`, recovery-failed); lineage isolation
   across action types; replay reproduces causes; non-catalogued causes
   refused; batch degrade re-issues as `RETRY` with one failure fingerprint per
   failed attempt and no double `integration.advance`. Manufacture hole via the
   explicit parameter (F6). No `INITIAL`-after-one found in any in-tree path
   (the thread-pool fallback waits for in-flight futures, so a degraded re-plan
   never races a live attempt).
7. **Transcripts.** Baseline + tampered copies: torn lines are caught,
   truncation visible, controls clean. No concurrent in-tree writer exists
   (sequential reviews; per-child files; the stub capture is lock-guarded), so
   the torn-write claim holds vacuously — noted as a residual, not tested
   against a real concurrent writer.
8. **No credential bytes.** `echo_secret` echoes the bearer verbatim (verified
   at the HTTP level); the admission redaction layer is what protects stores
   (implementer test re-verified); a planted secret in a workdir is caught by
   the raw scan; the wrong-real-key scan covers the capture. Attacked, not
   falsified.

## 4. Accounting assessment

- **429 re-sends** are invisible to the durable ledger by design: the ledger
  records only the completion that produced usage. Against a provider that
  bills rejections, up to `ORCHESTRATOR_MAX_RATE_LIMIT_RETRIES` (default 2)
  worst-case per-call charges (≈0.0024 USD at the stub prices, scale-linear)
  can go unrecorded. Bounded, disclosed, does not block.
- **Recovery settlement** settles from the durably recorded actual when one
  exists and conservatively at the commitment otherwise; the failed call is
  charged at its full commitment (over-charges relative to reality, never
  under). Kill+resume, double resume, and resume-after-completion all keep
  `committed == Σ actual` and nothing in flight.
- **Trial-end sweep** adds no double settlement (single event per attempt,
  idempotent re-sweep, matching fresh-process view).
- **Divergence window:** F2 — a concurrent resume settles a live trial's record
  at its mid-flight total; any further charges by the dying run appear in the
  trial's event store but in no ledger record (the file-level invariant
  `committed == Σ records` still holds, so the stress checks would not catch
  it). This is the only way I found to make the durable ledger diverge from
  what was actually charged, and it is closed by the F2 fix.

## 5. Residual unknowns

- Standard-scale wave not re-run (time); tiny baseline + targeted whole-trial
  probes instead. The implementer's standard-wave JSON was re-derived, not
  re-executed.
- Real-provider 429 behaviour remains stub-only (as the report discloses).
- Transcript durability under a genuinely concurrent writer is untestable
  in-tree (no concurrent writer exists); if Axis 1 ever shares one transcript
  across threads, lines larger than the write buffer can split.
- `--probe` exit 7 was read, not exercised.
- Exit code 7 classification and the `home/` stray directory cleanup are
  open from before this axis; neither blocks.

## 6. Secondary question — Axis 1 design input

**Recommendation: prompt-variant sensitivity as the agreement measurement, with
agreement statistics deferred until a second genuinely different reviewer
exists.** What I observed of the model boundary: one `ModelProfile` (base URL +
model id, temperature 0) feeds every arm; the reply boundary is a thin schema
gate; the grading signal is a `SEMANTIC_EVALUATION` verdict over a frozen
candidate set. Under one provider identity and model, two reviewers are the
same decoder run twice: "two model ids on the same provider" would mostly
measure provider-side routing noise (often the identical base model), so the
agreement number would mean little more than decoding determinism; "two
providers" measures cross-family agreement but imports cost, access, and
context-window confounds that would swamp the signal at free-tier scale;
"prompt-variant sensitivity" measures the decision noise of the grading
boundary itself on a fixed candidate set — which is exactly the quantity the
false-success denominator depends on, and it needs no second identity. The
agreement number under prompt variants would mean "how stable is the pass/fail
boundary under wording perturbation", which is the honest statistic to publish
alongside Axis 1's denominators; under the other two options it would mean
"how correlated two near-identical decoders are", which is not reviewer
independence.

**Open question 1 (exit code 7 classification): decide with Axis 1.** Exit 7 is
today only the `--probe` transport-failure path; Axis 1 reuses `--probe` for
billable patch calls and will want a machine-readable distinction between
"refused configuration" and "provider/transport failure", so the stress
driver's exit-code allowlist and retry decisions can key off it. Nothing in
Axis 2 depends on it.

**Open question 3 (transcript privacy/durability): decide before Axis 1.** The
patch-producing path will put untrusted, larger-than-observation content
(diffs) into transcripts and artefact stores; retention, redaction scope, and
the torn-write/durability policy should be settled before that content exists.
Today's bounded redacted lines (0600) plus the absence of concurrent writers
keep the current claim vacuously safe; Axis 1 changes both premises.

## 7. Reproduce everything

Frozen revision: `0bfe48c542234d8032375bd2a39b87f4708473e1` at
`/tmp/kilo/axis2-review` (worktree of `34d6ada` + the uncommitted change set).

```bash
# baselines
/home/ruben/Projectos/meta-project/orchestrator/.venv/bin/python -m pytest -q   # 573 passed
.venv/bin/python scripts/stress_run.py --scale tiny --skip-conformance \
  --root /tmp/kilo/axis2-base --json /tmp/kilo/axis2-base.json                  # exit 0, 11 checks

# probes (all under /tmp/kilo/axis2-probes/, python = the repo .venv)
probe_cause.py       # cause matrix + manufacture + replay reproduction
probe_funnel.py      # exception funnel over 9 injected kinds (needs NAMESPACE_V1)
probe_batch.py       # _execute_leaf_batch degrade, fail_index in {0,1,2} x seeds {1001,1003}
probe_sweep.py       # end-of-trial sweep, settlement uniqueness, resume-view equality
probe_resume.py      # kill+resume, double resume, resume-after-done, wrong auth, overrun, race
probe_429.py         # Retry-After matrix, deadline edges, interleaved faults (local HTTP server)
probe_soundness.py   # one attack per stress check against copies of the baseline wave
# inline probes (in the transcript): stub-kill mid-trial; stub fidelity matrix
#   (echo verbatim, oversized pre-parse, drip deadline, per-credential flaky counters)
```

Evidence kept: `/tmp/kilo/axis2-base/` + `/tmp/kilo/axis2-base.json` (baseline
tiny wave), `/tmp/kilo/axis2-probes/` (probe scripts), the mutated wave roots
`/tmp/kilo/soundness-*`, and the probe workdirs (`cause-*`, `funnel-*`,
`batch-*`, `resume-*`, `stub-kill-*`). The implementer's evidence waves
(`evidence/axis2-stress-20260923T174144Z`, `…T172825Z`) were re-derived from
their committed JSON reports, not re-run.

Not covered: standard-scale wave re-run, real-provider 429 hammering,
non-Linux behaviour, third-party network, real credentials.

## 8. Fix-round verification (confirmation)

Fix round frozen at `58a96c12dd07403976c0402255f8ca3690922274`
(`/tmp/kilo/axis2-review-v2`); diff vs the reviewed revision covers exactly the
nine findings plus the recovery documentation. Gates re-run: **579 passed**
(148 s) and `scripts/check` (docs + static + tests) exit 0. Fresh baseline tiny
wave on the fixed tree: exit 0, all twelve checks pass
(`/tmp/kilo/axis2-base-v2`). Every finding re-attacked:

- **F1 closed.** Deleting a hostile trial store now fails `stores_complete`
  ("ledger-recorded trial DIRECT-1003 has no store"); a corrupted store is a
  finding (`store unreadable: DatabaseError`), not a driver crash; deleting a
  whole workdir fails three checks; the classification swap is still caught.
- **F2 closed.** A `--resume` racing a live `run()` is refused with exit 1 and
  a clear lock message before touching trial state; the live run then completes
  normally (exit 0, four `CORRECT_ACCEPTANCE`, nothing in flight, no
  traceback). Kill+resume still works (flock is released by process death) and
  `SpendLedgerError` is additionally funneled to a clean exit 3.
- **F3 closed.** A child killed at its budget now fails `check_deadlines_bound`
  explicitly (`timed_out`).
- **F4 closed.** The degrade path handles every batch member: `fail_index=2`
  on the passing seed now completes the whole trial (`CORRECT_ACCEPTANCE`,
  review re-issued as `RETRY`, sibling executed with its original review); on
  the broken seed the abort residue is only never-attempted nodes, as
  documented in the `_TrialAborted` contract.
- **F5 closed.** A `RESERVED` straggler is cancelled and reclaimed on the sweep
  (state `CANCELLED`, `FINALLY_SETTLED`, actual 0.0) regardless of run outcome;
  sweep idempotent; fresh-process view matches; `docs/control-plane.md` updated.
- **F6 closed.** The `cause` parameter is gone; the manufacture probe now fails
  with `TypeError`, and every in-tree caller derives the cause.
- **F7 closed.** The report now says 56 hostile trials (14 × 4).
- **F8 closed.** The milestone-6 assertion reads `trial.retries ==
  planned_causes.count("RETRY")` from the store — a measurement — with the
  validation-failure relation demoted to a shape comment.
- **F9 closed.** Both the smoke-run header (exit status table incl. 7, the
  reconcile-but-decline-settlement split) and the report's residual now state
  the actual behaviour.

Leftover nit (cosmetic, not re-frozen into the verdict): the report's
acceptance-results heading still says "all eleven checks passed"; the driver
now runs twelve.
