<!-- informative -->

# Reviewer Brief: Axis 2 Robustness/Load — Adversarial Review

## Mandate

Independently falsify the Axis 2 robustness/load change set before Axis 1 (real-model inter-reviewer runs) and Axis
3 (paid capability freeze) rely on it. Axis 2 changed durable semantics that every later measurement depends on: how a
retry is recorded (`plan_attempt` cause derivation), when a trial's accounting is settled (the end-of-trial
reconciliation sweep), how a batched trial survives a provider failure, when a client may re-send a request (bounded
429 backoff), and how `--resume` closes a crashed trial's spend reservation. The implementer's own stress run reports
all acceptance checks green (`docs/research/axis2-robustness-report.md`); a green wave and a green suite are not a
verdict. Assume the happy path is covered and attack the new semantics.

**Independence:** you must not be the author of the Axis 2 change set (the four modified and four new files under "In
scope"), of `tests/test_axis2_robustness.py`, or of `docs/research/axis2-robustness-report.md`. Read the source and
the normative documents before the implementer's tests and report; treat the new tests as claims to check, not as
evidence.

## Target

- Repo: `/home/ruben/Projectos/meta-project/orchestrator`
- Revision reviewed: working tree of `34d6ada` plus the uncommitted Axis 2 change set. Snapshot the tree you actually
  reviewed (worktree or `git stash create`) and name it in the report.
- Work on a frozen copy:
  `git worktree add --detach /tmp/kilo/axis2-review <your-frozen-revision>`
- Normative documents: `docs/budgets.md` (no-overcommitment, settlement), `docs/control-plane.md` (attempt dispatch,
  crash recovery), `docs/executors.md` (trust boundaries), `docs/experiment.md`, `docs/event-catalogue.md`, and the
  generated tables `docs/tables/attempt-lifecycle.yaml`, `docs/tables/accounting-lifecycle.yaml`,
  `docs/tables/limits.yaml`. The attempt causes `INITIAL`/`RETRY` are catalogued; the retry-policy decision being
  derived rather than asserted is normative in `src/orchestrator/retry_policy.py`'s docstring stance.
- Context: `docs/research/milestone8-handoff.md` (Axis 2 scope and acceptance), `docs/research/axis2-robustness-report.md`
  (the claims under review).
- Baseline: `.venv/bin/python -m pytest -q` (expect 573 passed) and `scripts/stress_run.py --scale tiny
  --skip-conformance --root /tmp/kilo/axis2-base --json /tmp/kilo/axis2-base.json` (expect exit 0, eleven checks).
  Neither is a verdict.
- No source edits, no third-party network, no real credentials, no paid calls. A local stub provider is in-tree:
  `scripts/provider_stub.py` (scenario catalogue in its module docstring). Probes under `/tmp/kilo/`.

## The guarantees under review

Axis 2 claims the operational envelope holds under provider pressure. Each claim is taken from the handoff acceptance
line and the report; falsify any of them:

1. **No unhandled exception escapes a trial.** Not from a batched model review (`_execute_leaf_batch`), not from the
   admit/validate boundary, not from a provider fault. A trial may stop early, but only into recorded, settled state,
   and the comparison continues with the next trial.
2. **No pass from a hostile, oversized, or malformed reply.** Nothing from such a reply is admitted; no acceptance is
   claimed over a broken tree; admitted bodies (well-formed hostile content included) carry exactly the three
   semantic fields and a catalogued verdict; forged provenance fields cannot ride along.
3. **Deadlines bound every call.** Including the 429 retry budget, which must live inside the one per-call deadline,
   not on top of it. A drip or a stalled response cannot outlive its deadline by more than the bounded slack.
4. **Ledger consistency.** After a completed run: `experiment-spend.json` has `committed == Σ trial actuals` and
   `in_flight == {}`. After `--resume` of an interrupted run: the same, and the recovered trial's record is settled
   from its reconciled actual.
5. **Failures are recorded and settled.** Every failed call produces its failure event and an attempt that is terminal
   and `FINALLY_SETTLED` with a recorded actual when the trial completes (the end-of-trial sweep must not lose, double,
   or invent money while making that true).
6. **Retry lineage is honest.** A repair-loop re-issue is a fresh attempt recorded with cause `RETRY`, billed (failed
   call conservatively at its commitment, re-issue at its reported actual) and settled. The `retries` metric counts it;
   an accepted-on-first-attempt trial reports zero. The cause is derived from authoritative state and cannot be
   manufactured by a caller or a reply.
7. **The 429 decision holds.** Bounded backoff for 429 only (with `Retry-After` honoured and clamped); 5xx and
   timeouts are single-shot fail-closed with no hidden re-send; the one 400/`response_format` re-send shares the
   original deadline. Every re-send is visible in the transcript.
8. **Transcripts and credentials.** Transcript lines are complete JSON under concurrent load, truncated completions
   are recorded and admit nothing, control runs show no truncation, and no raw credential bytes appear in any store,
   transcript, or stub capture.

## In scope

- `src/orchestrator/controller.py`: `plan_attempt` cause derivation, `dispatch_attempt` failure handling as it
  interacts with the new callers.
- `src/orchestrator/experiment.py`: `_execute_leaf_batch` (the batched-review degrade path), `run_trial` (the boundary
  exception funnel and the end-of-trial `controller.recover` sweep), `_model_review`.
- `scripts/smoke_run.py`: `OpenAICompatibleClient.complete`/`_post`/`_rate_limit_pause` (dispatch loop, 429 backoff,
  400 re-send, failure recording), `resume` (recovery settlement of the experiment spend ledger), `validate`'s
  worst-case cap arithmetic.
- `scripts/provider_stub.py` and `scripts/stress_run.py`: stub fidelity and check soundness (a check that cannot fail
  is not a check).
- `tests/test_axis2_robustness.py` and the changed assertion in `tests/test_milestone6_acceptance.py`
  (`retries == 1 if validation_failures else 0`): is the new semantics actually pinned, or pinned to the
  implementation?
- `docs/research/axis2-robustness-report.md`: every quantitative claim.

## Out of scope

- The SPEND, EXECUTION, R08, MEASUREMENT, and model-reply gates (previously reviewed). Incidental interactions are
  welcome, but the verdict is scoped to the Axis 2 change set.
- Model capability, prompt quality, and the free-tier provider's behaviour as such.
- Axis 1 and Axis 3 design, except for the secondary question below.

## Known residuals — acknowledge, do not re-report as novel

- **429 billing assumption.** The client's bounded 429 re-send trusts the conforming-provider assumption (a 429 is
  not billed). A provider that bills rejections can make charges the durable accounting never sees, bounded by the
  retry count. 5xx/timeouts are deliberately excluded from this assumption.
- **Long-lived credential in `run.sh`.** Rotation is a handoff residual; nothing in the change set prints or persists
  it.
- **Killed-process liability.** A process killed mid-trial keeps its reservation visible in flight by design until
  settlement; `--resume` closes it only under the recorded authorization and says so otherwise.
- **Transcript is debug data.** Bounded, redacted, `0600`, opt-in, not evidence.
- **Real rate limits not deliberately hammered.** The 429 behaviour is pinned against the stub only.
- **Client-reported usage** remains unverifiable (previously reviewed).

## Attacks to attempt (minimum)

- **Cause derivation (`plan_attempt`).** Re-plan after: a failed dispatch, a successful result, a conservative
  settlement, a cancelled attempt, a recovered attempt, an attempt of the *other* action type with the same node.
  Try to obtain a `RETRY` with no terminal predecessor or an `INITIAL` after one; check the reducer accepts only the
  catalogued causes and that replay reproduces the same causes; check `schedule_child_attempts` and `run_model_review`
  land on the intended cause in both the single-leaf and batched paths.
- **The `retries` metric.** Decide whether `retries == 1 if validation_failures else 0` is a measurement or a
  tautology. Construct a trial where validation fails twice on one node and another where two different nodes each
  fail; the metric must describe attempts, not mirror another counter. Search every consumer of `retries` for the
  old "structurally zero" assumption.
- **End-of-trial sweep.** A trial with: a timed-out attempt (provisional), a reconciled attempt, a conservatively
  settled one, and one left `OPEN` by an injected error. Prove no double settlement (a second `ATTEMPT_COST_*` must
  not raise or double-charge), no invented actual, and no changed classification. Compare `run_trial`'s final
  `total_charged` against the same trial crashed and then `--resume`d: the settled actuals must agree.
- **`resume` settlement.** Double `--resume`; resume after full completion; resume with a wrong authorization; resume
  when a trial's reconciled actual exceeds its reservation; concurrent `--resume` and `run()` on one workdir. The
  `RECOVERED` status must not collide with catalogued outcome vocabularies (`docs/glossary.md`).
- **Batch degrade (`_execute_leaf_batch`).** Kill the provider mid-batch after 0, 1, and k successful reviews. Check:
  no double `integration.advance`, no review re-issued as `INITIAL`, sibling nodes reach recorded terminal state, the
  failure fingerprint is recorded exactly once per failed attempt, and `_TrialAborted` leaves nothing live or
  unsettled.
- **Exception funnel (`run_trial`).** Inject each caught type (`WorkflowError` subclasses including `VersionConflict`,
  `ProtocolViolation`, `CredentialRefused`) at review, execute, admit, and grading. The funnel must not swallow a
  `BudgetExhausted` without `budget_exhaustion=True`, must not swallow a bug-shaped error silently (the report must
  still show it), and must not catch `BaseException`.
- **429 backoff.** `Retry-After`: negative, zero, huge, non-numeric, missing; a 429 on the last microseconds of the
  deadline; a 429 storm that must exhaust quickly; interleaved 429/400/500. Prove the total wall time of one `complete`
  call stays within its deadline under every combination and that exhaustion is fail-closed and settled. Then bill the
  stub's 429s (simulate a provider that charges rejections) and quantify what the ledger cannot see.
- **Single-shot 5xx/timeout.** Prove the stub receives exactly one request per attempt for 500/502/503/timeout/stall,
  and that the repair loop's re-issue is a second, visible attempt.
- **400 re-send.** 400 with and without `response_format`; two 400s in a row; a 400 body that is not JSON; the re-send
  crossing the deadline.
- **Check soundness (`stress_run.py`).** Delete or corrupt one workdir, swap a classification in one store, inject a
  manufactured `ACCEPTANCE_CLAIMED` over a hostile workdir, leave one reservation in flight, tear one transcript line,
  put a known secret into one artefact: each corresponding check must fail. A vacuous pass is a P1 finding.
- **Stub fidelity.** Confirm `echo_secret` echoes the received bearer verbatim (so the redaction claim is real),
  `oversized_reply` exceeds `ORCHESTRATOR_MAX_REPLY_BYTES` before any parse, `slow_drip`/`drip_forever` outlast the
  deadline, and flaky counters are per-credential (two concurrent children do not share a failure budget).
- **Report numbers.** Re-derive the report's counts (64 hostile trials, 6 of 9 real calls timed out, committed sums)
  from `evidence/` or from your own wave; state any that do not reproduce.

## Method

- Read the in-scope source and the normative tables first. Drive the real `run_trial`/`smoke_run.py` paths against
  `scripts/provider_stub.py`; prefer whole-trial reproducer over unit-level simulation when the claim is about a trial.
- Keep the transcript on (`ORCHESTRATOR_DEBUG_TRANSCRIPT=1`) and use the event stores (`EventStore.history`/`state`)
  as the source of truth, not the smoke runner's stdout.
- For each attack record: exact commands, the scenario/patch (as a probe, not a source edit), observed result,
  expected result, and which claim is violated. Prefer a failing reproducer to an argument.
- Do not weaken or edit tests; report a failing reproducer instead.

## Deliverable

1. **Verdict:** GO / NO-GO for the Axis 2 change set, one sentence.
2. **Findings**, each with severity (P0/P1/P2), the claim violated, a minimal reproducer (commands and stub scenario),
   observed versus expected, and a proposed permanent regression test.
3. **Claims verified:** for each of the eight guarantees, how you attempted falsification and what you observed
   ("attacked, not falsified" is a valid outcome when the method is stated).
4. **Accounting assessment:** any way the new paths (429 re-send, recovery settlement, trial-end sweep) can make the
   durable ledger diverge from what was actually charged, and whether it blocks.
5. **Residual unknowns:** anything you could not test and why.

## Secondary question — Axis 1 design input (short, opinionated)

The handoff leaves open whether a second reviewer is meaningful when the free tier binds both to one provider identity
and model (`kilo-auto/free`), which would confound inter-reviewer agreement. Given what you observed of the model
boundary, recommend one of: two model ids on the same provider, two providers, prompt-variant sensitivity as the
measurement, or deferring agreement statistics — and say what the agreement number would actually mean under each.
Also state, in one paragraph each, whether open questions 1 (exit code 7 classification) and 3 (transcript
privacy/durability) should be decided before or with Axis 1.

## Exit criteria

- Every in-scope claim is either falsified with a reproducer or explicitly "attacked, not falsified" with the method.
- At least one soundness attack per `stress_run.py` acceptance check (a check that survives none of them is a finding).
- No source edits, no third-party network, no real credentials, no paid calls; host state left clean.
- Findings are reproducible from the report alone against the named frozen revision.
- The review states what it did not cover.

## Reproduce everything

List every stub scenario, environment override, and probe command here so the verdict can be re-run against a
corrective revision, and point at any wave root under `/tmp/kilo/` you keep as evidence.
