<!-- informative -->

# Axis 2 — Robustness/Load Stress Report

Axis 2 of [milestone8-handoff.md](milestone8-handoff.md#axis-2--robustnessload-stress-do-first):
validate the operational envelope under real provider pressure before any long
capability run. This report records what was run, what the acceptance criteria
showed, the provider-fault observations, and the retry/backoff decision.

Driver: `scripts/stress_run.py`. Stub provider: `scripts/provider_stub.py`
(scenario catalogue: hostile replies, hostile content, provider faults).
Regressions: `tests/test_axis2_robustness.py`. Evidence (git-ignored):
`evidence/axis2-stress-20260923T174144Z/` (clean standard wave) and
`evidence/axis2-stress-20260923T172825Z/` (first wave; used for the recovery
demonstration below).

## Method

One wave of 22 concurrent `scripts/smoke_run.py --json` processes, each with its
own workdir and spend authorization (the per-workdir ledger is the isolation
unit), all against one local stub except one real-provider control. Hostile
workloads run over seed 1003, whose deterministic script builds a broken tree:
a pass can appear only if a hostile reply manufactures one. Controls run over
seed 1001. Every child ran with the debug transcript on and
`ORCHESTRATOR_MAX_MODEL_CALLS` raised so the bounded repair loop had budget to
re-issue real calls.

| Group | Workloads | Provider behaviour |
| --- | --- | --- |
| hostile replies | truncation, oversized body/reply, invalid verdict (incl. oversized), prose, empty, invalid JSON body, deep nesting, reported-usage-but-unparseable, slow drip, endless drip | replies that must never be admitted or pass |
| hostile content | echo secret, forged fields | well-formed envelopes carrying hostile payloads |
| control | 2 × valid stub, 1 × real (`kilo-auto/free`) | the matrix stand-ins, in parallel with the storm |
| retry | `flaky_5xx` (one leading failure) | repair-loop re-issue under raised call budget |
| faults | 429 (bounded backoff), 429→valid, 5xx, stalled response | rate limits and ambiguous failures |

## Acceptance results (all twelve checks passed)

Against the handoff's Axis 2 acceptance line:

| Acceptance | Check | Result |
| --- | --- | --- |
| no unhandled exception escapes a trial | `no_unhandled_exception` | every child exited with a documented code, no traceback, one JSON document; every trial reached a recorded terminal outcome |
| no pass produced by any hostile/oversized/malformed reply | `no_pass_from_hostile`, `admitted_bodies_clean` | 56 hostile trials (14 hostile workloads × 4 arms): zero `ACCEPTANCE_CLAIMED`, all `CORRECTLY_UNRESOLVED`; unadmittable classes admitted zero model outputs; admitted bodies carried exactly the three schema fields and a catalogued verdict |
| deadlines bound every call | `deadlines_bound` | every child inside its workload budget; drip/stall classes failed closed at the per-call deadline (`ORCHESTRATOR_PROVIDER_TIMEOUT` = 2–3 s) |
| ledger `committed` == summed settled actuals, zero outstanding reservations | `ledger_consistent`, `failures_settled`, `stores_complete` | every workdir: `in_flight == {}`, `committed == Σ actual`; every attempt terminal and `FINALLY_SETTLED` with a recorded actual; every ledger-recorded trial has a readable store (so the trial-derived checks cannot pass vacuously) |
| failures recorded and settled | `failures_recorded`, `failures_settled` | every hostile/fault trial recorded its failure events (DISPATCH_FAILED / EXECUTOR_FAILED / MODEL_OUTPUT_REJECTED / FAILURE_RECORDED) and settled them |
| transcripts show no truncation | `transcripts_sound` | every transcript line is complete JSON (no torn writes under load); truncated completions are recorded as errors and admit nothing; control transcripts contain zero truncation |
| no credential bytes in any store | `no_credential_bytes` | raw scan of every workdir file and the stub capture for each child's secret (including the real API key): zero hits |

Retry criterion (scope item 2), from the `retry-flaky-5xx` workload and
`tests/test_axis2_robustness.py`: the first review failed and the bounded repair
loop re-issued the call as a **fresh attempt** (`ATTEMPT_PLANNED` causes
`INITIAL` then `RETRY`, distinct ids), each **billed** (the failed call settled
conservatively at its 5.0 commitment, the re-issue reconciled its reported
actual) and **settled** (`FINALLY_SETTLED`); the trial then completed
`CORRECT_ACCEPTANCE` and the workdir ledger closed with nothing in flight.

Rate-limit criterion: with `flaky_429` (two leading 429s) the client's bounded
backoff absorbed the rejections inside one attempt — no dispatch failed, no
review was re-issued (`rate_limit_recovery`).

## Provider-fault observations (scope item 4)

Observed in the pre-change client and pinned by
`tests/test_axis2_robustness.py`:

| Fault | Handling before | Handling now |
| --- | --- | --- |
| HTTP 429 | fail closed on the first rejection; the attempt fails and is settled conservatively (5.0 charge per rate-limit blip); two identical failures drive the node terminal | bounded backoff retries **within the one per-call deadline** (`ORCHESTRATOR_MAX_RATE_LIMIT_RETRIES`, `ORCHESTRATOR_RATE_LIMIT_BACKOFF`, honouring `Retry-After`, never sleeping past the deadline); on exhaustion the failure path is unchanged |
| HTTP 5xx | fail closed, no retry | unchanged: fail closed, no hidden retry |
| timeout / stall / drip | fail closed at the deadline | unchanged |
| HTTP 400 with `response_format` | intended one re-send without the field — but the re-send called `_post(body)` without the deadline argument and crashed with `TypeError` (latent defect) | the re-send works and shares the original deadline |

### Decision: bounded retry/backoff in the client

**Add bounded backoff for 429 only. Do not add it for 5xx or timeouts.**

- A 429 is a rejection *before processing*: under the conforming-provider
  accounting assumption the call is not billed, so re-sending it cannot double
  charge and cannot desynchronise the ledger. Without it, one rate-limit blip
  cost a full conservative settlement and, twice in a row, the node itself.
- A 5xx or a timeout is an *ambiguous* outcome: the provider may have processed
  and billed the call. A hidden re-send could double-charge while the durable
  accounting records one attempt, which would break the "each retry is a fresh
  attempt, is billed, and settles exactly once" invariant. These therefore keep
  fail-closed single-shot dispatch; any retry is a **fresh attempt through the
  repair policy** (visible in the event log, with its own commitment and
  settlement), not an invisible client loop.
- Bounds: the retry count is small and configurable (default 2), the backoff
  doubles from `ORCHESTRATOR_RATE_LIMIT_BACKOFF` (default 0.5 s) and honours
  `Retry-After`, and the entire retry budget lives inside the one per-call
  deadline. Every retry is written to the debug transcript.

The distinction is recorded in the client's docstring
(`scripts/smoke_run.py`, `OpenAICompatibleClient`) so it survives the move to
the in-tree model boundary.

## Real-matrix results (honest scope)

The real child (`kilo-auto/free`, 4 trials, seed 1001) ran in parallel with the
hostile storm and completed in 424 s. DIRECT recorded `CORRECT_ACCEPTANCE`; the
three decomposed arms recorded `CORRECTLY_UNRESOLVED` because 6 of 9 real calls
timed out at the 60 s provider timeout and the bounded repair loop drove the
affected nodes to recorded terminal outcomes instead of guessing. This is the
transport/accounting smoke shape the runner documents, **not** a model-capability
measurement. The real workdir ledger closed with `committed == Σ actual` and
nothing in flight (0.0007 USD-level cost in COST_UNITS-priced accounting).

## Recovery under interruption

The first standard wave's real child was killed by a driver-side wall budget
that was smaller than four per-trial deadlines (fixed: the budget now budgets
`4 × ORCHESTRATOR_WALL_SECONDS` plus overhead). The kill left one trial's
reservation in flight — the conservative, auditable behaviour. `--resume` then
recovered the unfinished attempt and settled the trial's spend record
(`status: RECOVERED`), after which `in_flight == {}` and `committed == Σ
actual`. Three gaps were closed here: `--resume` now settles the experiment-spend
reservation once it has reconciled the attempts (previously the liability
stayed in flight forever); `run` and `--resume` now take one exclusive
per-workdir lock, so a resume can never mutate a live run (Axis 2 review F2);
and the stress wave pins one `ORCHESTRATOR_AUTHORIZED_AT` stamp per wave so
recovery can reopen the ledger under the recorded authorization.

## Defects found and fixed by this axis

1. **Provider failure escaping a batched trial.** A model-review failure inside
   `_execute_leaf_batch` (the concurrent arm's batched reviews) raised out of
   `run_trial` unhandled and aborted the whole comparison. The batch now
   records the failure and degrades to the bounded single-leaf repair path, and
   `run_trial` funnels recorded boundary failures (`WorkflowError`,
   `ProtocolViolation`, `CredentialRefused`) into the graded outcome. Pinned by
   `test_concurrent_arm_survives_provider_failures_without_escaping`.
2. **400-response_format re-send crashed** with `TypeError` (missing deadline
   argument). Fixed and pinned by `test_format_rejection_resends_within_the_same_deadline`.
3. **Unrecorded client failures.** A reply that was not valid JSON and a
   reply-size violation surfaced as bare exceptions without a transcript entry.
   Both are now recorded before surfacing (`transcripts_sound`).
4. **`--resume` left the experiment-spend reservation in flight** (see above).
5. **The `retries` metric was structurally zero** — repair-loop re-issues were
   planned as `INITIAL`. Attempt cause is now derived from authoritative state
   (a re-issue after a terminal attempt of the same action is `RETRY`), which
   makes the metric honest and is what the retry criterion is asserted from. The
   milestone-6 integrity test was updated to the honest semantics.
6. **End-of-trial accounting sweep.** A timed-out attempt used to stay
   `PROVISIONALLY_SETTLED` when the trial completed; `run_trial` now applies the
   same idempotent reconciliation recovery does, so every completed trial's
   attempts are `FINALLY_SETTLED` (this is what `failures_settled` checks).

## Residuals

- The provider credential used by the real matrix is still the long-lived key in
  `run.sh` (the handoff's rotate-later residual). Nothing in this work printed
  or persisted it; the credential scan covers every workdir and the stub
  capture.
- A process killed mid-trial keeps its reservation visible in flight (by
  design, until settlement). Recovery settles it when `--resume` runs under the
  recorded authorization; under a *different* authorization a resume still
  reconciles the attempt accounting (idempotent) but declines spend settlement
  with a NOTE and leaves the reservation visibly in flight.
- The client trusts the conforming-provider accounting assumption for its 429
  path (a hostile provider that bills 429s can make unrecorded charges within
  the bounded retry count). 5xx/timeouts are not exposed to this assumption.
- Real-provider rate limits were not deliberately hammered; the real matrix
  experienced only ordinary one-child load. The 429 behaviour is pinned against
  the stub.

## Adversarial review outcome

[The independent review](axis2-robustness-review.md) returned **GO** with two
P1 findings (both now fixed) and six lower-severity findings (all now fixed
except the two noted below). Its eight guarantee attacks otherwise held: cause
derivation, settlement, funnel, 429 bounds, transcript, and credential claims
were "attacked, not falsified".

| Finding | Fix |
| --- | --- |
| F1 (P1) deleted trial store vacuously passed the trial-derived checks | new `stores_complete` check reconciles ledger records against openable stores; `_trial_records` reports unreadable stores as findings |
| F2 (P1) `--resume` racing a live `run()` corrupted the live trial and crashed with a traceback | one exclusive per-workdir lock for `run`/`--resume` (the loser is refused with exit 1, documented); `spend.SpendLedgerError` is funneled to a clean exit 3 instead of a traceback |
| F3 (P2) `check_deadlines_bound` could not fail | the driver records the kill explicitly (`timed_out`) and the check owns it |
| F4 (P2) batch degrade dropped reviewed-but-unexecuted siblings | the degrade path now handles every batch member (reviewed siblings execute with their original review); never-attempted nodes after an abort are documented residue |
| F5 (P2) `recover()` left PLANNED/RESERVED stragglers accounting-OPEN | recovery cancels and settles them regardless of run outcome |
| F6 (P2) `plan_attempt` accepted a caller-manufactured cause | the `cause` parameter is gone; the cause is always derived from state |
| F7 (P2) "64 hostile trials" did not reproduce | corrected to 56 (14 workloads × 4 arms) |
| F8 (P2) milestone-6 `retries` equality was pinned to scenario shape | the metric is asserted from the store's RETRY causes; the validation-failure relation is a comment-grade cross-check |
| F9 (P3) "refuses" overstated the wrong-authorization resume | documented behaviour: attempts reconcile, settlement declines with a NOTE, exit 0 |

Not changed (disclosed residuals, not defects): the 429 conforming-provider
billing assumption, and transcript durability under a genuinely concurrent
writer (no in-tree writer exists today; the review flags it for Axis 1's
patch-producing path). The review's Axis 1 design input — prompt-variant
sensitivity as the agreement measurement, open question 3 decided before
Axis 1 — is carried in [the review](axis2-robustness-review.md#6-secondary-question--axis-1-design-input).
