<!-- informative -->

# Reviewer Brief: Untrusted Model-Reply Adversarial Review

## Mandate

Independently falsify the untrusted-model-reply boundary on the current revision. A model reply is the primary
untrusted input once a real provider is used. The SPEND, EXECUTION, R08, and MEASUREMENT gates returned GO, yet two
subsequent real-capable runs produced falsifications the green suite had missed (a non-JSON reply aborted the run; a
schema-invalid reply aborted the run). Both were fixed; the boundary is not exhausted. Assume the happy path is
covered and attack the reply path.

**Independence:** you must not be an author of `d6b48c9` or `9a63e25` (the reply-rejection and provider-failure fixes)
or of the reply-boundary tests. Read the source and the normative documents before the implementer's tests.

## Target

- Repo: `/home/ruben/Projectos/meta-project/orchestrator`
- Revision reviewed: `d6b48c9` (reply rejection recording and debug transcript).
- Work on a frozen revision:
  `git worktree add --detach /tmp/kilo/reply-review-d6b48c9 d6b48c9`
- Normative documents: `docs/executors.md` (trust boundary), `docs/control-plane.md` (model-output admission, crash
  recovery), `docs/glossary.md` (reasoning output types, semantic verdicts, model output states), and the generated
  tables `docs/tables/model-output-lifecycle.yaml`, `docs/tables/attempt-lifecycle.yaml`.
- Baseline: `.venv/bin/python -m pytest -q` (expect 545 passed). A green suite is not a verdict.
- No source edits, no third-party network, no real credentials, no paid calls. A **local stub HTTP server is the
  recommended harness**: point the client at `http://127.0.0.1:<port>` and return crafted replies. Enable the
  transcript with `ORCHESTRATOR_DEBUG_TRANSCRIPT=1` (or a path) so the exact prompt and reply are inspectable.

## The guarantee under review

A model reply is untrusted text. It must not affect authoritative state except by becoming a **validated typed
output**, and any reply that fails validation must be **recorded as a rejection**, must not be admitted, must not
abort the trial, and must never be graded as a pass or leave a live run or unaccounted spend.

Supporting claims:

1. **Strict admission:** only a schema-, provenance-, authority-, scope-, version-, hash-, and policy-valid output is
   admitted; everything else is rejected.
2. **Recorded rejection:** a rejected reply produces `MODEL_OUTPUT_REJECTED` with the trusted reason only; the
   untrusted body is never stored, and the trial continues on its deterministic result.
3. **No abort / no live run:** no reply can raise an exception out of the trial or leave an unresolved, unsettled
   attempt.
4. **Identity binding:** provenance (`model_id`, `model_version`, `call_id`, `prompt_hash`, `policy_version`) is bound
   to the recorded call; a reply cannot choose or forge it.
5. **Resource safety:** reply size, depth, and structure cannot exhaust memory or hang past the wall deadline.
6. **Accounting honesty:** reported usage is charged without clipping; a billed call is not silently lost.
7. **Debug data safety:** the opt-in transcript is bounded, redacted, `0600`, disabled by default, and never contains
   the credential.

## In scope

- `scripts/smoke_run.py`: `OpenAICompatibleClient` (`_post`, `complete`, `_record`, `parse_provider_completion`,
  `_extract_json_object`), `transcript_write`, and the probe/full-run wiring.
- `src/orchestrator/model_adapter.py`: `ModelAdapter.dispatch`/`_dispatch`, completion caching, `validate_output`,
  `_OUTPUT_SCHEMAS`, `_SCOPE_PATH_FIELDS`, provenance checks.
- `src/orchestrator/controller.py`: `admit_model_outputs`, `admit_observation_and_result`, `dispatch_attempt` failure
  handling, `_execute_leaf`-facing `ExecutorFailed`.
- `src/orchestrator/reducer.py`: `MODEL_OUTPUT_ADMITTED` / `MODEL_OUTPUT_REJECTED` / `MODEL_OUTPUT_CONSUMED` handling
  and the model-output table.
- The `MODEL_OUTPUT_REJECTED` transition and its generated tables/catalogue entries.

## Out of scope

- The SPEND, EXECUTION, R08, and MEASUREMENT gates (already reviewed and GO), and the other Batch 6 findings.
- Provider-specific API semantics beyond the OpenAI-compatible shape; treat a different transport as an adaptation,
  not a finding. Incidental interactions are welcome, but keep the verdict scoped to the reply boundary.

## Known residuals — acknowledge, do not re-report as novel

- **Client-reported usage.** The adapter cannot verify provider-side token counts; a malicious client (not a reply)
  can under-report usage. The reply boundary is the text, not the transport's honesty.
- **Billed-but-unrecorded parse failures.** A reply that returns usage but cannot be parsed is billed by the provider
  but currently raises before the cost is computed, so it is not recorded. Quantify the exposure and state whether it
  blocks; closing it means carrying usage on the failure.
- **Unforgeability when values are known.** A reply that already knows the expected hidden values could masquerade;
  the R08 claim is reachability, not unforgeability.
- **Transcript is debug data.** It is bounded, redacted, opt-in, and separate from the event store; it is not evidence.
- **Local stub only.** Real provider quirks (streaming, tool calls, refusals, provider error bodies) are simulated.

## Attacks to attempt (minimum)

- **Shape and decoding:** empty reply; non-JSON prose; JSON in code fences or with prose around it; a JSON array or a
  scalar; truncated JSON with `finish_reason=length`; content as typed parts; multiple choices; nested content with
  `\u0000`/invalid UTF-8; a reply larger than the field bound; deeply nested JSON (parse DoS).
- **Schema and semantics:** missing/extra fields; wrong types; a non-catalogued verdict; an output type that mimics
  another (`ARTEFACT` vs `OBSERVATION`); an empty or whitespace-only rationale; duplicate `output_id`s across a reply.
- **Forgery and authority:** a reply that sets its own `model_id`/`model_version`/`call_id`/`prompt_hash`/
  `policy_version`; a reply claiming `PASS` to override a failing executor record; a reply referencing artefacts or
  paths outside the delegated scope; a reply embedding hidden content it should not have seen.
- **Scope/leakage:** path traversal in output fields; references to unresolved artefacts; a body crafted so a rejected
  reply could reach the store or the transcript unredacted.
- **Resource and lifecycle:** a reply that hangs the server past the wall deadline; a reply that arrives after the
  attempt was cancelled; two concurrent deliveries of one attempt/token with an invalid reply; a reply whose
  validation fails after the attempt already reached a terminal state.
- **Accounting:** a reply whose usage is huge or non-finite; a reply that reports usage but fails to parse; a reply
  that overruns the commitment.
- **Failure mapping:** confirm each reply failure maps to `MODEL_OUTPUT_REJECTED` / `CANDIDATE_TIMEOUT` /
  `INFRASTRUCTURE_ERROR` as appropriate, never to a pass, and never leaves a live or unsettled run.

## Method

- Read the in-scope source first. Build a local stub provider that returns crafted replies to the real client, so the
  whole path (transport → parse → adapter caching → admission → reducer) is exercised.
- Enable the transcript for your probes; use it to confirm what was actually sent and received.
- For each attack record: exact commands, the crafted reply, observed result, expected result, and whether the claim
  is violated. Prefer a failing reproducer to an argument.
- Do not weaken or edit tests; report a failing reproducer instead.

## Deliverable

1. **Verdict:** GO / NO-GO for the untrusted-model-reply boundary, one sentence.
2. **Findings**, each with severity (P0/P1/P2), the claim violated, a minimal reproducer (stub reply and commands),
   observed versus expected, and a proposed permanent regression test.
3. **Claims verified:** for each in-scope claim, how you attempted falsification and what you observed.
4. **Accounting assessment:** the exposure of any billed-but-unrecorded reply and whether it blocks.
5. **Residual unknowns:** anything you could not test and why.

## Exit criteria

- Every in-scope claim is either falsified with a reproducer or explicitly "attacked, not falsified" with the method.
- No source edits, no third-party network, no real credentials, no paid calls; host state left clean.
- Findings are reproducible from the report alone, against `d6b48c9`.
- The review states what it did not cover.

## Reproduce everything

List every stub-server scenario and probe command here so the verdict can be re-run against a corrective revision.
