<!-- informative -->

# Reviewer Brief: R08 Evaluator-Isolation Adversarial Review

## Mandate

Independently attempt to falsify the R08 evaluator-isolation guarantee on the current revision. The automated suite
is green and is not a verdict; the SPEND-gate and EXECUTION-gate reviews each found reachable P1 falsifications that
the delivered mechanisms missed. Assume the happy path is covered and attack the boundary: what candidate code can
reach, suppress, forge, or escape.

**Independence:** you must not be the author of commit `18ad723` (Batch 4) or of the R08 tests. Read the source and
the normative documents before the implementer's tests.

## Target

- Repo: `/home/ruben/Projectos/meta-project/orchestrator`
- Revision reviewed: `18ad723` (Batch 4, R08 evaluator isolation). Preceded by `f025a76`, `85959ad`, `33945bb`,
  `5be1c23`.
- Work on a frozen revision, not a moving tree:
  `git worktree add --detach /tmp/kilo/r08-review-18ad723 18ad723`
- Primary reference: `docs/report-milestone8-readiness.md`, finding **R08** (its "Required" and "Regression" clauses
  are the acceptance criteria).
- Normative documents: `docs/experiment.md` (the independent-oracle section), `docs/executors.md`,
  `docs/control-plane.md`.
- Baseline: `.venv/bin/python -m pytest -q` (expect 509 passed). A green suite is not a verdict.
- Run probes under `/tmp/kilo/`; do not edit source; no network, no paid calls, synthetic fixtures only.

## The guarantee under review

A hidden check must not share an interpreter with candidate code, and candidate code must not be able to reach the
expected values, suppress the observation, forge a passing observation without solving the task, or escape the
boundary to influence grading. Supporting claims:

1. **Expected-value secrecy:** no expected value, check id, or completion sentinel is reachable from the sandbox.
2. **Forgery resistance:** a forged or injected observation passes only if it equals the hidden expected value.
3. **Failure mapping:** early exit, suppression, malformed output, timeout, and infrastructure failure map to FAIL,
   CANDIDATE_TIMEOUT, and INFRASTRUCTURE_ERROR respectively, and can never be PASS.
4. **Boundary integrity:** the candidate cannot reach the collector/controller or alter the comparison, and teardown
   leaves no child, cgroup, or temporary artefact.
5. **Determinism and disclosure:** grading is deterministic per tree, tamper detection survives a concurrent writer,
   and the manifest's execution disclosure matches what actually runs.

## In scope

- `experiment.py`: `_HIDDEN_DRIVERS`, `_HIDDEN_EXPECTED`, `_collector_script`, `hidden_runner`, `_HIDDEN_ENV`,
  `hidden_suite_manifest_hash`, `build_hidden_oracle`, and how `oracle.grade_manifest` invokes the runner.
- The candidate driver / collector / controller comparison path, end to end.
- The per-check disposable tree copy and its digest check.
- The observed failure mapping and the resource behaviour of a check (child process, output bounding, teardown).

## Out of scope

- The SPEND gate and the EXECUTION gate (already reviewed and fixed).
- R10 repair/replan, R13 grade persistence, R14 recovery breadth, R17 output admission, R18 scheduling, R20 metrics,
  R22 timers — owned by Batches 5 and 6. Incidental interactions are welcome, but keep the verdict scoped to R08.

## Known residuals — acknowledge, do not re-report as novel

- **Unforgeability.** A candidate that already knows the expected values could forge the child's JSON. The claim is
  *reachability of expected values from the sandbox*, not unforgeability. Test reachability.
- **Frozen calibration.** The five-case calibration is unchanged; early exit and evaluator interference are pinned by
  dedicated hidden-runner tests rather than calibration cases. Extending the frozen calibration is deferred.
- **Controller death.** Teardown runs only if the controller survives; unchanged and documented.

## Attacks to attempt (minimum)

- **Expected-value secrecy:** search the materialised tree, `/work`, `/tmp`, the collector's argv and
  `/proc/<pid>/cmdline`, `/proc/<pid>/environ`, and the driver's argv for expected values, check ids, and sentinels.
  A driver or collector argument that embeds the expected payload is a finding.
- **Forgery:** candidate prints `DRIVER:` and `COLLECT:` lines before and after the driver's own line; registers
  `atexit`, threads, or signal handlers to emit later; writes to fd 1 or fd 2 after the observation; emits multiple
  `DRIVER:` lines. Confirm the collector selects the intended line and that a payload unequal to the expected value
  never passes.
- **Suppression/DoS:** candidate closes or redirects stdout, replaces `sys.stdout`, monkeypatches `json`, `subprocess`,
  or `builtins`, kills or signals the parent collector, spawns a fork bomb, writes unbounded output, spins to the
  timeout, and exits with assorted codes. Confirm no non-PASS is misclassified as infrastructure and that host state
  is clean.
- **Boundary escape:** candidate attempts to reach the collector/controller (ptrace, `/proc` of the parent, inherited
  fds, signals), or to leave the sandbox. Confirm grading is unaffected and teardown kills the child.
- **Tree-mutation race:** a background writer mutates `/work` between the snapshot and the digest comparison; confirm
  the tamper check catches it or the race cannot turn a broken candidate into a PASS.
- **Determinism and disclosure:** grade the same tree twice; grade a tree mutated between grades; compare the
  manifest's `execution` disclosure to the code path that actually runs.
- **Resource honesty:** is the child's captured output bounded in memory? Are the child, its cgroup, and the per-check
  tree gone after a check, including on timeout and on failure?

## Method

- Read first: `experiment.py` (the symbols above), `oracle.py` (`grade_manifest`, `rerun_non_gradeable`,
  `OracleCheck.probe`), and `sandbox.py` (`SandboxRunner.run`, `_kill`, teardown) for how the check is executed.
- Build minimal reproducers that isolate one claim each. Prefer a failing reproducer to an argument.
- For each attack record: exact commands, observed result, expected result, and whether the claim is violated.
- Do not weaken or edit tests; report a failing reproducer instead.

## Deliverable

1. **Verdict:** GO / NO-GO for the R08 evaluator-isolation guarantee, one sentence.
2. **Findings**, each with severity (P0/P1/P2), the claim violated, a minimal reproducer (commands and code),
   observed versus expected, and a proposed permanent regression test.
3. **Claims verified:** for each in-scope claim, how you attempted falsification and what you observed. "Verified"
   means attacked, not that tests pass.
4. **Residual unknowns:** anything you could not test and why.
5. **Reachability audit:** every path by which sandboxed candidate code could observe an expected value or influence
   the comparison, and why each is closed or not.

## Exit criteria

- Every in-scope claim is either falsified with a reproducer or explicitly "attacked, not falsified" with the method.
- No source edits, no network, no paid calls; host state left clean (no orphaned processes, cgroups, or temp trees).
- Findings are reproducible from the report alone, against `18ad723`.
- The review states what it did not cover.

## Reproduce everything

List every probe command here so the verdict can be re-run against the corrective revision.
