<!-- informative -->

# Reviewer Brief: Phase 3 Project-Owned Oracle — Adversarial Review

## Mandate

Independently falsify the project-owned oracle change set before Phase 4 (task
decompositions) builds on it. Two commits are under review:

1. **`d4d6267` — Extend the grading sandbox's toolchain view** (`sandbox_launcher.py`,
   `hidden_oracle.py`, `oracle.py`). Four environment changes so real gate
   toolchains run in the per-check sandbox: read-write binds land before
   read-only ones (a nested toolchain mount inside the graded tree's `/work`
   was being shadowed); the host's real `/dev` device nodes are bind-mounted
   over the sandbox's placeholder files and landlock admits `/dev` (gate
   scripts write `>/dev/null`); probes declare their nested mount points and
   the harness prepares them before the tamper digest; probes take an explicit
   `interpreter` and `limits` (lychee's TLS allocation exceeds a 1 GiB address
   space — measured). **This commit changes the sandbox's shape — attack the
   new surface first.**
2. **`6be581d` — Phase 3: the project-owned oracle** (`project_oracle.py`,
   `tests/test_phase3_project_oracle.py`, `docs/research/phase3-project-oracle-design.md`,
   `docs/self-host.md`). `build_project_oracle` builds an acceptance manifest
   whose checks are the repository's own gates: command gates with the gate
   script's **content hash frozen** in the compare (a candidate rewriting its
   own gate to `exit 0` is graded FAIL unless the oracle is deliberately
   re-frozen), and the meta-registry floors frozen as gates (`min_covered`,
   `min_fixed`, `min_open`, `min_parameters`, the fidelity ratchet, and the
   structural rules) compared controller-side against freeze-time values. All
   checks are `SANDBOXED_DATA_DRIVER` probes — the N5 freeze enforcement holds
   with no unverified-probe allowance.

Three deviations are declared in the design brief and are part of the mandate
to adjudicate, not footnotes: `orchestrator.conformance` and the external-link
half of `scripts/check-docs` are **trusted-side** gates (nested sandbox
construction is impossible from a grading sandbox — measured `/proc/<pid>/setgroups`
EACCES; external links need the network the net namespace denies), and
`gate-check-tests` runs the suite's **deterministic subset** (17
runner-spawning modules plus two nested-environment-sensitive tests excluded
in the frozen, bound argv). The acceptance gap that leaves (behavioural
regressions in the excluded modules) is graded by the trusted gates only.

The phase's headline claim is the `unchanged` corpus case: an **internally
consistent** floor demotion (a covered finding demoted to pending with
`min_covered` lowered to match) passes the repository's own meta-gates and is
graded `CORRECTLY_UNRESOLVED` **only** because the frozen floor disagrees.
Attack that claim's non-vacuity directly.

A green suite blessed the previous patch semantics while they inverted a
differential; assume the happy path is covered and attack the seams.

**Independence:** you must not be the author of the two commits under review,
of `tests/test_phase3_project_oracle.py`, or of the
`phase3-project-oracle-design.md` brief. Read the normative documents and the
pre-images (`git show d4d6267^:…` for every changed function) before the
implementer's tests and docstrings; treat the new test and the brief as claims
to verify, not evidence.

## Target

- Repo: `/home/ruben/Projectos/meta-project/orchestrator`
- Revision reviewed: **`6be581d`** (tip; `d4d6267` is its parent). Both are
  local commits not yet pushed. Name the exact tree you reviewed
  (`git worktree add --detach /tmp/kilo/phase3-review 6be581d`).
- Normative documents: `docs/experiment.md` (independent oracle, calibration,
  freeze discipline), `docs/executors.md` (trust boundaries),
  `docs/budgets.md`, `docs/control-plane.md`, `docs/glossary.md`,
  `docs/research/phase3-project-oracle-design.md` (the claims being tested),
  and `docs/research/phase2-remediation-re-review.md` (the probe-oracle ruling
  — your comparison baseline for every containment question).
- Context: `docs/self-host.md` Phase 3–4; `docs/research/a2-oracle-consolidation-review-brief.md`
  (dependent adjudication; your containment matrix feeds it).
- Baseline (none of these is a verdict): `.venv/bin/python -m pytest -q`
  (expect **738 passed**); `scripts/check-tests --fast` (**712 selected, 26
  deselected** — the slow corpus among them); `scripts/check-docs` (**204**
  links OK); `tests/test_static_gates.py` 5 passed; gated lint clean.
- No source edits, no third-party network, no real credentials, no paid calls.
  Probes and reproducers under `/tmp/kilo/`. Note the grading sandbox's net
  namespace denies network — host-side probes only where the gate itself is
  host-side.

## The guarantees under review

Each is a claim made by the two commits. Falsify any of them.

1. **Frozen gate identity.** A candidate that edits, replaces, truncates,
    symlinks, or swaps a gate script is graded FAIL by the gate's check (the
    compare holds the freeze-time content hash); an unchanged gate's exit code
    is graded as before. No accepted path lets a modified gate produce PASS
    without a deliberate re-freeze (which changes the R12 binding and is a new
    experiment identity). The same holds for ruff/pytest configuration,
    conftests/startup hooks, and the frozen baseline test corpus: their path
    sets and content hashes are checked before the gate exit can bless a tree.
    Editing, adding, or deleting a test/config requires deliberate re-freeze.
2. **Frozen registry floors.** A candidate that lowers a floor, demotes a
   covered finding, deletes a mapped pin, removes a declared deviation,
   floods the registries with junk, or replaces a registry wholesale is
   graded FAIL by the registry checks even when every repository meta-test
   passes its edits — the thresholds and structural rules are compared
   controller-side against freeze-time values, and the `unchanged` corpus case
   fails **only** here (its own meta-gate green on the same tree).
3. **Binding by content, not label.** Any change to a floor, gate argv, gate
   hash, driver, or expectation changes `probe_identity` and therefore
   `oracle_target_binding`/`hidden_suite_manifest_hash` (mismatch rejection at
   `run_trial`/`calibrate_oracle`/`freeze_manifest`); two probes sharing a
   `__qualname__` or produced by one factory with different parameters have
   different identities; the binding is reproducible across processes.
4. **Probe-oracle ruling conformance.** Every check is the ruling's
   `SANDBOXED_DATA_DRIVER` shape with no `allow_unverified_probes` escape
   (the N5 freeze check holds); the expected values (floors, gate hashes) live
   only in the controller-side compares — a candidate process reading its own
   command line, driver source, environment, or tree cannot recover them; the
   per-check disposable copy, tamper digest, and harness deadline wrap every
   check (`oracle._run_probe`), and a hung, flooding, mutating, or
   `BaseException`-raising candidate is contained exactly as the Phase 2
   matrix recorded.
5. **The trusted-side split is honest and fail-closed.** `build_project_oracle`
   refuses to build unless `conformance.run_full_suite()` and
   `scripts/check-docs` pass host-side; the in-manifest `gate-check-tests` argv
   is frozen with its exclusions visible in the bound identity; a **new**
   runner-spawning test module makes the in-manifest gate red on a good tree
   (fail-closed drift, never silent acceptance); and the residual acceptance
   surface (the excluded modules' behavioural regressions) is exactly what the
   brief declares — no more.
6. **The five-case corpus is non-vacuous and correctly classified.**
   known-good → `CORRECT_ACCEPTANCE`, unchanged → `CORRECTLY_UNRESOLVED`,
   known-broken → `CORRECTLY_UNRESOLVED`, known-timeout → `FALSE_SUCCESS`
   (the timeout landing on `gate-check-tests`), infrastructure →
   `INDETERMINATE`; bidirectional coverage validates between the manifest's
   three criteria and its checks. Mutating the corpus fixtures (identical
   trees in both differential slots; a good tree that breaks exactly one gate)
   must move the classification — the gate cannot pass vacuously.
7. **The sandbox toolchain changes preserve containment (R01/R08/R15).**
   The `/dev` device binds and the landlock `/dev` grant open no write channel
   to the host (the sandbox's `/dev` is its own tmpfs plus exactly four
   device nodes; writes to `null` discard, not accumulate); read-only
   toolchain mounts remain unwritable from the workload; rw-before-ro ordering
   changes no mount's access mode; nested mount-point preparation does not
   escape the tamper digest's honesty (it happens before the before-digest and
   an empty mount point is not a semantic tree change — or argue otherwise);
   and a candidate cannot use `.venv/` shadowing (the bound mount covers the
   tree's directory) as a channel to hide content that the checks should see.
8. **Resource bounds are real.** The gate probes' `limits` (8 GiB address
   space, 600 CPU-seconds, 256 processes, 300s wall) actually confine a
   hostile candidate: a YAML alias bomb in a registry, a deeply nested test
   file for the AST passes, a fork bomb under the nested suite, or an
   output flood terminates as timeout/infrastructure/fail-closed — never an
   OOM of the controller, never an unbounded run.

## In scope

- `src/orchestrator/project_oracle.py` (`build_project_oracle`,
  `frozen_floors`, the driver templates, the compare closures, the frozen
  exclusion lists), `sandbox_launcher.py` (mount order, `/dev` devices,
  landlock), `hidden_oracle.py` (`run_sandboxed_probe`/`sandboxed_probe`:
  `ro_paths`, `interpreter`, `limits`, `_mount_points`), `oracle.py`
  (`_run_probe`'s mount-point preparation).
- `tests/test_phase3_project_oracle.py` (as claims to verify) and the corpus
  mutations' hardcoded anchors (R23's entry, R01's pin name).
- `docs/research/phase3-project-oracle-design.md` and the `docs/self-host.md`
  Phase 3 status wording (as claims to verify).

## Out of scope

- Phases 1–2 and the three patch-boundary review rounds (already closed,
  except where these commits touch them).
- A2 oracle consolidation (gated on its own R08 re-review — but your
  containment matrix and any ruling on the trusted-side split are inputs to it).
- Phase 4 (task decompositions) and real-model measurement (unclaimed).
- The `MODEL_CLIENT_FACTORY`/`SECRETS_SOURCE` per-run threading (known
  deferral).

## Known residuals and disclosed concerns — acknowledge, do not re-report as novel

- **Closed by the Phase 3 review remediation:** pytest's `-k` exclusions are
  substring-based, but the gate now hashes the frozen baseline test corpus
  and passes those exact paths explicitly to pytest (it does not discover or
  execute candidate-added tests); the controller compare requires the frozen
  path/hash map. A rename that injects an excluded substring changes the
  frozen file hash and fails before the exit code can bless the tree; pinned
  by `test_gate_config_and_baseline_test_corpus_neutralization_is_rejected`.
- **Closed by the Phase 3 review remediation:** open-defect node identities
  normalize pytest parameter suffixes (`test_fn[case]` binds to the registered
  function `test_fn`); pinned by
  `test_open_defect_identity_normalizes_parameterized_node_ids`.
- **Closed by the Phase 3 review remediation:** the public
  `build_project_oracle` API has no `run_trusted_gates=False` bypass; trusted
  conformance and documentation preconditions always execute before a
  manifest is returned, pinned by
  `test_project_oracle_has_no_trusted_gate_bypass_flag`.
- **Closed by the Phase 3 review remediation:** frozen gate/config/test/
  registry inputs are verified against `TargetSnapshot.commit` during oracle
  construction; the corpus fixture now commits an exact snapshot of the tree
  under test, then grades copies of that snapshot.
- **Known, by design:** the corpus mutations are anchored to R23's entry text
  and R01's pin name (the test asserts if they move). Brittle, disclosed.
- **Known trade:** the gate probes run with generous resource bounds (8 GiB
  address space for lychee's TLS). The bounds are part of the bound check
  construction; the trade is confinement headroom for real toolchains.
- **Known, from the Phase 2 re-review (unchanged):** the harness deadline
  binds the main thread (SIGALRM); a probe run off the main thread relies on
  its own budget and the sandbox wall. No shipped path grades off the main
  thread.

## Attacks to attempt (minimum)

- **Gate tampering census (guarantees 1, 3).** For each gate script: rewrite
  to `exit 0`; wrap the original and exit 0; truncate; flip one byte; replace
  with a directory; replace with a symlink to a passing script (identical
  content vs different content — the hash reads through links); a FIFO that
  hangs the hash read; a symlink to `/dev/zero`. Record accept/reject and the
  binding delta. The only acceptable PASS is a byte-identical gate.
- **Registry tampering census (guarantee 2).** Lower each floor one notch with
  the registry kept consistent; demote a finding with `min_covered` following;
  delete a mapped test; delete a pin's `tests:` block; add junk pins and junk
  `test_bug_` definitions (the unregistered flood); a parameterized pin (the
  disclosed concern); replace a registry with a symlink to `/dev/null`, to an
  alias bomb, to `/proc/self/environ`, and to a UTF-8-invalid blob. For the
  consistent-demotion rows, additionally run the repository's own meta-tests
  over the same tree and show them **green** — the frozen floor must be the
  only thing that objects (guarantee 2's headline).
- **The `-k` rename dodge (closed, mutation pin required).** Rename a genuinely
  failing test so its node id contains
  `test_frozen_corpus_branch_coverage_meets_the_floor` (or the dripping-reply
  name), leave it failing, and observe whether `gate-check-tests` goes green.
  Then search the frozen argv, drivers, and compares for sibling
  fuzzy-match surfaces.
- **Binding and identity (guarantee 3).** Hand-build a manifest whose floors
  or gate hashes are stale; swap a probe for a factory-produced same-qualname
  probe; re-parameterise a compare's closure; freeze, then recompute
  `oracle_target_binding`/`hidden_suite_manifest_hash`/`content_hash` in a
  fresh process. Every drift must change the binding or be refused at freeze.
- **Containment matrix, re-run for this shape (guarantees 4, 7).** The Phase 2
  rows plus: a gate that writes `/dev/null` at volume (discarded, no growth,
  no host channel); a workload that writes through `/dev` to reach the host; a
  write to `/usr/bin/x` (ro mount); a candidate that plants content under
  `.venv/` in its tree (shadowed by the bound mount — classify the channel); a
  driver that reads `/proc/self/environ` or `/proc/self/cmdline` and returns
  them in its payload (the compare must not leak expectations into that view).
- **Corpus non-vacuity (guarantee 6).** Swap the two differential slots'
  trees (both good, both broken); break exactly one gate in the good tree and
  confirm the classification moves to the failure family; make the timeout
  tree hang a *different* check (a registry driver, `gate-check-static`) and
  confirm `expect_candidate_timeout="gate-check-tests"` is then unsatisfiable
  — i.e. the corpus is specific, not forgiving.
- **Resource bounds (guarantee 8).** The alias bomb, a 50k-line test file for
  the AST passes, `while True: os.fork()` in a driver, a 100 MiB stdout
  flood from a gate — each must terminate as timeout/infrastructure/fail-closed
  with the controller healthy.

## Method

- Read the normative documents and the pre-images (`git show d4d6267^:…`) of
  every changed function first; list the semantic deltas of the sandbox
  changes before testing any of them.
- Drive the real entry points (`build_project_oracle`, `frozen_floors`,
  `calibrate_oracle`/`freeze_manifest`, the corpus's shape) with tree copies
  under `/tmp/kilo/`; prefer whole-grading reproducers (a case tree plus the
  manifest) over unit pokes, and direct boundary calls for the launcher
  questions. The container's `git apply` remains the oracle for any diff
  fidelity question.
- Use the graded outcomes (`oracle.grade_manifest` over case trees) and the
  recomputed bindings as the source of truth; a claim about the meta-gate's
  silence must be demonstrated by running the meta-tests over the same mutated
  tree.
- For each attack: exact commands, the probe (not a source edit), observed vs
  expected, and the guarantee violated. Prefer a failing reproducer. Do not
  weaken or edit tests.

## Deliverable

1. **Verdict:** GO / NO-GO for Phase 4 to build on this base.
2. **A ruling on the trusted-side split and freeze/re-freeze policy**: the
   acceptance surface's true boundary (what in-manifest grading does and does
   not cover), whether test/config edits being re-freeze-required and new test
   modules being deferred until re-freeze are the right R12 posture for
   self-development, and the mandatory constraints if Phase 4 or a production
   run-entry adopts this oracle — written so the A2 re-review can adopt it
   alongside the Phase 2 probe ruling. The `-k` rename class is pinned closed
   by the frozen corpus identity.
3. **Findings** with severity (P0/P1/P2), the guarantee violated, a minimal
   reproducer, observed vs expected, and the proposed permanent regression test.
4. **Claims verified:** each of the eight guarantees with its falsification
   attempt and outcome ("attacked, not falsified" is valid with the method stated).
5. **Gate/registry tampering census:** the accept/reject/binding-delta table
   across both censuses, with every fail-open row called out (any row where a
   modified gate or lowered floor produces PASS is P0-class).
6. **Residual unknowns:** what you could not test and why (network-dependent
   rows, other hosts' nesting behaviour, real-model behaviour).

## Exit criteria

- Every guarantee is either falsified with a reproducer or explicitly
  "attacked, not falsified" with the method.
- Both censuses exist and their fail-open rows are stated (none is acceptable
  silently).
- The `unchanged`-case claim is demonstrated end to end: the mutated tree,
  the meta-gates' silence on it, and the frozen floor's objection to it.
- The `-k` rename-dodge cell is resolved and sibling surfaces are searched.
- The containment matrix for this shape is complete (the Phase 2 rows plus
  the `/dev`, ro-mount, and `.venv`-shadow rows), each cell
  prevent/detect/tolerate/unaddressed.
- No source edits, no real credentials, no paid calls; host state left clean.
- Findings are reproducible from the report alone against the named revision.
- The review states what it did not cover.

## Reproduce everything

List every probe command, tree-mutation script, census corpus, and worktree
revision here so the verdict can be re-run against a corrective revision.
