<!-- informative -->

# Milestone Eight Readiness Review

Reviewed revision: `c26a0ec6c894ba42c1ac68df6491e6f44e6e77fb`.

## Findings and Decision

**NO-GO for paid pilots or measured runs on the reviewed implementation. GO for offline remediation and provider development.**

The project has substantial working infrastructure and a green regression suite. However, independently reproduced
counterexamples invalidate several guarantees needed before real credentials, billable requests, or generated code
are introduced. The highest-severity issue is candidate-controlled Git metadata crossing into trusted, unsandboxed
Git execution. Spending controls, credential redaction, acceptance guards, and the experiment's oracle and audit
records also have concrete failures.

This is not a recommendation for another architectural redesign. Fix the boundaries below and replace the relevant
success-only or structural demonstrations with negative, race, crash, and adversarial regression tests.

Priorities in this report:

- **P0:** stop untrusted candidate execution/import until fixed.
- **P1:** required before relying on the affected paid-run or measured-comparison path.
- **P2:** required before enabling the affected capability or reporting its results; some can be explicitly excluded
  from a narrowly scoped first smoke call.

Gates are distinct: **SPEND** covers any paid request, **EXECUTION** covers untrusted candidate execution/import, and
**MEASUREMENT** covers pilots used for calibration and measured comparison. Disabling a capability is an acceptable
temporary gate only when the harness rejects its use, not when the limitation exists solely in prose.

| ID | Priority | Gate | Item |
| --- | --- | --- | --- |
| R01 | P0 | EXECUTION | Candidate Git metadata reaches trusted Git commands |
| R02 | P1 | SPEND | Monetary accounting clips overruns and confuses dimensions |
| R03 | P1 | SPEND | Frozen call, token, and aggregate limits are not enforced |
| R04 | P1 | SPEND, EXECUTION | Cancellation does not reliably stop work or retain liability |
| R05 | P1 | SPEND | Credentials escape redaction and restart coverage |
| R06 | P1 | SPEND | The actual task prompt never reaches the model client |
| R07 | P1 | EXECUTION | Candidate scope and hashes do not describe the committed tree |
| R08 | P1 | MEASUREMENT | Candidate early exit earns full hidden-oracle credit |
| R09 | P1 | MEASUREMENT | Acceptance and approval paths do not enforce their evidence guards |
| R10 | P1 | MEASUREMENT | Validation repair is broken and nonterminal runs are graded |
| R11 | P1 | SPEND | Repeated trial identity deletes previous audit and cost records |
| R12 | P1 | SPEND, MEASUREMENT | Frozen manifest labels do not bind actual execution |
| R13 | P1 | MEASUREMENT | Actual oracle grades and rerun results are not persisted |
| R14 | P1 | SPEND, MEASUREMENT | Recovery omits terminal accounting and interrupted continuations |
| R15 | P1 | EXECUTION | Sandbox resource setup and cleanup can fail open |
| R16 | P2 | Concurrent admission | Proposal revalidation is outside serialized admission |
| R17 | P2 | Model-output use | Output freshness, model identity, and scalar paths are mishandled |
| R18 | P2 | Recursive comparison | Concurrent scheduling bypasses recursive selection |
| R19 | P2 | Model-driven graph input | Optional children can claim mandatory coverage |
| R20 | P2 | MEASUREMENT | Cost, speedup, and utilization metrics are misleading |
| R21 | P2 | Default executor use | Cleanup deletes the only stored candidate commit |
| R22 | P2 | Timer integration | Obsolete acknowledgement timers can terminate running attempts |
| R23 | P2 | Alternate oracle use | Unsupported non-test oracles can be ignored during classification |

## Blocking Findings

### R01: Candidate Git Metadata Crosses the Trust Boundary

Sources: `src/orchestrator/worktree.py:98-113,231-264,312-324`.

`snapshot()` hides `.git` directories, but `_copy_tree()` copies their contents into the trusted integration repository.
The controller then executes `git add` and `git commit` outside the sandbox using that repository's metadata.

**Reproduced:** a candidate containing only `.git/config` and an inert, non-executable `.git/hooks/pre-commit` produced
`changed_paths=[]` and `import_verified=true`; both files appeared in trusted Git metadata. No malicious hook was
executed during review. The same copy path preserves executable modes, exposing trusted Git commands to a
candidate-controlled hook or configuration.

**Required:** reject candidate Git metadata before import; keep trusted metadata outside the copy destination; disable
hooks and sanitize Git configuration/environment. Prefer importing an explicitly enumerated, validated blob manifest
through Git plumbing rather than allowing candidate files to influence ordinary Git staging/filter behavior.

**Regression:** hostile metadata must be rejected without invoking a candidate-controlled hook, filter, or configuration.

### R02: Spending Limits Bound the Ledger, Not Necessarily the Bill

Sources: `src/orchestrator/model_adapter.py:424-495`, `src/orchestrator/controller.py:486-489,615-618`,
`src/orchestrator/model.py:261-285,342-381`, `src/orchestrator/schemas.py:95-107`.

The adapter trusts `client.max_cost()`, calls the client, then records `min(actual, commitment)`. It also discards the
commitment dimension when constructing `hard_max_charge`. Token usage is telemetry, not separately enforced spending.

**Reproduced:** two calls declaring a maximum of 5 and reporting actual cost 9 consumed 18 backend-reported units while
the ledger recorded 10 under a budget of 10. A five-token commitment authorized a completion costing 4 USD despite a
monetary budget of 1; the ledger charged tokens rather than money. A one-token run budget admitted 4,000 reported tokens.

Non-finite numbers also defeat the comparisons. Infinite run limits and a NaN maximum estimate were accepted. In a
separate ledger probe, admitting NaN actual cost allowed a subsequent commitment of 1,000 against an initial budget of 10.

**Required:** construct enforceable, priced provider request bounds before I/O; preserve reported usage without clipping;
stop further calls and retain explicit overrun liability on discrepancy. Preserve dimensions and units. Reject
non-monetary commitments at this adapter until simultaneous token/money reservations are supported. Reject non-finite,
negative, boolean, and malformed numeric values at configuration, request, response, and ledger boundaries.

**Regression:** exact backend token ceilings and pricing bounds must be asserted; overruns remain visible and prevent
the next call; dimension mismatches and invalid preflight numbers produce zero client invocations. Update
`tests/test_milestone4_conformance.py:228-247`, which currently endorses clipped accounting.

### R03: Frozen Experiment Limits Are Mostly Descriptive

Sources: `src/orchestrator/experiment.py:359-409,1683-1699,1928-1940,2057-2083`,
`src/orchestrator/model_adapter.py:418-433,478-479`.

The experiment forwards only monetary capacity into run configuration. Call, input/output token, action, node, depth,
and fan-out limits are not all enforced from the frozen manifest. Wall time is applied to individual execution phases,
not as one aggregate trial deadline. Child allocations are hard-coded rather than derived from the selected budget.

**Reproduced:** limits of 1 for calls, input tokens, output tokens, actions, nodes, depth, and children still yielded
4 calls, 400 input tokens, 80 output tokens, 4 actions, 6 nodes, depth 2, and `CORRECT_ACCEPTANCE`. Lowering root cost
capacity to 10 let DIRECT run but made fixed decomposition abort because its child allocations total 90.

Concurrent duplicate delivery also invokes the model client twice: response caching is not an atomic in-flight claim.
One commitment and one cached result can therefore correspond to two requests.

**Required:** authoritative pre-dispatch counters/reservations for every enabled limit, a trial-wide deadline, compatible
child allocations, and single-flight protection for each attempt/token. Add a durable owner-authorized experiment-wide
spend ceiling that includes pilots, retries, and uncertain in-flight charges, not just separate per-trial budgets.

**Regression:** exhaust each limit independently and assert refusal before the next billable request. Two simultaneous
deliveries must invoke the client once. Restart must not reset already consumed experiment capacity.

### R04: Cancellation Neither Stops All Work Nor Preserves Billable Liability

Sources: `src/orchestrator/controller.py:182-219,1714-1718`, `src/orchestrator/reducer.py:588-604,673-718`,
`src/orchestrator/command_executor.py:105-106,139-162`.

`_cancel_attempt()` reconciles actual cost as zero and reclaims the full commitment even after dispatch. Root cancellation
only visits root attempts, not child attempts. New attempts can still be planned and executed after the run is cancelled.
Command cancellation only sets a flag; it does not interrupt the blocking runner or recheck after the final command.

**Reproduced:** a dispatched response reporting actual cost 3 was cancelled and finally settled at zero with all 5
committed units reclaimed. A cancelled root retained a reserved child attempt. A new attempt executed and charged money
after the run outcome became `CANCELLED`. Cancelling during the final command still returned a completed candidate.

A direct late `RESULT_ADMITTED` against a cancelled attempt also creates a `CURRENT` freshness record because result
side effects run even when the attempt transition was retained as a no-op.

**Required:** reject new work after termination, cancel the entire run's active work, propagate cancellation to process
supervisors, and distinguish never-dispatched reclamation from uncertain dispatched liability. Settle authoritative
actuals when available; otherwise retain or conservatively settle liability. Late execution results must not create
current results or artefacts, while eligible cost-only observations remain processable.

### R05: Known Credentials Reach Durable Records

Sources: `src/orchestrator/secrets.py:53-61,99-101,131-137,221-227`,
`src/orchestrator/controller.py:110-113,738-754`.

Redaction and secret detection inspect mapping values but not keys. Response validation precedes redaction, and its
untrusted diagnostic text can be persisted verbatim. Restart creates an empty secrets source unless explicitly restored,
even when replay contains active credential audit records.

**Reproduced using synthetic secrets only:** a credential used as a nested JSON key persisted in observation, result,
and model-output events. A malformed response identity containing the credential persisted through a protocol-error
message. After reopening the store with a fresh controller, a response echoing an earlier credential was retained raw.

**Required:** cover mapping keys, values, and every persisted diagnostic; reject redaction-induced key collisions.
Require restored external redaction coverage before resuming affected boundaries, or fail closed. Do not persist
plaintext credentials to solve recovery. Complete actual retirement, expiry, and provider revocation wiring before
using the live credential source; the existing in-memory lifecycle is not proof of those controls.

### R06: The Model Client Receives No Task Prompt

Sources: `src/orchestrator/controller.py:534-542,1103-1110`, `src/orchestrator/model_adapter.py:483-495`.

`run_model_review()` constructs `_prompt` and discards it. The client receives identifiers, references, and a hash, not
the task or visible context text. Preflight pricing receives `sha256:unset` rather than the completion's prompt binding.

**Reproduced:** distinctive task/context strings were absent from the captured client request. This is an interface
defect, not simply the expected absence of a live provider implementation.

**Required:** build one immutable, filtered request containing the actual prompt, context bindings, model configuration,
and enforceable token bounds; use the same request for estimation and completion. Test that visible content arrives,
hidden content does not, and the hash describes exactly what was sent.

### R07: Verified Candidate Hashes and Scopes Do Not Bind the Git Commit

Sources: `src/orchestrator/worktree.py:107-113,198-201,231-264`.

Import verification hashes the working directory, not the referenced Git tree. `git add -A` can omit ignored files that
the digest includes. Filtering hidden paths at copy-in and then replacing the whole base tree at copy-out silently
deletes protected content. Directory symlinks are omitted from both snapshot and copy-out.

**Reproduced:** an ignored candidate file was reported changed and verified, but absent from the candidate commit.
Without any candidate edits, a tracked hidden-test file and a tracked directory symlink were deleted in separate
fixtures; both returned `changed_paths=[]` and `import_verified=true`.

**Required:** preserve excluded base content, inventory symlinks correctly, and apply allowed changes rather than
replace an incomplete view of the whole repository. Verify the committed tree and enforce scope against its complete
base-to-candidate diff. Define hashes over Git-representable content, not transient filesystem metadata.

### R08: The Hidden Oracle Can Be Bypassed Without Escaping the Sandbox

Sources: `src/orchestrator/experiment.py:525-563,599,619`, `src/orchestrator/acceptance_checks.py:57-64`.

The hidden assertions run in the same Python interpreter that imports candidate modules. Exit zero is treated as proof
that all assertions ran. Candidate code can terminate that interpreter before assertions execute; the hidden script is
also available through its command-line arguments.

**Reproduced:** replacing every solution module with `import os; os._exit(0)` produced four hidden `PASS` results,
`verified_progress=1.0`, and `CORRECT_ACCEPTANCE`, without implementing the required functions.

**Required:** isolate the evaluator's assertions and verdict production from candidate execution. For this target,
send inputs to a candidate subprocess and validate its bounded responses in a trusted harness; missing/malformed
responses fail even if the process exits zero. A success marker writable by candidate code is not sufficient.
Calibrate early exit, evaluator interference, ordinary wrong results, timeout, and infrastructure failure.

### R09: Acceptance and Human Approval Are Not Enforced at the Authority Boundary

Sources: `src/orchestrator/controller.py:945-1013,1160-1195,1292-1337,1381-1448`,
`src/orchestrator/reducer.py:558-586`.

Acceptance checks neither require nonempty check sets nor verify evidence ownership, result existence, and exact
bindings. Older public helpers bypass the newer evidence path. Deferred acceptance resumes without verifying that a
matching approval was granted; its evidence is stored only in a process-local dictionary.

**Reproduced:** acceptance with no attempt and an empty evidence mapping; acceptance of a nonexistent result from an
empty visible bundle; acceptance using another child's empty bundle; and deferred acceptance while its approval
remained `PENDING`.

**Required:** enforce one acceptance invariant in authoritative admission/reduction, shared by all controller helpers.
Require independently obtained nonempty evidence for the exact node/result/versions, mandatory child obligations,
blocking-conflict clearance, and a matching current grant when policy requires it. Persist deferred bindings and reject
missing recovery state rather than default to empty evidence. Direct event admission must not bypass these guards.

### R10: Repair and Terminal Failure Behavior Do Not Match the Experiment Contract

Sources: `src/orchestrator/controller.py:970-991`, `src/orchestrator/retry_policy.py:46-66`,
`src/orchestrator/experiment.py:1719-1727,1887-1888,1626-1628`.

`authorise_acceptance()` starts validation before calculating retry eligibility. The eligibility function requires
`READY`, so the resulting `VALIDATING` state makes the first failure non-retryable. Validation event idempotency keys
are per node rather than validation cycle, preventing a subsequent valid cycle on the same node.

The experiment stops its driver on failure and grades immediately instead of following repair/replan to termination.
**Reproduced:** a failed DIRECT trial returned `CORRECTLY_UNRESOLVED` and `honest_blocked=True` with controller outcome
`None` and root state `REPLAN_REQUIRED`. A separate controller probe made a retry-eligible first failure non-retryable;
after a legal replan/readiness sequence, successful validation still failed because the start event was deduplicated.

**Required:** calculate retry policy from the appropriate action lineage, use cycle-specific event identities, and
drive bounded repair/replan to a recorded terminal outcome before grading. An explicit frozen stopping policy is
acceptable; an exception leaving a live run is not an honest terminal outcome or an iterative DIRECT baseline.

### R11: Trial Restart Deletes Previous Evidence and Spending History

Source: `src/orchestrator/experiment.py:1664-1673,2185,2231-2234`.

Run identity and directory are derived only from arm and seed. `run_trial()` recursively deletes an existing directory
before creating its database. Pilots reuse a fixed seed.

**Reproduced:** after completing a trial, invoking the same arm/seed and stopping before execution left the replacement
database with no run IDs; the first trial's history was already deleted.

**Required:** allocate persistent experiment/phase/trial identities and refuse accidental overwrite. Resume/recovery
must inspect existing records. Preserve completed and interrupted paid trials, including failures, and account for all
of them in the experiment-wide cap. Test duplicate invocation and startup failure before enabling credentials.

### R12: The Manifest Does Not Bind What Actually Runs

Sources: `src/orchestrator/experiment.py:2360-2388`, `src/orchestrator/controller.py:232-267`,
`src/orchestrator/model_adapter.py:433-455`.

Measured comparison uses global `MODEL_PROFILE` and an independently supplied oracle instead of verifying the frozen
model/sampling/oracle configuration. The hash check compares an input label with the same echoed label. Frozen
dataclasses still contain mutable nested dictionaries. The controller validates kernel/manifest/policy fields but does
not persist the supplied kernel/manifest references and emits policy version 1 regardless of the submitted version.
The adapter also replaces the completion's model identity with its configured identity.

**Reproduced:** a manifest declaring another model and `oracle-v1` ran with `experiment-model` and `different-oracle`
while all reported manifest hashes matched. Nested sampling mutation changed the frozen hash. A submitted policy
version of 999 was accepted and all emitted events recorded 1; unique submitted kernel/manifest values were absent
from history.

**Required:** persist immutable serialized run/experiment specifications before dispatch; derive actual execution
configuration from them and reject component, oracle, model, version, and environment mismatches. Record any permitted
model alias resolution explicitly. Replay and reopened observability must recover the exact executed binding.

### R13: Experiment Oracle Outcomes Are Not Durable

Sources: `src/orchestrator/experiment.py:1575,1724-1739`, `src/orchestrator/observability.py:216-236`.

The experiment computes grades, returns metrics, and closes the store without persisting the grading record. The
observability backend reads hidden evidence that this path never creates. Initial outcomes are returned in one metric
while classification uses effective rerun outcomes.

**Reproduced:** reopening a `CORRECT_ACCEPTANCE` trial yielded no oracle grades and no hidden evidence.

**Required:** persist candidate identity, oracle/manifest binding, initial outcomes, permitted reruns, effective
outcomes, and classification before advancing. Preserve infrastructure and timeout categories rather than collapse
them into booleans. Grading must remain separate from authoritative acceptance mutation. Reconstruct the exact grade
and classification from a reopened real experiment store in a regression test.

### R14: Replay Does Not Recover Interrupted Controller Continuations

Sources: `src/orchestrator/controller.py:145-170,366-402,630-654,1722-1748`,
`src/orchestrator/store.py:163-178,246-251`.

Derived cancellation and rejection effects execute after the causing event commits. Replay discards effects, while
recovery only handles three nonterminal execution states. Decomposition is likewise a sequence of separately committed
events without a durable continuation. Terminal attempts with unsettled accounting are not recovered.

**Reproduced:** a crash after the terminal budget-exhaustion event left a reserved attempt and open commitment after
recovery; a crash after decomposition admission left a blocked parent and child budget but no child node; a timed-out
attempt remained provisionally settled with its commitment open after reopening and recovery.

**Required:** atomically persist essential related state or durable continuation/intents, and reconcile unfinished
controller-owned consequences idempotently at recovery. Include terminal-but-unsettled attempts. Normal replay must
still never redispatch paid work. Inject crashes between every event in these workflows, not only around the outer
executor call.

### R15: Resource Setup and Process Cleanup Are Not Reliably Fail-Closed

Sources: `src/orchestrator/sandbox.py:463-488,565-605,632-634`,
`src/orchestrator/sandbox_launcher.py:162,406-413`, `src/orchestrator/command_executor.py:129-143`.

Cgroup controller activation, limit writes, and process placement swallow failures. Memory and PID limits are attempted,
but aggregate CPU consumption is only read afterward. The runner lacks unconditional cleanup: setup failure does not
reap/remove everything, and interruption during wait bypasses termination. The launcher lacks a controller-parent-death
guarantee; the controller enforces the wall deadline. Each command also receives the whole attempt timeout afresh.

**Reproduced with mocked processes and ordinary temporary files:** denied cgroup writes did not fail; limit files
contained no CPU enforcement; setup error caused no wait or cgroup removal; injected interruption caused no kill.
Cancellation during the final command still imported output, as noted in R04.

**Required:** verify required limits and membership before releasing work; fail closed or explicitly reject unsupported
resource profiles. Enforce aggregate CPU budgets when promised, one monotonic attempt deadline, unconditional
terminate/reap/close/remove cleanup, and controller-death handling or an independent supervisor deadline. Verify process
termination before removing its filesystem. The documented per-process fallback is not an aggregate resource guarantee.

## Capability-Specific Findings

### R16: Proposal Revalidation Races with Admission

Sources: `src/orchestrator/controller.py:280-291`, `src/orchestrator/store.py:182-203`.

The store serializes reduction/persistence, but proposal versions and expected absences are checked outside that lock.
A two-thread barrier probe made two proposals expecting the same relation to be absent both pass revalidation and be
persisted; neither produced a version conflict. The second event was silently retained after the first created the relation.

Move revalidation into the same serialized admission operation as reduction and commit. Test competing proposals and
unrelated changes separately. Resolve before concurrent model-driven proposals are enabled.

### R17: Model Output Admission Manufactures Freshness and Rejects Valid Paths

Sources: `src/orchestrator/controller.py:884-906`, `src/orchestrator/model_adapter.py:197-233,433-455`.

Output input versions/hashes are reconstructed from current state, not the dispatched context. Unknown references are
omitted from validation. A stale-context output referring to a real artefact plus a nonexistent one was admitted and
stamped with current bindings. Completion model identity is also ignored, as covered by R12.

Separately, the common path loop iterates scalar `ARTEFACT.path` character by character. A valid `src/app.py` output under
`src/**` was rejected because `s` was out of scope, after the call had already completed.

Freeze complete input bindings at dispatch, reject unresolved references, and validate scalar versus list fields
explicitly. Add stale/missing/changed input tests and an in-scope scalar-path positive test before consuming those output
classes in a paid workflow.

### R18: Concurrent Recursive Execution Changes the Decomposition Policy

Source: `src/orchestrator/experiment.py:1840-1856`.

Multiple disjoint ready children go directly to a leaf batch, bypassing recursive action selection. A fixture with two
ready composite children whose selectors both returned decomposition produced one leaf batch and zero recursive
selections. The current target masks this because concurrently ready children are leaves.

Select each node's action before batching; batch only executable leaves. Require the recursive sequential and
concurrent arms to admit equivalent intended graphs on a two-composite-child fixture. Also disclose that current model
reviews are sequential even inside the concurrent arm (`experiment.py:1904-1910`).

### R19: Decomposition Coverage Does Not Require a Mandatory Child

Source: `src/orchestrator/decomposition.py:110-132`.

Coverage trusts `mandatory_cover` without checking the referenced child's `mandatory` flag. A proposal with only an
optional child covering mandatory criteria was admitted; parent acceptance then ignored that child. The missing-set
calculation also uses all parent criterion IDs, incorrectly requiring optional criteria to appear in mandatory coverage.

Validate actual child mandatory status and calculate coverage only over mandatory parent criteria. Test mixed optional
and mandatory criteria and a falsely labeled optional covering child. Fix before accepting model-generated graph input;
the current fixed arm templates do not exercise this defect.

### R20: Comparative Metrics Need Correct Denominators and Measurements

Sources: `src/orchestrator/experiment.py:1601-1628,2391-2408,2463`, `src/orchestrator/decisions.py:113-116`.

- Failed trials contribute zero to averaged cost-per-accepted-root. Two trials costing 10 each with one acceptance report
  5, rather than total spend per accepted root of 20.
- Sequential success time is used as the reference for concurrent failures. A synthetic 100-second success and 1-second
  failure produced 100x speedup.
- Worker utilization is peak batch size divided by worker count, not occupied worker-time. Successful DIRECT execution
  reported zero utilization.
- No recorded concurrent batch is interpreted as proof that no actions were concurrently ready, even when the driver
  deliberately processes ready actions sequentially.

Use explicit denominator definitions, comparable outcome/seed cohorts, and actual worker-time/readiness measurements.
Otherwise list these metrics as unmeasured and do not use them to retain or reopen architectural decisions.

### R21: Default Executor Cleanup Deletes Its Candidate Objects

Sources: `src/orchestrator/command_executor.py:217-218`, `src/orchestrator/worktree.py:283-292`.

After dispatch, the default executor removes the integration repository containing the new commit. Artefact storage
retains only JSON; the source repository does not contain the commit. This was reproduced with an inert command fixture.

Persist the verified object graph or bundle in controller-owned storage before cleanup. The experiment's persistent
worktree subclass avoids this specific deletion; this is a gate for the default command-executor path, not a claim that
the existing experiment loses every candidate immediately.

### R22: Acknowledgement Timers Remain Applicable After Acknowledgement

Sources: `src/orchestrator/controller.py:587-597,690-697`, `src/orchestrator/reducer.py:658-698`.

A recorded acknowledgement timer fired against an acknowledged `RUNNING` attempt and changed it to `TIMED_OUT` in a
direct event probe. Timer records lack purpose/liveness sufficient to distinguish stale acknowledgement deadlines from
execution deadlines.

Disarm acknowledgement timers on acknowledgement and reject/retain obsolete deliveries without execution transitions.
Use absolute recorded deadlines and a clearly defined timer service; the current stored acknowledgement deadline is
the configured timeout duration rather than a dispatch-relative absolute timestamp.

### R23: Non-Test Oracle Declarations Can Escape Correctness Classification

Source: `src/orchestrator/oracle.py:139-151,291-303`.

A manifest with one tested mandatory criterion and another declaring an independent non-test oracle validated and
returned `CORRECT_ACCEPTANCE`, while criterion progress was 0.5. No independent non-test result was evaluated.

Reject this unsupported manifest option until those results participate in completeness, failure, and indeterminacy.
The existing four-check target does not use the option, so a fail-closed restriction is sufficient for milestone 8.

## Milestone Eight Scope Must Stay Honest

The following work is explicitly unfinished in the roadmap, rather than a regression in an implemented live provider:

- Owner authorization of model access and spend, recorded before pilots as well as measured trials.
- A real provider implementation, frozen identity/sampling/pricing and transport behavior, bounded retries/timeouts,
  and live credential/endpoint wiring.
- Reassessment of the recorded design decisions using actual model observations.

There is also an important experimental choice to make before paying. The current harness obtains review outputs, but
candidate content comes from `_SOLUTION_FILES` through `_commands_for` and `_write_script`
(`src/orchestrator/experiment.py:1935-1940,2086-2145`). Returning a `FAIL` model verdict still produced the same successful
candidate in a local probe. Script seeds select injected scenarios; they are not currently provider sampling seeds.

Simply replacing `ExperimentModelClient` with a paid client would buy reviews of scripted work. It would not measure
model-generated implementation quality or model-driven decomposition. The roadmap explicitly excludes recursive
planning on model outputs; this review does not require adding it.

Before the first paid pilot, choose and freeze one honest scope:

1. A real review-only study over fixed candidates, with useful review accuracy/schema/usage outcomes and explicit limits
   on any claim about orchestration effectiveness.
2. A bounded model-produced implementation or repair path inside the already fixed arm structures, with deterministic
   proposal/artefact admission and independent validation. No recursive model planner is required for this option.

Either way, validate offline that changing the model response changes the outcome that the experiment claims to study.
The first option must not relabel scripted candidate success as evidence of model coding capability.

## Remediation Order and Exit Gates

### Before Any Paid Call

Resolve R02-R06 and R11-R12 for the selected path. Add durable experiment-wide authorization/cap accounting, single-flight
dispatch, real prompt delivery, redacted diagnostics/restart behavior, immutable request/model bindings, and preservation
of every trial. Exercise transport failure, unknown usage, cancellation, timeout, and restart without real credentials.
Address the relevant accounting recovery portions of R14. Reject unsupported output classes until their R17 issues are fixed.

Then authorize only a tiny, explicitly capped transport smoke test, with no untrusted repository execution. Account for
it as real spending, not as a free pilot. A safe smoke test is not permission for the full comparison.

### Before Untrusted Candidate Execution

Resolve R01, R04, R07, and R15, plus R21 if using the default executor. Run isolated hostile-import, symlink, scope,
process-tree cancellation, setup-failure, controller-death, and resource-limit tests on the actual execution host.
Do not rely on a green capability probe as proof that copy-out and cleanup are safe.

### Before Paid Pilots and Measured Comparison

Resolve R08-R10 and R13-R14, and all enabled capability-specific findings. Preserve frozen configuration and completed
grades across restarts. Require a terminal controller outcome and correct accounting for every graded trial. Prove
manifest cap enforcement with deliberately tiny values. Correct or disclose R20 metrics before interpreting them.

Run the experiment entry gate before any pilot through the only supported paid entry point. The low-level
`run_measured_comparison()` itself currently has no gate, and the milestone-7 test fixture runs conformance after its
pilot/comparison (`tests/test_milestone7_acceptance.py:112-126`); do not copy that ordering into the paid launcher.

These fixes should end in concrete regression tests and an explicit owner spend decision, not another general research
round. Continue to exclude distributed execution, routing, speculative work, and other unrelated expansion.

## Verification and Review Limits

The following baseline checks passed on the reviewed revision:

- `.venv/bin/python -m pytest -q`: **236 passed**, with no skipped tests reported.
- `scripts/check-docs`: generated tables, Markdown lint, and **160 links** passed before adding this report.
- `PYTHONPATH=src .venv/bin/python -m orchestrator.conformance`: all reported checks passed.
- `.venv/bin/python -m pip check`: no broken requirements.

After adding this report, `scripts/check-docs` passed again with **162 links**, and `git diff --check` passed.

Additional offline probes reproduced the defects described above:

- **17 controller probes:** acceptance/approval bypass, cancellation/liability, post-terminal work, late results, retry
  behavior, interrupted workflows, proposal races, NaN accounting, graph coverage, frozen inputs, and stale timers.
- **14 model-boundary probes:** pricing/dimensions, redaction, restart, prompts, provenance, duplicate delivery, and paths.
- Experiment probes exercised the real local sandbox and scripted clients, including early-exit oracle bypass, tiny
  ignored limits, model/oracle substitution, missing grade persistence, nonterminal failure, and destructive trial reuse.
- Git/sandbox probes used inert temporary repositories and mocked processes. No executable candidate hook, real secret,
  host breakout payload, or host cgroup modification was used.

The temporary reproductions are available on this machine:

```bash
PYTHONPATH=src:tests .venv/bin/python -m pytest -q /tmp/kilo/pre_token_controller_review.py
PYTHONPATH=src:tests .venv/bin/python -m pytest -q /tmp/kilo/review_model_c26a0ec.py
PYTHONPATH=src .venv/bin/python /tmp/kilo/review_m8_harness.py
PYTHONPATH=src .venv/bin/python /tmp/kilo/m8_sandbox_review.py
```

The probe assertions deliberately confirm current defects: their passing is not a safety verdict. They are temporary
review artifacts, not a replacement for permanent regression tests asserting the corrected behavior.

No paid/provider calls were made. Actual provider billing, revocation, transport retry semantics, and live-model quality
remain untested. Review findings are grounded in the current implementation and local counterexamples, not a formal
proof of all possible failures. Existing suite success establishes covered behavior, not every normative guard or
adversarial boundary; some tests currently bless behavior that must change.

Only this informative report was added to the repository. An unrelated untracked `home/` directory appeared during the
review and was left unchanged.
