<!-- informative -->

# Performance Plan

Status: proposal. This plan reached consensus in review but is not adopted, and no code has changed. It is the supporting
detail behind the performance workstream of the [self-hosting roadmap](../self-host.md): the roadmap carries the phase
placement and gates, this document carries the measurements, the work plan, the recipe, and the rationale.

The problem it addresses: runs are slow even with no real model calls, and the cost profile does not degrade gracefully,
so a real repository with many events would make the orchestrator unusable. The goal is to make product-run time scale
with the work the target actually does, not with the orchestrator's own bookkeeping.

## Two Budgets

Two separate budgets are tracked, and a win in one is not a win in the other:

- **Dev-loop time** — the test suite and the repeated experiment builds used to develop the project. Owned by the
  session-scoped fixture work (A3).
- **Product-run time** — the orchestrator's runtime on a real repository. Owned by the oracle, sandbox, and control-plane
  work (A1, A2, A4, B-lite).

The user-facing concern — "a real project will take forever" — is product-run time. Dev-loop wins are real and worth
having, but they do not count against it.

## Measured Baseline

Measured on the development host. Profiled figures inflate wall-clock by roughly 25%, so they are labelled. Absolute
numbers vary with hardware; the ratios are the signal.

| Cost | Measurement | Source |
|---|---|---|
| Scripted smoke run (4 trials + calibration + pilots) | 20.4 s real, 6.6 s user, 2.3 s sys | `scripts/smoke_run.py` |
| Sandboxed command execution | 17.7 s over 61 commands (~0.29 s each) | `sandbox.py` `SandboxRunner.run` |
| Trivial sandboxed command | ~50 ms setup + ~32 ms run (+~30 ms cgroup placement) | direct measurement |
| Hidden-oracle grading | 15.1 s over 35 checks (~0.43 s each); calibration 12.7 s | `experiment.py:709` `hidden_runner` |
| Reducer state clone | 3,708 clones, 3.04 M `deepcopy` calls, 5.9 s (profiled) | `reducer.py:382` → `model.py:475` |
| Full-log replay | 2.9 s over 77 calls (profiled) | `store.py` `replay_fresh` |
| Slow test suite | 124.5 s (113 s on review), dominated by three experiment builds | `pytest --durations` |

Two facts that shape the plan:

- The 14.7 s of `time.sleep` in the profile is **not** removable overhead. It comes from `subprocess._wait`
  (`subprocess.py`), which probes with `WNOHANG` and returns as soon as the child exits; the parent's sleep is the
  child's runtime observed by the waiting parent. The only recoverable part is exit-detection granularity
  (≤ 0.05 s per wait, ~3 s across 61 commands at worst).
- The dominant cost **now** is hidden-oracle grading; the dominant cost **at scale** is the reducer cloning the entire
  state on every event (`reducer.py:380-382`, `model.py:475`) plus full-log replay from genesis (`store.py`
  `replay_fresh`), which is `O(events × state)` and re-pays it on every replay.

## Phase A — Remove Overhead and Add the Projection

Placement: a tracked step **after the current capability work** (self-host Phase 1 / Axis 1, with the Axis 3 seed
matrix folded in) and **before target-generalization work** (self-host Phase 2). It is an insertion, not a reordering:
the existing order — phase-1.3 executor unification → Axis 1 → this block — is compatible, and the phase-1.3 executor
unification is the natural seam for per-attempt sandboxing (A4). Do it from a clean base (the working tree currently
carries a large uncommitted hardening diff; land it first). These are localized to the harness, sandbox, and oracle
internals; the trust boundaries are untouched.

### A3: Session-scoped test fixtures

- **What**: stop rebuilding the whole experiment per test module; use a session-scoped fixture or a precomputed
  manifest.
- **Why**: three module fixtures dominate the slow suite (axis2 stress 45.4 s, milestone6 calibration setup 43.7 s,
  milestone7 comparison setup 26.6 s).
- **Budget**: dev-loop only.
- **Risk**: none to the product.
- **Coupling**: moving fixtures changes the meta-registry scan scope. The fixture-fidelity and open-defect AST scans
  enumerate `tests/*.py` non-recursively (`tests/test_fixture_fidelity.py:43-44`,
  `tests/test_open_defect_registry.py:31-32`), so a fixture that migrates into a subpackage must move the scan scope
  with it or the "new fake ships without a fidelity row" guard goes blind. The counterexample registry is not an AST
  scan — it parses the readiness report's finding table and pytest collection — so it is unaffected by fixture moves.

### A1: Oracle grade caching

- **What**: cache decisive grades keyed by `(tree digest, oracle version, manifest hash)`, scoped per experiment run,
  and skip re-grading unchanged trees.
- **Constraint**: never cache `CANDIDATE_TIMEOUT` or `INFRASTRUCTURE_ERROR`; the rerun policy
  (`oracle.py` `rerun_non_gradeable`) exists precisely to re-execute those, and caching them silently defeats it.
- **Acceptance**: an old-vs-new grading equivalence test over the calibration corpus.

### A2: Oracle consolidation

- **What**: reduce per-check overhead — one sandbox per candidate and one read-only copy (or copy-on-write overlay)
  instead of `copytree` + digest + fresh sandbox per check.
- **Constraint**: this re-opens the R08 evaluator-isolation verdict. The current design (`experiment.py:709-759`) is a
  per-check disposable copy with a before/after tamper digest and expected values held controller-side; any consolidation
  must preserve equivalent isolation and tamper detection, or be shown to supersede it.
- **Acceptance**: the R08 adversarial re-review is an acceptance condition of this item, not an assumed guardrail.
- **Status: closed as no change** ([A2 review brief](a2-oracle-consolidation-review-brief.md)).
  Re-measured against the Phase 3 oracle, a grade is ~15.3 s and `gate-check-tests` is
  91 % of it; the `copytree` + digest half is still worth ~nothing, and sharing one sandbox
  across the checks is worth ~0.25 s (~1.6 %). The R08 GO is left untouched, and the residual
  cost is booked against the gate that runs the target's real test suite per candidate.

### A4: One sandbox per attempt

- **What**: run the attempt's whole command sequence in one sandbox instead of provisioning a fresh sandbox per command.
- **Permitted**: `executors.md` requires every executor-controlled subprocess to run inside an OS-enforced sandbox, not a
  fresh sandbox per command.
- **Constraint**: keep cancellation interruptible (the poll at `sandbox.py:668-670` exists so a cancellation can
  interrupt a blocking runner, per R04); do not replace it with a naive blocking wait.
- **Seam**: the phase-1.3 executor unification is where this belongs, since it unifies the executor base that currently
  carries the trusted-fixture deviations (no dispatch lock, no protocol validation).
- **Budget**: product-run; **measure, do not assume** the win, since the parent's sleep is the child's runtime.
- **Status: delivered** (`sandbox_launcher.py` command-sequence mode + `SandboxRunner.run_sequence`, consumed by
  `CommandExecutor._run_commands` for both the command and patch executors). The trusted in-sandbox sequencer runs each
  command as a child with full-stream output hashing and per-command wall budgets; the runner's deadline and
  cancellation poll govern the whole tree exactly as before (R04), and a deadline kill surfaces as a sequence timeout,
  never as fabricated evidence. Per-command evidence (exit codes, timeouts, hashes, `limits_verified`) is preserved.
- **Measured (2026-09, 2-command BUILD+TEST attempt)**: one sandbox per command **393 ms** → one sandbox per attempt
  **247 ms**; **146 ms saved per attempt (−37% of sandbox provisioning overhead)**. The end-to-end trial win is smaller
  (a ~3.2 s trial drops on the order of 0.1–0.5 s depending on attempt count) because grading, git, and bookkeeping
  dominate outside the sandbox — the review's expectation of a small end-to-end win holds, and the provisioning saving
  is real where attempts are many.

### B-lite: Cheap state clone

- **What**: replace `copy.deepcopy` in `RunState.clone()` with a cheap clone that preserves the pure-reducer contract
  (`reducer.py:380-381`, "Apply one admitted event to a cloned state. Never mutates its input.").
- **Gate**: the inventory-independent input-purity property (see below), including raising calls.
- **Review**: B-lite is the only Phase A item that touches control-plane state internals, so it lands as a separately
  reviewed change, not alongside the harness work.
- **Status: delivered as the separately reviewable slice** (`RunState.clone` in `model.py`, gate
  `tests/test_state_purity.py`). The recipe's inventory is implemented (fresh containers, copied `LedgerEntry`
  objects and `BudgetRecord.limits`, copied `failure_fingerprints`/`failure_strategies` inner lists, fresh
  `version_conflicts`); the gate is the inventory-independent property over a fuzzer-shaped corpus interleaved with
  synthetic failing events, asserting the input deep-equals a pre-call `deepcopy` of itself on success and on every
  raise.
- **Measured (2026-09, 147-event `RECURSIVE_LAZY` trial)**: clone **0.179 s → 0.034 s (5.3×)**; full-log replay
  **0.142 s → 0.025 s (5.7×)**. The state fuzzer (9 walks) runs green on the cheap clone.

### Instrumentation and the full-B trigger

- **What**: project events × state-size and clone+replay time per run, and record a numeric threshold for full B.
- **Why**: without the projection the full-B trigger is unenforceable prose.
- **Status: delivered** (`src/orchestrator/projection.py`). `project_run` times a real full replay from genesis and
  `events` real clones of the run's final state (the live path's clone upper bound), and `full_b_triggered` evaluates the
  threshold.
- **Threshold (fixed 2026-09 at first measurement)**: full B triggers when the projection shows clone+replay ≥ **60 s**,
  or ≥ **25%** of the run's wall-clock, whichever is lower.
- **First measurement** (frozen harness, `RECURSIVE_LAZY` seed 1001, 147 events, ~42 KB state): clone+replay **0.32 s**
  against a **3.24 s** trial wall-clock (~10%). The threshold therefore bounds future event volume and state growth, not
  today's runs; it stays as proposed.

## Phase B — Scaling

Triggered at self-host Phases 2-3 ("Arbitrary Target Repository", "Project-Owned Oracle"), where real repositories and
real event counts first enter, by the projection from Phase A.

### B5: Structural sharing

- Replace the cheap clone with structural sharing / persistent collections so clone cost is `O(changed)`, not
  `O(state)`, preserving the reducer's API and purity.
- **Rejected**: an in-place reducer with an undo log. The reducer raises mid-event on invalid input, so a missed undo
  entry silently corrupts replay equivalence and crash rollback; structural purity is the only safe route.
- **Safety net**: the existing state fuzzer, ledger property tests, and replay conformance.

### B6: Snapshots as a replay cache

- Checkpoint state so replay and recovery replay only the tail since the last snapshot.
- **Requires normative amendments**: `control-plane.md` requires recovery to "replay the complete authoritative event
  log", and `store.py` commits to "never uses snapshots". Snapshots are permitted only as a performance optimisation
  (see [Non-Goals for the First Version](../roadmap.md#non-goals-for-the-first-version)) and must stay non-authoritative,
  covered by the replay-equivalence tests.

### B7: Git-native worktrees and content-addressed digests

- Use Git worktree/checkout instead of full tree copies per attempt, and accelerate digests.
- **Constraint**: tamper detection stays content-addressed. Only accelerate digests where the adversary cannot influence
  the metadata; mtime/size keys are candidate-influencable and would let a candidate rewrite content, restore metadata,
  and be graded `PASS`.

## Phase C — Throughput

At self-host Phase 6 ("Operational Envelope"), after the experiment design is frozen.

- Parallelise trials and arms, and model calls, bounded by the spend cap and conflict policy.
- **Blocked on the experiment-design freeze**, not merely deferred: the manifest freezes the worker treatment and records
  `sequential_model_reviews_in_concurrent_arm` (`experiment.py:1125-1127`), so parallelising changes the experiment's
  definition and needs a manifest-version bump and deliberate metrics interpretation.
- Parallel trials must go through the multi-writer spend-ledger path.

## B-lite Recipe and Input-Purity Property

The recipe is small but exact. A cheap clone must give the reducer a state whose every container it can mutate is
unshared:

- fresh `RunState` top-level containers;
- a fresh `Ledger` and its six dicts (`budgets`, `entries`, `reservations`, `commitments`, `charged`, `overruns`) —
  `Ledger` is a nested record (`model.py:470`) mutated by the reducer at `reducer.py:410, 436, 593, 734, 836, 883`;
- copies of the `LedgerEntry` objects the ledger holds — the ledger flips `entry.active` in place
  (`model.py:342, 358, 394`; field at `model.py:199`), so a fresh `entries` dict whose values are shared is not enough;
- per-record shallow copies of records whose scalar fields the reducer assigns;
- a copy of `BudgetRecord.limits` — `Ledger.transfer` writes `source.limits[dimension]` / `target.limits[dimension]`
  in place (`model.py:445-446`);
- copies of the inner lists in `failure_fingerprints` / `failure_strategies` — appended in place at
  `reducer.py:1027-1028`.

The gate is an **inventory-independent input-purity property**, not a hand-written list: after `reduce(state, event)`,
the input `state` must deep-equal a pre-call `deepcopy` of itself, applied across the fuzzer corpus **including calls
that raise** (`ReducerError` mid-application is exactly where a partial in-place mutation would hide). A per-call check
is sufficient to catch cross-call aliasing too: any in-place mutation targets an object reachable from the immediate
input, so the input changes at the call where the mutation happens and the check catches it there. The property's own
`deepcopy` use is test-only and does not affect production clone cost.

## Guardrails

These must not regress, and they constrain every item above:

- Untrusted candidate and model code runs only inside the sandbox; per-check evaluator isolation (R08) is preserved or
  superseded by a reviewed equivalent.
- Snapshots are never authoritative; determinism and replay equivalence hold.
- Concurrency stays within budget and conflict policy.
- Failures fail closed and are recorded; budget exhaustion, depth limits, and timeouts never imply success.

## Review Rationale

The plan was adversarially reviewed; the following corrections are folded in:

- **A4**: the poll is a deliberate cancellation mechanism (R04), and its wall-clock win is small because the parent's
  sleep is the child's runtime; re-scoped to per-attempt sandboxing and downgraded to "measure, do not assume".
- **A2**: the per-check disposable-copy design is the basis of the R08 GO, so consolidation must schedule the R08
  re-review as acceptance.
- **A1**: cache only decisive grades; never cache non-deterministic outcomes.
- **B6**: snapshots require normative amendments and must stay non-authoritative.
- **B7**: tamper detection stays content-addressed.
- **B5**: the in-place undo-log option is rejected; prefer the cheap clone, then structural sharing.
- **B-lite**: the initial in-place-mutation inventory was incomplete. `LedgerEntry.active` is flipped in place, and the
  `failure_fingerprints` / `failure_strategies` inner lists are appended in place; both are now in the recipe, and the
  gate is the inventory-independent purity property rather than an enumerated list.
- **Optional deferral (considered, rejected)**: the folded Axis 3 runs execute on the current performance profile, so
  deferring them behind this block would make them cheaper but delay the capability signal. Keeping them folded with
  Axis 1 preserves the handoff's intent and is the agreed choice.
