<!-- normative for harness -->

# Experiment

Measured orchestration trials are prohibited until the controller passes the following deterministic conformance suites.

## Transition Completeness

- every documented event has a legal transition from each applicable state;
- every policy outcome resolves to a legal event path;
- every unlisted transition is rejected;
- terminal states cannot be reopened except through explicitly separate accounting settlement;
- root terminal states map to exactly one run outcome or an atomic replacement.

## Event and Replay

- duplicate events are idempotent;
- conflicting idempotency keys are rejected;
- stale proposals produce `VERSION_CONFLICT_RECORDED`;
- event replay produces state and ledger equivalence;
- timer, capacity, scheduling, and cancellation decisions are replayed rather than recomputed.

## Attempt and Accounting

- successful execution proceeds through result, validation, and acceptance;
- exhausted retries leave validation through a legal replan or terminal path;
- timeout followed by late usage settles cost without admitting a result;
- missing final usage causes conservative settlement;
- settlement is idempotent.

## Recovery

- crashes before dispatch produce no dispatched attempt;
- crashes after recorded dispatch fail the interrupted attempt under recovery;
- crashes after observation admission do not duplicate result or cost;
- crashes during integration or publication cannot expose an unrecorded authoritative artefact.

## Isolation

- sandbox escapes cannot modify controller state, authoritative Git metadata, hidden tests, credentials, host files, or
  unrelated worktrees;
- out-of-scope candidate paths are rejected at the publication gate;
- network and credential restrictions are enforced.

## Oracle Calibration

Before measured trials, run the frozen oracle against:

- a known-good solution;
- the unchanged repository;
- deliberately broken candidates;
- a candidate that times out;
- injected oracle infrastructure failures.

Record calibration outcomes in the experiment manifest. Oracle calibration and unmeasured model pilots do not contribute
to comparative results.

## Evaluation

### Outcome metrics

- root acceptance rate;
- hidden-suite verified progress;
- criterion-weighted verified progress;
- verified correctness;
- false-success rate;
- correctly unresolved rate;
- regression rate;
- constraint violations;
- termination reason distribution;
- hidden-oracle infrastructure-error rate.

`False-success rate` is the fraction of gradeable trials where the controller reports root `ACCEPTED` and at least one
mandatory hidden acceptance check has outcome `FAIL` or `CANDIDATE_TIMEOUT`.

This corresponds to the `FALSE_SUCCESS` classification in
[Experiment procedure](experiment.md#experiment-procedure). Trials with a remaining non-gradeable oracle
outcome are `INDETERMINATE` and are excluded from the false-success denominator.

### Decomposition metrics

- semantic coverage;
- branching factor;
- depth;
- decomposition rejection rate;
- duplicate work;
- integration failures;
- reusable accepted nodes;
- invalidations caused by hidden coupling.

### Parallelism metrics

- ready-action concurrency;
- realised speed-up;
- conflict rate;
- stale-result rate;
- merge or integration overhead;
- worker utilisation;
- work cancelled after invalidation.

### Cost metrics

- model calls;
- tokens;
- deterministic actions;
- validation cost;
- integration cost;
- retries;
- speculative cost;
- cost per accepted root result;
- model cost avoided by deterministic execution.

### Stopping metrics

- premature leaf decisions;
- unnecessary decomposition;
- no-progress recursion;
- budget exhaustion;
- honest blocked outcomes;
- false acceptance.

### Routing metrics

- verified success by model and task type;
- cost per accepted output;
- schema-failure rate;
- validation-failure rate;
- fallback frequency;
- value of independent evaluation.

## Experiment procedure

### Fixed target

Use one frozen repository snapshot containing a small but non-trivial implementation task. The repository must:

- build and test offline in the fixed experiment image;
- require changes across at least three components;
- permit at least two changes with statically disjoint write scopes;
- include at least one integration dependency;
- complete without production deployment, personal data, financial data, network access, or external mutable services.

The fixed target's integration dependency must be exercised through the incremental integration-tree protocol. A
dependant action must receive a `base_commit` containing its accepted predecessor's integrated changes.

Before any measured run, publish an experiment manifest containing:

- target repository commit;
- project-kernel version and hash;
- task statement and contract hash;
- visible build, lint, type-check, and test commands;
- environment-image digest;
- controller, reducer, policy, schema, tool, and model-adapter versions;
- model identifier and fixed sampling parameters;
- allowed tools;
- fixed budgets;
- hidden-suite manifest hash;
- trial seed and arm assignment.

The project kernel and task statement are identical across arms. Model context differs only where the arm's
orchestration method requires it.

### Project kernel

The project kernel maintained in the manifest comprises:

- purpose;
- target users;
- desired outcomes;
- current scope;
- current non-goals;
- critical constraints;
- system-level acceptance approach;
- decision authority;
- critical risks.

### Independent oracle

The experiment owner writes and freezes a hidden acceptance suite before measured runs.

The suite:

- is unavailable to executors and reasoning models;
- maps every hidden test to one or more root acceptance-criterion identifiers;
- runs in the same fixed environment from the final integrated repository tree;
- records pass, fail, timeout, and infrastructure-error outcomes;
- distinguishes infrastructure failure from project failure;
- is unchanged across arms and trials.

Model-generated tests do not contribute to the hidden score.

The frozen oracle manifest must satisfy bidirectional coverage before trials:

1. every hidden check maps to at least one root acceptance criterion;
2. every mandatory root acceptance criterion maps to a nonempty set of mandatory independent checks or to one explicitly
   declared independent non-test oracle;
3. every criterion-to-check mapping references existing immutable identifiers;
4. every mandatory check has a declared expected outcome, timeout policy, and weight;
5. no mandatory criterion receives credit from an empty check set;
6. the manifest contains no duplicate check identifier with conflicting mappings.

The experiment harness validates these conditions deterministically. An incomplete or inconsistent mapping invalidates
the experiment manifest and no trial may start.

### First-round arms

Compare only:

1. `DIRECT`: one undecomposed root node is given to the same model adapter used by the other arms. The model may use the
   same permitted repository tools, visible build and test feedback, repair loop, retry policy, and aggregate
   model-token, action, tool, cost, and wall-clock caps. It may not create child nodes, delegate work, or execute
   parallel model attempts.
2. `FIXED_ONE_LEVEL`: a fixed predeclared one-level decomposition executes sequentially;
3. `RECURSIVE_LAZY`: the controller permits recursive lazy decomposition and executes ready work sequentially;
4. `RECURSIVE_CONCURRENT`: the same policy as arm 3, with concurrent execution only for deterministically disjoint write
   scopes.

Use the same single model adapter and model profile in all four arms.

Multi-model routing is deferred until multiple adapters and routing policies exist. Deterministic replacement of model
work is measured later as an ablation on tasks with a valid deterministic substitute, not as a first-round arm.

An attempt is not synonymous with one generation. A model attempt may contain the protocol-defined iterative
tool-feedback loop until it returns, fails, times out, is cancelled, or exhausts its attempt commitment.

Any difference in per-attempt feedback, retry permissions, tool access, or aggregate budget must be treated as a
separate experimental factor and may not be attributed solely to decomposition.

The first-round `DIRECT` arm is therefore an undecomposed baseline, not a single-shot baseline.

### Fixed budgets

Each trial uses these default hard limits unless the experiment manifest records a different frozen set before any run:

<!-- gen:limits -->

```yaml
limits:
  wall_clock_seconds: 3600
  root_model_input_tokens: 120000
  root_model_output_tokens: 30000
  model_calls: 24
  actions: 120
  nodes: 30
  maximum_depth: 5
  retries_per_action_lineage: 2
  consecutive_same_diagnostic_failures: 2
  concurrent_workers:
    DIRECT: 1
    FIXED_ONE_LEVEL: 1
    RECURSIVE_LAZY: 1
    RECURSIVE_CONCURRENT: 4
  maximum_children_per_node: 6
```

<!-- /gen:limits -->

The concurrency difference is the intended treatment in arm 4. Cost limits, model parameters, retry policy, target, and
oracle remain equal across arms.

Run multiple independently seeded trials per arm. Trial count is fixed in the experiment manifest before results are
inspected.

The MVP does not spend a separate model call to choose between `EXECUTE` and `PROPOSE_DECOMPOSITION`. For every `READY`
node, the controller deterministically evaluates action eligibility. The selector is specified in
[READY-node action selection](control-plane.md#ready-node-action-selection).

`root_model_input_tokens` means billed input tokens across every turn of every model tool-feedback loop in the run,
including repeated system context, repository excerpts, tool observations, retry prompts and decomposition prompts.
Cached-token treatment must follow the provider invoice semantics frozen in the manifest and must be reported separately
where available.

A model call is one provider request. An attempt may contain multiple model calls in an iterative tool-feedback loop.
Both per-attempt and root aggregate limits are enforced.

The displayed limits are initial placeholders, not validated scientific constants. Implementation-sequence steps 29 and
30 run unmeasured pilots to calibrate model calls, billed input and output tokens, per-attempt commitments, wall-clock
limits, node limits and retry limits. The calibrated values and calibration method are frozen in the experiment manifest
before arm assignments or measured outcomes are inspected. No limit changes are permitted after the manifest freezes.

### Verified progress

After the run terminates, execute the hidden suite exactly once against the final integrated root tree.

The hidden suite produces a comparable score only when every mandatory hidden check reaches a gradeable outcome.

Gradeable outcomes are:

- `PASS`
- `FAIL`
- `CANDIDATE_TIMEOUT`

Non-gradeable outcomes are:

- `ORACLE_INFRASTRUCTURE_ERROR`
- `NOT_EXECUTED`
- `RESULT_LOST`

A timeout is `CANDIDATE_TIMEOUT` only when the calibrated timeout is exceeded by candidate behaviour inside a healthy
oracle environment. It counts as `FAIL`.

A timeout caused by runner startup, unavailable infrastructure, corrupted test fixtures, or harness failure is
`ORACLE_INFRASTRUCTURE_ERROR`.

If any mandatory check has a non-gradeable outcome, the trial's hidden result is `INDETERMINATE` and neither
`verified_progress` nor `criterion_verified_progress` is reported as a numeric value.

For a complete oracle execution:

```text
verified_progress =
    passed_mandatory_hidden_checks
    / total_mandatory_hidden_checks
```

Also report criterion-weighted progress:

```text
criterion_verified_progress =
    sum(weights of mandatory root criteria whose nonempty mandatory oracle set completed and passed)
    / sum(weights of all mandatory root criteria)
```

A mandatory criterion with an incomplete oracle execution is ungraded, not passed.

Weights are fixed before the run. The primary metric is unweighted `verified_progress`.

The experiment manifest must freeze an infrastructure-only rerun policy before trials. The default policy permits one
rerun of only the non-gradeable checks against the identical frozen candidate tree, oracle version, and environment
image.

Candidate failures, candidate timeouts, and ordinary test failures are never rerun.

If any mandatory check remains non-gradeable after the permitted infrastructure-only rerun, the trial remains
`INDETERMINATE` and is excluded from comparative effectiveness estimates. Its cost and infrastructure failure remain
reported.

For a complete hidden-oracle execution:

- `CORRECT_ACCEPTANCE`: controller outcome is `ACCEPTANCE_CLAIMED` and every mandatory hidden check passes.
- `FALSE_SUCCESS`: controller outcome is `ACCEPTANCE_CLAIMED` and at least one mandatory hidden check fails or produces
  `CANDIDATE_TIMEOUT`.
- `CORRECTLY_UNRESOLVED`: controller does not claim acceptance and at least one mandatory hidden check fails or produces
  `CANDIDATE_TIMEOUT`.
- `MISSED_SUCCESS`: controller does not claim acceptance and every mandatory hidden check passes.
- `INDETERMINATE`: at least one mandatory hidden check remains non-gradeable.

No correctness classification is assigned to an indeterminate trial.

### Cost and comparison

For every trial, report:

- verified progress;
- criterion-weighted verified progress;
- root acceptance outcome;
- false-success indicator;
- correctly unresolved indicator;
- total model input and output tokens;
- model cost;
- deterministic action count and compute time;
- validation and integration cost;
- elapsed wall-clock time;
- attempts, retries, nodes, depth, and conflicts;
- stale or cancelled work;
- final termination reason.

Compare arms using distributions across trials, not only the best run.