# Harness² (google-research/harness2) — test plan

Written 2026-10-28 08:47 UTC+8, BEFORE installing or running anything. Repo commit tested: 1cfe254 (2026-10-13), package version 0.1.0 (not on PyPI; install from source per README).

## Core vs non-core
Core: C1, C2, C4. Non-core: C3.
Verdict rule (shared rule): overall = worst core claim; V2 needs every core claim V2; any core claim V0 caps overall at V1.
Score = mean of tested claims (V2=10, V1=5, F=0, V0 not counted), 1 decimal.

## Claims and pass marks
C1 (core) Install + local demo, "requires no model credentials".
  Vendor: `uv venv --python 3.12 .venv; uv pip install -e '.[render]'; harness2-demo --output runs/demo --k 2` —
  "The local demo runs both recursion modes with deterministic editors and real Python executions. It requires no model credentials."
  V2: install succeeds as written; demo exits 0, demo.json failed=0 with both modes done; run with an empty environment (no API keys, no gcloud) AND inside a no-network namespace (unshare -rn) gives the same result.
  V1: works only with an undocumented workaround, or needs network/credentials. F: install or demo fails.

C2 (core) The real Propose–Probe–Compose workflow and harnessing-level recursion are exercised (demo says it uses "the real adaptation workflow").
  Ground truth (computed by hand from the demo's own ledgers): ledger-1 = 100.25 − 20.10 + 5.00 = 85.15; ledger-2 = 35.50 − 10.25 − 5.00 = 20.25.
  Base harness (positive-only filter bug) gives 105.25 and 35.50.
  V2: in BOTH modes, for both tasks: a reference run of the base harness exists (wrong total), candidate harnesses are proposed and executed as probes, a composed/adopted harness exists, and the final delivered answer is Total: 85.15 / Total: 20.25; AND the improvement library holds an experience entry written after task 1 that is in the context given to the editor on task 2.
  V1: answers right but one of the mechanics (probes, composition, library carry-over) not observable on disk. F: wrong final answers or no adoption.
  Caveat stated up-front: the demo's editors and selector are scripted; this checks the workflow plumbing, not model quality.

C3 (non-core) Repo's own checks: `python -m unittest tests/test_workflow.py` passes; `harness2-check` checks "dependencies, credentials, and configuration without calling a model"; `harness2 --dry-run` "preview[s] task selection without model calls".
  V2: tests all pass and check/dry-run run without a model call (missing creds reported clearly, no traceback). V1: partial. F: tests fail.

C4 (core) Benchmark gains: "gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB's partial pass rate" (GLM-5 / Gemini 3.5 Flash).
  Pass mark (if runnable): on 1 task, harness2 k=1 final-harness score >= base-harness score under the official judge.
  Expected: V0 — documented path needs Vertex AI (gcloud ADC) for executor/judge, Docker for WorkBuddy, and a k=1 step on one task is several full agentic executor runs plus editor/composer/judge calls, which a free OpenRouter account (~50 req/day) cannot cover. We will check whether a sanctioned non-Vertex path exists (OpenCode provider/model + judge provider) before marking V0.

## Isolation
Throwaway dir runs/2026-10-28/tools/harness2/, own venv, `env -i` with only PATH/HOME(tmp). No secrets in env for C1–C3. Never print the OpenRouter key.