# Headroom test plan (fixed BEFORE installing or running Headroom)
Written: 2026-10-20 11:18:15 UTC+8. Grok Bot box: Linux x86_64, 8 vCPU, 15 GB RAM, no GPU. No LLM API key; a local model via Ollama on CPU stands in for the LLM.
Version under test: latest PyPI headroom-ai (GitHub release v0.41.0 at 11:17). Telemetry beacon is on by default per README; we turn it off with HEADROOM_TELEMETRY=off and DO_NOT_TRACK=1 for every run.

## Vendor claims (README, https://github.com/headroomlabs-ai/headroom, fetched 2026-10-20 11:17 UTC+8; 76,311 stars)
C1 "20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers." (repo tagline) and the README proof table (seeded): code search 21%, SRE incident 57%, codebase exploration 42%, GitHub issue triage 30%, reproducible with 'uv run python benchmarks/index_proof_table.py --seed 20260902'.
C2 "Same answers, fraction of the tokens." (accuracy)
C3 "Compression costs well under a millisecond: 0.21 ms p50 on a 10K-token JSON search result, 1.4 ms at 100K tokens."
C4 "Compression runs on your machine; no prompt or file content is sent anywhere to be compressed."
Pricing: open source Apache-2.0, free; paid "Headroom Enterprise" for teams (no public price; "3 months free with a 12-month plan").

## Ground truth and method
C1a: run the vendor's own proof script with the published seed; pass if our numbers are within 2 percentage points of the README table.
C1b: our own JSON payloads, independent of the vendor: (1) GitHub REST API list of 100 issues from a public repo, (2) 100 PyPI package JSON 'info' blocks reduced to a list, (3) a synthetic 2,000-line JSON log array with repeated structure and one injected FATAL record, (4) a 300-result JSON search result built from Wikipedia search API output. Measure tokens before/after with Headroom's own counter and with tiktoken o200k_base. JSON claim: V2 if all 4 payloads save >= 60%; V1 if some do; F if none.
C2: for each of the 4 payloads, 5 questions with exact answers we compute from the raw JSON by code (counts, a specific field of a specific item, the FATAL message, max/min values) = 20 questions. (a) Programmatic: is each answer string still present in the compressed text, or retrievable through Headroom's documented retrieval (CCR)? (b) LLM: a local model (Ollama, small Qwen on CPU) answers each question from the original and from the compressed input, temperature 0; score exact match. Pass: V2 if compressed accuracy >= original accuracy - 1 question and >= 95% of answers retained or retrievable; V1 if retention 70-95% or accuracy drop 2-4; F otherwise.
C3: time compress() on a 10K-token and a 100K-token JSON search result, 50 runs each after 3 warm-ups, report p50; V2 if p50 <= 1 ms at 10K tokens, V1 if <= 10 ms, F above.
C4: run compression with networking watched (ss/strace on connect syscalls) after models are downloaded; V2 if no outbound connection during compress() with telemetry off; note any connection with telemetry on is out of scope (we keep it off).
Overall = worst of C1b, C2, C3, C4; C1a reported. Access level L0 (local, public/synthetic data); typical_level L1 (in real use it reads your agent's tool outputs and files).