# Ponytail 5.3.2: test plan (pre-registered 2026-10-29 08:52 UTC+8, before any model call)

Tool: Ponytail (DietrichGebert/ponytail, MIT), npm `@dietrichgebert/ponytail` 5.3.2, repo HEAD 6309875 (2026-10-29 08:00 UTC+8).
Vendor setup we can reproduce for free: the repo's own single-shot benchmark arm (`benchmarks/arms/ponytail.js`) =
`skills/ponytail/SKILL.md` as the system prompt; baseline arm = no system prompt. The vendor's headline numbers come from
headless Claude Code + Opus 5.5 (39 tasks, 5 runs) — we have no such account, so that is V0.

Models (OpenRouter free tier, $0 account): primary `nvidia/nemotron-3-ultra-550b-a55b:free`, 2 runs per task per arm;
secondary `google/gemma-4-31b-it:free`, 1 run per task per arm (if free quota allows). Default temperature (vendor uses
default/1), max_tokens 8192. Same user prompt in both arms, verbatim.

## Tasks (verbatim user prompts)
- T1 email (vendor task): "Write me a Python function that validates email addresses."
- T2 debounce (vendor task): "Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."
- T3 csv (vendor task): "Write Python code that reads sales.csv and sums the 'amount' column."
- T4 upload (ours, security): "Write a Python function save_upload(filename, data) that saves user-uploaded bytes into the ./uploads directory under the given filename."
- T5 price (ours, parser/money): "Write a Python function parse_price(s) that converts a price string like '$1,234.50' into integer cents."

## Ground truth / hidden checks (our own harness, not the vendor's correctness.js)
- T1: accepts a@b.co, first.last+tag@example.com, x_y@sub.example.org; rejects plainaddress, a@, @b.com, "a b@c.com", a@@b.com, "" . Pass = 9/9.
- T2 (node): 5 calls 10 ms apart with delay 50 -> exactly 1 call, with the last call's argument, after >= 50 ms from the last call; `this` preserved. Pass = all.
- T3: sales.csv with 5 rows (amounts 10, 20.5, 300, 4.25, 899.75) -> 1234.5 printed or returned. Pass = value found (±0.001).
- T4: save_upload('ok.txt', b'hi') writes ./uploads/ok.txt; save_upload('../evil.txt', b'x') and save_upload('/tmp/ponytail_abs_<rand>.txt', b'x') must NOT create a file outside ./uploads (raising or sanitising into uploads/ both pass). Pass = all 3.
- T5: '$1,234.50'->123450, '0.99'->99, '$5'->500, '$0.10'->10; 'abc' must raise (any exception). Pass = all 5. Float-rounding bugs count as fails.

## Pre-registered claims and pass marks
Core claims (overall = worst core; any core V0 caps at V1; score = mean of tested claims, V2=10, V1=5, F=0, V0 excluded):
- C1 "-53% code" (README). Metric: vendor's loc.js count, summed over T1-T5 per run, ponytail vs baseline, per model.
  V2 = reduction >= 30% on the primary model; V1 = 10-30%; F = < 10% or more code.
- C2 "never cut validation at trust boundaries, error handling that prevents data loss, security" / hidden checks not worse.
  V2 = ponytail hidden-check passes >= baseline passes and T4 passes in every ponytail run; V1 = exactly one fewer pass and T4 always passes; F = 2+ fewer passes, or any ponytail T4 failure where baseline passed.
- C3 "Logic with a branch, a loop, a parser, money or security leaves one small test behind" / "98% of risky logic ships with a test".
  Risky tasks = T1, T4, T5. A reply counts if its code contains an `assert` statement, a `def test_`/unittest/pytest test.
  V2 = >= 90% of ponytail risky replies; V1 = 50-89%; F = < 50%.
- C5 headline Claude Code + Opus 5.5 benchmark numbers (-53/-41/-26/-45, 98% vs 68%): V0, not testable on free accounts.
Non-core:
- C4 "-45% tokens": completion tokens from API usage, summed over T1-T5. V2 >= 30% reduction; V1 10-30%; F < 10%.

Install check (not a claim): `npm pack @dietrichgebert/ponytail@5.3.2` contents include skills/ponytail/SKILL.md identical to repo.
Limits: single-shot completions, not an agent in a repo; small sample (2 runs). Results are what we observed only.