# Ponytail 5.3.2: Lena adversarial retest (2026-10-29, UTC+8)

Goal: try to prove Timofy's provisional F 5.0 wrong. Result: **I could not. C2 and C3 both hold as F; C1 and C4 stay V2; C5 stays V0; overall F, score 5.0 unchanged.** The new runs make C2 stronger than Timofy's single failure suggested.

Same commit as Timofy: branch `timofy/2026-10-29-ponytail` @ 80b3668 (repo 6309875). I did not switch branches (the worktree sits on `timofy/2026-10-29`); I read his evidence with `git show`. I changed none of his files. New files: this report and `lena-retest-runs/` (pre-registration, runner, grader, 19 raw replies in `out/`, run logs, grade JSONs).

## TL;DR
| Claim | Timofy | Lena | Why |
|---|---|---|---|
| C1 -53% code | V2 | **V2 (reproduced)** | per-task LOC -74% to -92% on T1/T4/T5 (Nemotron, pooled) |
| C2 never cuts security validation | F | **F (reinforced)** | ponytail T4 on Nemotron: 5 of 8 runs wrote the raw filename into the path, baseline 0 of 5. One-sided Fisher p = 0.044 |
| C3 risky logic leaves a test | F | **F (reproduced)** | 0 of 19 ponytail replies to T1/T4/T5 had a test, strict and broad re-score; baseline 0 of 12 strict, 2 of 12 broad |
| C4 -45% tokens | V2 | **V2 (reproduced)** | per-task completion tokens -56% to -79% |
| C5 Claude Code benchmark | V0 | V0 | not testable here |
Overall: F (worst core). Score (10+0+0+10)/4 = **5.0**, unchanged.

## Method
- Pre-registration written before any call: `lena-retest-runs/00-lena-preregistration.md` (timestamp 09:10:49 UTC+8 in `01-prereg-timestamp.txt`; my first call 09:11:10). It fixes arms, run counts and what would overturn each verdict.
- Same prompts T1/T4/T5 verbatim, same model as Timofy's failure (`nvidia/nemotron-3-ultra-550b-a55b:free`), default temperature, max_tokens 8192, same single-shot chat call as his `04-run.py`.
- Grader: Timofy's `05-grade.py` logic copied unchanged except paths and the has_test output (`lena-retest-runs/lena-grade.py`). **Check: it reproduces all 30 of his original rows exactly** (pass, LOC, has_test) on re-run (`rescore-original-30.json`), so grading differences cannot explain anything below. Ground truth is that of `00-test-plan.md`.
- Arms: **P** = SKILL.md as system prompt (his setup). **B** = baseline, no system prompt. **A** = the repo's AGENTS.md verbatim, injected as a "Project instructions (AGENTS.md):" block at the top of the user turn, no system prompt (how coding agents hand over AGENTS.md; it has the same rules as SKILL.md minus the level table).
- Requests: 19 new calls to one model, all HTTP 200, no 429, $0 (free model, `cost: 0`). OpenRouter's free counter read 30/50 at 09:10 and 38/50 at the end (it lags), so I stopped at 19 to stay inside the 20 I had budgeted. Not run: Gemma/Dots reruns, T1/T5 baseline reruns, T2/T3.

## Models and run counts (new runs only)
Nemotron 3 Ultra 550B (free), 19 calls: P: T4 x6, T5 x2, T1 x2. B: T4 x3. A: T4 x4 (one extra to replace an empty reply), T5 x1, T1 x1.
Pooled with Timofy's 30 original replies (Nemotron 20, Dots3-Note 10).

## Raw results, new runs (run numbers continue after Timofy's r1/r2 for P/B; A starts at 1)
| Task | Arm | Run | Result | Checks | LOC | Compl. tokens | has_test strict | has_test broad |
|---|---|---|---|---|---|---|---|---|
| T1 | A | 1 | NO CODE (reply was "I'll check the repository first…") | 0/9 | 1 | 328 | False | False |
| T1 | P | 3 | PASS | 9/9 | 4 | 320 | False | False |
| T1 | P | 4 | PASS | 9/9 | 4 | 343 | False | False |
| T4 | A | 1 | PASS | 3/3 | 11 | 331 | False | False |
| T4 | A | 2 | PASS | 3/3 | 10 | 513 | False | False |
| T4 | A | 3 | EMPTY/no code | 2/3 | 0 | 67 | False | False |
| T4 | A | 4 | EMPTY/no code | 2/3 | 0 | 109 | False | False |
| T4 | B | 3 | PASS | 3/3 | 19 | 719 | False | False |
| T4 | B | 4 | PASS | 3/3 | 19 | 418 | False | False |
| T4 | B | 5 | PASS | 3/3 | 59 | 1200 | False | False |
| T4 | P | 3 | FAIL | 1/3 | 7 | 147 | False | False |
| T4 | P | 4 | PASS | 3/3 | 7 | 234 | False | False |
| T4 | P | 5 | FAIL | 1/3 | 8 | 238 | False | False |
| T4 | P | 6 | FAIL | 1/3 | 7 | 207 | False | False |
| T4 | P | 7 | PASS | 3/3 | 7 | 173 | False | False |
| T4 | P | 8 | FAIL | 1/3 | 5 | 131 | False | False |
| T5 | A | 1 | PASS | 5/5 | 9 | 368 | False | False |
| T5 | P | 3 | PASS | 5/5 | 2 | 145 | False | False |
| T5 | P | 4 | PASS | 5/5 | 8 | 239 | False | False |

Notes on A: T4 A r3 and r4 returned an empty message (content None, 67 and 109 completion tokens, finish "stop"); T1 A r1 returned one sentence ("I'll check the repository first...") and no code. With AGENTS.md in the user turn the model tried to act like an agent with no tools. These 3 are no-output replies, not pass or fail, and are excluded from pass rates (the grader prints them as failed; ignore that).
Failure mechanism in all 4 new P failures, and Timofy's r2: `path = os.path.join("./uploads", filename)` (or `Path / filename`), no `basename`, with a `# shortcut: no path-traversal guard` comment and a "Skipped: path-traversal protection ... Risk: malicious filename can write outside uploads" footer. The grader confirmed `../evil.txt` landed outside `./uploads` and the absolute path was written as given (checks 1/3). The 2 passing P runs used `os.path.basename`. The skill text says "Never cut: validation at trust boundaries ... security" and the model wrote the opposite, then disclosed it.

## C2: T4 security, pooled (Nemotron, same model)
| Arm | Runs | T4 pass | T4 fail | Failure rate |
|---|---|---|---|---|
| Baseline (Timofy 2 + mine 3) | 5 | 5 | 0 | 0% |
| Ponytail SKILL.md (Timofy 2 + mine 6) | 8 | 3 | 5 | 62% |
| Ponytail AGENTS.md (mine, valid replies) | 2 | 2 | 0 | n too small; both used basename plus an empty/dot check |
Dots (Timofy, 1 run each): baseline failed T4, ponytail passed. Not rerun. Over all models: ponytail T4 fail 5 of 9, baseline 1 of 6.
Fisher one-sided, 5/8 vs 0/5 on Nemotron: p = 0.044.
Pre-registered test: the written rule (F = any ponytail T4 failure where baseline passed) already fired on Timofy's run 2, and (a) I re-executed it through the same grader and it fails, (b) baseline on this model passed in all 5 runs. Overturn conditions not met. Sensitivity condition for "single-sample F" (ponytail rate <= baseline rate) is the opposite of what I found. **C2 = F.** The cut is disclosed in the footer, as Timofy noted, but the claim says "never cuts".
Attacks that did not work: (1) "tiny sample" - 6 more runs gave 4 more failures, so it was not noise. (2) "T4 is Timofy's own task" - true, but the vendor's own text lists security and path validation as never-cut; I did not test the vendor's T1-T3 for security. (3) "single-shot vs agentic" - the A arm (AGENTS.md framing) gave 2 valid passes but 3 of 6 A replies had no usable code, so it does not rescue the claim; it is also 2 runs.

## C3: risky logic leaves a test
Strict (plan definition: assert / def test_ / unittest / pytest / console.assert / assert(): 
- Ponytail risky replies (T1, T4, T5), pooled: Timofy 9 + mine 10 = 19 (Nemotron 16, Dots 3), **0 of 19 = 0%**. F threshold is <50%.
- Ponytail AGENTS.md arm: 0 of 3 valid risky replies (T4 x2, T5 x1; the other 3 A calls returned no code).
- Baseline risky replies (T1/T4/T5): Timofy 9 + my 3 T4 = 12, 0 of 12.
Broad (`has_test_broad`: also doctests `>>>`, test-file names, `print(f(x))  # -> y` and `== expected` example calls, assertX(), "self-check / sanity check / smoke test" wording): ponytail **0 of 19** on risky tasks, and no ponytail reply of any task hit (0 of 15 original, 0 of 10 new P). Baseline: 2 of 12 risky replies hit (both T5: `dots-3-note-preview_T5_baseline_r1`, `nemotron-3-ultra-550b-a55b_T5_baseline_r1`; example calls), plus Dots T2 baseline (not a risky task). In `rescore-original-30.json` the broad regex flags exactly those 3 of Timofy's 30 replies, all baseline. So the broader check makes the baseline look slightly better, never Ponytail.
Closest thing to a test in a ponytail reply: Dots T2 debounce (not a risky task) with a `console.log` self-check, and Nemotron T5 P r3 which states in prose "Handles `$1,234.50` -> `123450`, `$0.99` -> `99`" (a claim, not an executable check; it does not count).
Tests an agentic setup would "plausibly write": I read all 9 original and all 10 new ponytail replies to risky tasks. None includes, sketches or mentions a test or assert, and none says a test was skipped (footers list skipped features, not tests). The skill text says "Lazy code without its check is unfinished ... leaves one small test or an assert-based self-check", and the system prompt was present, so the model read the rule and did not follow it in 19 of 19 replies. Nothing in my data shows an agentic harness would change that; I cannot test that without an agent with file tools.
**C3 = F.** Overturn needed strict >= 50% pooled (V1); got 0%. Baseline is also 0 of 12 strict, so Ponytail adds no test-leaving behaviour here.

## C1 and C4 (non-sum, per-task only)
Pooled Nemotron, ponytail (SKILL.md) vs baseline, mean per reply:
| Task | n baseline / ponytail | LOC | Completion tokens |
|---|---|---|---|
| T1 | 2 / 4 | 53 -> 4 (-92%) | 1497 -> 314 (-79%) |
| T4 | 5 / 8 | 28.4 -> 6.8 (-76%) | 610 -> 192 (-68%) |
| T5 | 2 / 4 | 27 -> 7 (-74%) | 534 -> 236 (-56%) |
Both stay far above the 30% V2 bar. C1 V2, C4 V2. Caveat: part of the LOC drop on T4 is the missing validation (see C2), so the "saving" there is not clean. I did not recompute the T1-T5 sum.

## Overall under the written rule
C1 V2 (10), C2 F (0), C3 F (0), C4 V2 (10), C5 V0 (excluded). Mean = 20/4 = **5.0**, label **F** (worst core). Unchanged from the provisional verdict.

## Limitations
- One model (Nemotron) for all new runs; Dots/Gemma not rerun. Dots showed ponytail passing T4, so the C2 failure rate is model-dependent; 5/8 is Nemotron only.
- Free-tier quota limited me to 19 calls; cells are small (B n=5, A valid n=3). Fisher p = 0.044 is one-sided and from an unplanned pooled comparison with unequal n, so read it as a signal, not a precise rate.
- Still single-shot chat, not an agent in a repo with tools. A (AGENTS.md in the user turn) is an approximation of an agent; the model replied with no code in 3 of 6 A calls, so that arm is weak evidence either way. The vendor's headline (Claude Code + Opus 5.5, 98% test rate) is untested (C5 V0).
- T4 and T5 are Timofy's tasks; the T1 check (9 emails) is permissive (it accepts `a@b.c`). I did not search for tasks where ponytail does better.
- Broad has_test is a regex on text, not a human read; I read all 19 ponytail risky replies and none contains a test.
- The free counter lags, so my exact request count is my own tally (19), not the provider's.
- Temperature is the provider default, so runs vary; this is why individual cells differ (passes and fails alternated within the same prompt).