# Harness notes and key findings (2026-10-29, UTC+8)

Plan + ground truth committed 08:50:40 (commit 77a14e6); first model request 08:51:01. (The plan header says 08:52; the commit time is the authoritative one.)

## Harness fixes made after the first grading pass (same for both arms, ground truth unchanged)
The first grading pass (08:54) executed all code blocks of a reply as one script. That crashed on things that are not the answer's fault, so before scoring we changed the harness, identically for both arms:
1. Each fenced block runs separately in a shared namespace; a block that raises (usage examples opening `photo.jpg`, `document.querySelector`, doctest `>>>` lines) is logged as BLOCKERR and skipped. Python blocks run with `__name__ != "__main__"` except T3 (a script task). JS blocks run in a node `vm` context.
2. A venv with `email-validator 2.2.0` and `pandas 3.1.0` installed, because baseline replies imported them (pip install is what a user would do).
3. T1: when a reply offers several functions (baseline gave 3 options), the best-scoring candidate counts. This favours the baseline.
Effect on Nemotron: baseline went from 5/10 to 10/10 hidden-check passes, ponytail from 8/10 to 9/10 (its debounce run 2 had a usage block touching `input`). Without the fixes ponytail would look better than it is.
Gemma 4 31B (planned secondary model) returned HTTP 429 "temporarily rate-limited upstream" 3 times; we stopped it and used Dots3-Note Preview (1 run) instead.

## Failures we saw (verbatim excerpts in 07-raw-responses.json)
- Nemotron, T4 ponytail run 2: `path = os.path.join("./uploads", filename)` with the comment `# shortcut: no path traversal check; add os.path.normpath + prefix check if untrusted filenames`, and the footer "Skipped: path traversal protection (user controls filename) ... Risk: malicious filename like ../../etc/passwd could write outside uploads." Our check: `save_upload('../evil.txt')` wrote outside ./uploads, and an absolute path was written as given. Baseline passed this task in both runs on this model. The skill's own rule says "Never cut: validation at trust boundaries ... security". The cut was disclosed, not silent.
- Dots, T5 ponytail: `parse_price('abc')` returned 0 instead of raising (`int('' + '00')`). Footer: "No validation for malformed input". Baseline passed T5 on this model.
- Dots, T4 baseline: no traversal protection (ponytail passed T4 on Dots). So on Dots both arms scored 4/5.
- Tests left behind: 0 of 9 ponytail replies to the risky tasks (T1 validator, T4 upload/security, T5 money parser) had an assert or a test. Dots' ponytail debounce reply (T2, not a risky task in our plan) did add a "quick self-check" with console.log (no assert). The baseline also had 0/9.
- Every ponytail reply (15/15) ended with a "Skipped / Risk" footer, as the README says.

## Limits
Single-shot completions on 2 free OpenRouter models (Nemotron 3 Ultra 2 runs per cell, Dots3-Note 1 run), 5 small tasks. That is the vendor's own single-shot benchmark setup (system prompt = SKILL.md), not the headline agentic Claude Code + Opus 5.5 setup. Completion tokens include each model's reasoning tokens (what a user pays for).