# security-audit-skill — independent adversarial retest

Author: Lena · 2026-10-28 (retest ran ~09:15–09:30 UTC+8) · Subject: Timofy's rating "V1, 8.3, unpublished" (commit 7ae19b4)
Method: try to prove the rating wrong; reproduce only where the evidence forces it. Nothing committed, pushed or edited; this is the only file written.
Scratch work was in /tmp/lena (copies of the skill and of the evidence extracted with `git archive 7ae19b4`; the worktree's checked-out branch is `timofy/2026-10-28` at 446c691, which does NOT contain the evidence files, so I read them from 7ae19b4 without checking anything out).

## TL;DR
- **Overall label V1: reproduce. Score 8.3: overturn (it should be 10.0 under the written arithmetic, see §5 — and that number is itself a rubric oddity).**
- C1 V2: reproduce. C2 V2: reproduce (fully re-run, plus 12 extra bad inputs of my own, all rejected).
- **C3 V1: overturn → V0.** Timofy's own plan has no V1 condition that matches what happened, but it has a V0 condition that matches word for word. Their `verdict_rule` text admits this ("which was our pre-registered V0 condition. We rate it V1") — that is a post-hoc override of pre-registered text.
- Detection facts in C3 are real and I found no leakage or cherry-picking. Nothing here suggests the tool is bad; the problem is only the level label and the score arithmetic.
- Could not do: a fresh C3 run (no free OpenRouter key in my environment, and the free quota is hard-capped until 2026-10-29 08:00 UTC+8 anyway). I did not look for or take any credential.

## 1. What I checked in the evidence

| Check | Result |
|---|---|
| Plan, ground truth and target were frozen before the run | OK. `bb26772` committed 08:53:26 (UTC+8); install 08:53:34; run started 08:54:21. `git diff bb26772 7ae19b4` on plan / ground-truth / target = empty (not edited afterwards). |
| Target actually matches ground truth lines | OK. In `02-target-server.py`: P1 SQLi = l.22 (fn 20-23); P2 traversal = l.39-40 (fn 38-41); P3 IDOR = fn 32-35, call at l.82; decoys get_note 26-29 (parameterised + owner) and read_avatar 44-49 (abspath + prefix check) are in fact safe. |
| Target used in run = target in plan | OK. `diff 02-target-server.py proj/app/server.py` identical. (Plan says git 7f244ec, run used re-initialised repo b0a0ca4; Timofy disclosed this; content identical.) |
| Ground-truth leakage into prompt | None. Prompt (`05-c3-prompt.txt`, byte-identical to `c3/prompt.txt`) names no bug class, function or line; it only says quick profile, budget 12, no execution, output dir. I queried the OpenCode sqlite DB (`home/.local/share/opencode/opencode-local.db`) for every tool call of all 8 sessions: reads/globs touched only `proj/` files, the skill's own files and the agent's own `notes/run-1/` output. **Nobody read `02-ground-truth.json` / `00-test-plan.md`**, even though they sat in the same worktree and the run used `--dangerously-skip-permissions` (a risk that did not materialise). No match for "ground-truth/tool-evidence/decoy/planted" in the DB or logs. |
| Soft hint in the target | `proj/README.md` says "Every user may only see and change their own notes and attachments." That states the security invariant (legitimate input for an audit, and a real project would have it) but it makes IDOR/traversal easier. Minor. |
| Findings vs ground truth | OK. `05-c3-findings.json`: idor-delete-endpoint evidence l.32/34/81/82; path-traversal l.38/39/40/70; sql-injection l.20/22/23/66. All inside ground-truth ranges. Decoys appear only as "correct pattern for comparison" evidence lines, never as findings; the ledger marks GET /avatar and GET /note "covered". 3/3, 0 decoys confirmed: **claim accurate.** |
| Validators pass on agent output | Re-run by me: `node validate-findings.cjs 05-c3-findings.json` → `PASS: 3 findings valid` (exit 0); `validate-coverage-ledger.cjs 05-c3-coverage-ledger.json` → `PASS: 6 coverage units valid` (exit 0). |
| Incomplete run | OK, and worse in detail than the summary: 8 LLM errors in `dev.log`, not 8 × 429 — 7 × HTTP 429 `free-models-per-day` (X-RateLimit-Limit 50, Remaining 0, Reset 1793232000000 = 2026-10-29 00:00 UTC = 08:00 UTC+8) and 1 × HTTP 402 (the title helper trying paid `anthropic/claude-haiku-4.5`, refused: "Insufficient credits", so nothing charged — confirms Timofy's OpenCode side note). The 429s hit the 2nd "final verification" sub-agent (path traversal) at 09:06:16 UTC+8; the DB shows that task still `running`. So only 1 of 3 findings got the phase-5 independent record verification. No REPORT.md, `run_status: in_progress`. The reset time is 22 h away, so there was no way to finish inside the 09:30 time box. |
| Skipped recon agents | Confirmed in DB: tasks were hunter (first try with unknown subagent type `research` errored, retried as `explore`), critic, 3 validators, 2 final verifiers (1 done, 1 running). No recon sub-agents. Budget "12 invocations" not exceeded (8 task calls). |
| "Confirmed without executing" | Confirmed, and slightly understated. Every ledger `local_check` is `source`. Each finding's `confidence.reason` says "Independent verification confirmed exploitability" and `observed_result` reads as if the exploit ran (e.g. "Server returns contents of /etc/passwd…", "UNION injection … returns all notes from all users"). Nothing was executed. The README says that without a sandbox leads should stay `needs_validation`; the model ignored that. The plan's V2 text allows "confirmed or needs_validation", so this does not change the rubric outcome, but it is a real defect to keep on the page. |
| Cherry-picking | None found. Single run, single model, failures (429, skipped recon, fabricated observed results, 64 requests vs 50/day) are all in the page text. Small slip: `05-c3-run-summary.txt` says "ERROR lines (429 …): 8"; it is 7 × 429 + 1 × 402. Cosmetic. |
| Difficulty of the task | Weakness of the test design, not a rule violation: 1 file, ~90 lines, bugs sit next to correctly-written twins (get_note / read_avatar) that any reviewer pattern-matches. 3/3 on this says "the loop works end-to-end up to phase 4", not much about audit quality. Timofy's caveat already says "one small target". |

## 2. C1 — install (non-core) — reproduce V2
Command (scratch HOME, empty git repo, Node 20.19.2):
```
cd /tmp/lena/c1/proj && git init -q . && HOME=/tmp/lena/c1/home npx -y skills add https://github.com/cloudflare/security-audit-skill --skill security-audit -a opencode -y
```
Result: exit 0 in 3.7 s; `diff -r .agents/skills/security-audit <repo>/skills/security-audit` → identical (repo at c1c8a8c). Matches the plan's V2 definition. **V2.**

## 3. C2 — validators (core) — reproduce V2
Re-run from a copy of the installed skill (`/tmp/lena/c2/skill`, diffed identical to repo c1c8a8c):
1. `cd /tmp/lena/c2/skill && node --test *.test.cjs` → `# tests 65 / pass 65 / fail 0` (exactly Timofy's 65/65; plain Node, no npm install).
2. Timofy's 6 fixtures (`04-c2-fixtures/*.json`), real exit codes read via `rc=$?` directly after the node call (note: Timofy's log shows `exit=0` for every file because of `| head`; they added a "real exit codes" block and I confirm it): valid.json 0, valid-needs-validation.json 0, bad1-missing-root-cause 1, bad2-needs-validation-with-severity 1, bad3-unsorted 1, bad4-line-zero 1. **2/2 accept, 4/4 reject with clear messages — claim accurate.**
3. My own extra inputs (not from the tests), all derived from valid.json — `validate-findings.cjs`: bad verdict value, empty trace, severity as string, extra unknown field, root not an array, duplicate fingerprint, negative line (-5), trace file `../../etc/passwd` → all 8 exit 1 with specific messages. `validate-coverage-ledger.cjs` on the agent's ledger with one corruption each: invalid status, `../../etc/passwd` in reviewed_paths, duplicate coverage_id, missing agent_id → all 4 exit 1. Valid agent output passes (see §1).
Caveat: Timofy's bad fixtures were built from the shape in the validator's own test file (make.cjs says so) after "glancing" at the test suite (disclosed in the plan), so those 4 are not independent of the vendor tests; my 12 extra cases are, and agree. **V2.**

## 4. C3 — six-phase audit (core): V1 → overturn to V0
Pre-registered text (`00-test-plan.md`, committed before running):
- V2: 3/3 planted at the right function, 0 decoys confirmed, findings.json passes the validator.
- V1: "1-2/3, or output fails validation, or a decoy confirmed."
- F: completed run with 0/3.
- **V0: "the run cannot complete within the free model quota (~50 requests/day) / time box (until 09:30 UTC+8)."**
- Header: "any core V0 caps overall at V1"; TRY-TOOLS.md §8: "V0 not counted" in the score.

Facts: quota-limited at 50/day (X-RateLimit-Limit 50); the run needed ≥64 LLM starts; HTTP 429 at 09:06:16; reset not until 08:00 tomorrow; no REPORT.md; phase 5 done for 1 of 3 findings; phase 6 not started. This is the V0 sentence nearly verbatim — including the "~50 requests/day" figure the plan itself named.

Why V1 is not available: none of V1's three listed conditions occurred (3/3 found, output validated, no decoy confirmed). Timofy reached V1 by reading the V0 trigger as "partial credit", which the plan does not provide. Why not V2: the V2 sentence is silent on completion, so the V2 bar was met on detection — that is the genuine dispute. But the claim being tested is a full six-phase audit (the plan's C3 line: "full audit … by a coding agent"; the prompt itself required "REPORT.md written"), and V0 is the only rule that speaks to completion, so the specific condition governs the general detection bar. A lawyer-style tie-break therefore gives V0, and the rubric's own purpose for V0 ("not tested", the item could not be run to the end in the allowed resources) fits. I considered V2 seriously and rejected it because it would reward a run whose phases 5-6 were never seen and whose `observed_result` fields are fabricated.

**C3 = V0 (partial evidence reported: detection 3/3, 0 decoys, validators pass, phases 5-6 unseen).** The page should say "V0 on the full-audit claim; what we did see:" and keep the tagline's facts.
If the founder prefers V1/V2 for such runs, the plan has to be amended openly (new commit, stated as post-hoc) — it cannot be done silently inside `verdict_rule`.

## 5. Overall verdict and score
- Rule (plan + TRY-TOOLS.md §8): overall = worst core claim; any core V0 → at most V1. Core = C2 (V2), C3 (V0) → **overall V1. Holds.** (Also true under Timofy's V1 label.)
- Score = mean of tested claims, V0 excluded: C1 10, C2 10, C3 not counted → **(10+10)/2 = 10.0, not 8.3.** 8.3 = (10+10+5)/3 only works with C3 counted as V1 (5), which §4 rejects.
- Rubric oddity to flag to the founder: a tool whose main claim is V0 would show 10.0. That is what the written arithmetic yields (same effect exists for KiviDB/CLI-Anything), but it reads as misleading next to a V1 label. Suggest showing it as "10.0 on 2 of 3 claims (core claim C3 untested to completion)" or holding the numeric score; either is a policy call for the founder, not for me. If the founder keeps 8.3, it must be justified as an explicit amendment, not by the pre-registered text.

## 6. What I did not / could not do
- **No fresh C3 run.** No `OPENROUTER_API_KEY` or opencode in my environment and I was told not to obtain credentials. Even with Timofy's key, the same free account is at 0/50 until 2026-10-29 08:00 UTC+8 (X-RateLimit-Reset), so a rerun was impossible today. Nothing paid was used anywhere; I never invoked a model.
- Judged C3 purely from logs + the OpenCode sqlite DB + re-validation of the output. I did not independently verify that a different/paid model would finish or would keep findings `needs_validation`.
- The sqlite DB was opened read-only; session text of sub-agents was not exhaustively read beyond tool-call inputs, task statuses and outputs.
- Single-run, single-model, one small target remain valid limits for any verdict (also in Timofy's caveat).

## 7. Per-claim verdicts
| Claim | Timofy | Lena | Call |
|---|---|---|---|
| C1 install (non-core) | V2 | V2 | reproduce |
| C2 validators (core) | V2 | V2 | reproduce |
| C3 six-phase audit (core) | V1 | V0 (partial evidence) | **overturn** |
| Overall | V1 | V1 | reproduce |
| Score | 8.3 | 10.0 (rubric-literal; presentation needs a founder call) | **overturn** |