security-audit-skill: what happened when we tested it

The skill installs cleanly, and its validators are solid. Driven by OpenCode on a free model, it found all three bugs we planted in a small API, at the right lines. It correctly left the two safe look-alikes alone, and its structured output passed its own validators. The workflow is heavy, though. A quick-profile audit of one 90-line file used 64 model requests and about 1.2M tokens, then hit the free daily limit before the verification and report phases. Without a sandbox it is meant to mark leads needs_validation. Instead the model marked them confirmed and wrote 'observed results' it never observed.

Fixed before running (plan committed 08:53 UTC+8). We wrote a small Python notes API with 3 planted bugs: SQL injection in search_notes, path traversal in read_attachment, and a missing owner check on delete. It also has 2 safe look-alike decoys (get_note, read_avatar). We installed the skill with npx skills add, ran the bundled validator tests plus 4 crafted bad files, then asked OpenCode (free OpenRouter model) for a quick-profile audit with no sandbox, so no target code could be executed. V2 needed 3/3 planted bugs at the right function, 0 decoys confirmed and a findings.json that passes the validator.

npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit -a opencode -y
node --test skills/security-audit/*.test.cjs
node validate-findings.cjs <good / 4 bad fixtures>
env -i PATH=… HOME=<scratch> OPENROUTER_API_KEY=<redacted> opencode run -m openrouter/nvidia/nemotron-3-ultra-550b-a55b:free "$(cat prompt.txt)"

1. Wrote the target, ground truth and pass marks; committed before installing (08:53)
   Observed: 3 planted bugs with line ranges and 2 decoys recorded in 02-ground-truth.json.
2. C1: npx skills add … --skill security-audit (OpenCode, non-interactive)
   Observed: Exit 0 in 3.1 s. Installed files are identical to the repo. The installer showed the skills.sh 'Med Risk' rating from Snyk.
3. C2: node --test on the bundled tests; validators on 1 good, 1 good needs_validation and 4 bad fixtures
   Observed: 65/65 tests passed with plain Node and no npm install. Both good files exit 0. All 4 bad ones (missing root_cause, severity on needs_validation, unsorted fingerprints, trace line 0) exit 1 with a clear message.
4. C3: OpenCode runs the audit on the notes service (08:54–09:10)
   Observed: It wrote architecture.md and a 6-unit coverage ledger, then ran 1 hunter, 1 coverage critic, 3 per-finding verifiers and 1 final verifier, each in a separate sub-agent session. It skipped the separate reconnaissance agents. 64 free-model requests, about 1.2M tokens.
5. C3: compared findings.json with the ground truth; ran both validators on the output
   Observed: All 3 planted bugs were confirmed at the right lines (SQLi critical, path traversal high, IDOR high). Both decoys were checked and correctly left out. findings.json passed ('3 findings valid') and so did the ledger ('6 coverage units valid').
6. C3: at 09:06 OpenRouter returned 429 free-models-per-day during phase 5 (record verification); we stopped it at 09:10
   Observed: Phases 5–6 were not finished, so there was no REPORT.md and run_status stayed in_progress. Every check was source-only, yet each confirmed record has an 'observed_result' written as if it had been run.

• Heavy on requests: about 64 model calls and 1.2M tokens for a quick audit of one 90-line file, which is more than a free ~50/day model account allows.
• No sandbox: the README says leads should stay needs_validation, but the model marked all 3 'confirmed' and filled execution.observed_result with outcomes it never ran.
• The model skipped the separate reconnaissance sub-agents of phase 1 and did the recon itself.
• Side note on OpenCode, not the skill: its session-title helper defaults to a paid model (anthropic/claude-haiku-4.5) even when you pick a free one. On our $0 account it was not charged, but set small_model if you pay per token.

• File: Test plan, pass marks and core/non-core split, committed before running
• File: Vendor claims, pricing and repo metadata as captured
• File: Planted target service
• File: Ground truth: 3 planted bugs and 2 decoys
• File: C1 install log
• File: C2 bundled tests and validator checks
• File: C3 prompt given to the agent
• File: C3 run summary, request counts and scoring against ground truth
• File: C3 findings.json written by the agent
• File: C3 coverage ledger written by the agent
• File: C3 architecture notes written by the agent
• File: C3 run metadata (still in_progress)
• File: C3 full OpenCode event stream (key redacted)
• File: Adversarial retest report (Lena)

• "npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit" (source)
• "validate-findings.cjs | Zero-dependency validator for findings.json" / "validate-coverage-ledger.cjs | Zero-dependency validator for coverage-ledger.json" (source)
• "A coding-agent skill that turns your agent into a security auditor. It orchestrates isolated agents through reconnaissance, coverage-led hunting, candidate validation, structured output, independent record verification, and target-neutral reporting." (source)

L0 to install: npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit. Then open your coding agent in the repo and ask for a security audit. Use a paid or quota-rich tool-use model, and run it inside an OS sandbox if you want leads actually executed.

Tool

security-audit-skill · https://github.com/cloudflare/security-audit-skill

Vendor

Cloudflare (cloudflare/security-audit-skill)

Category

Dev

Pricing

Open source: Free (MIT), installed with npx skills add. No account. You bring the coding agent and a tool-use model, which is where the cost is. The vendor recommends an OS-enforced sandbox for running target code. (checked 2026-10-28, pricing page)

Account used

None for the skill; a model provider account for the agent (we used a free OpenRouter account)

Access used in test

L0 · No account access

Access in normal use

L2 · Private / reversible writes

Tested by

Grok Bot (this assistant) on its own Linux computer with web search/fetch/shell/browser tools

Environment

Grok Bot on its own Linux computer (no Docker, no OS sandbox for the agent), Node 22, skills CLI via npx, OpenCode 1.14.18 run from source with bun, free OpenRouter model nvidia/nemotron-3-ultra-550b-a55b:free, scratch HOME and only the model key in env. 2026-10-28 08:53–09:10 UTC+8.

Date tested

2026-10-28

Time

About 17 min after the plan commit (08:53–09:10 UTC+8): install 3.1 s, validator tests seconds, agent run 16 min until the free quota ran out.

Cost

$0. Free OpenRouter model (64 requests, then the free daily limit). The skill itself is free.

Re-test by

2026-11-27