# Playwright MCP test plan (written 2026-10-26 before installing or running anything, UTC+8)

Tool: Playwright MCP (microsoft/playwright-mcp), npm `@playwright/mcp@latest` (0.0.83 per `npm view` at 08:45).
Vendor claims captured from the GitHub README on 2026-10-26 (runs/2026-10-26/tools/playwright-mcp/README.md).

## Task and ground truth
Our own local test page (site/index.html + server.py, served on 127.0.0.1) with an order form and a decoy
newsletter form. Both forms have a button labelled exactly "Submit". The "Part number" field appears 1.5 s after load.
Fill the ORDER form with exactly these values and submit it, then read back the confirmation code:

| field | value |
|---|---|
| name | Mariana Okafor-Lindqvist |
| email | m.okafor@example.org |
| country | JP (Japan) |
| qty | 7 |
| speed | express |
| gift | yes (checked) |
| notes | Leave at reception, ask for Kenji. |
| part | AX-4471-B |

Ground truth: the server logs exactly one POST to /order with those 8 fields and none to /newsletter; the page shows
confirmation code BA36DB1F (sha256 of the sorted fields, computed before running).

Client: the official MCP TypeScript SDK client over stdio, launching the server with the vendor's standard config
(`npx @playwright/mcp@latest`) plus `--headless` (our box runs agents headless) and `--isolated`.
No screenshots are used anywhere in C2/C3: only browser_snapshot text and ref-based tools.

## Core claims and pass marks
- C1 "Standard config works in most of the tools" / Node 18+: `npx @playwright/mcp@latest` starts, an MCP client
  lists tools and browser_navigate loads our page with no other setup = V2; works only after an undocumented extra
  step = V1 (documented steps such as installing a browser are fine and noted); does not run = F.
- C2 "enables LLMs to interact with web pages through structured accessibility snapshots, bypassing the need for
  screenshots" / "No vision models needed":
  (a) scripted run: our script reads refs from the snapshot text by label and fills the form; all 8 fields correct.
  (b) LLM run: a free text-only model on OpenRouter ($0 account) gets the task, the tool list and only the text
      results, max 25 turns. V2 = (a) 8/8 and (b) 8/8 fields + correct code reported; V1 = (a) 8/8 and (b) 6-7/8
      or wrong/missing code; F = (a) fails, or (b) below 6/8 because the snapshot lacked needed information.
- C3 "Deterministic tool application. Avoids ambiguity": 3 fresh sessions of the scripted run. V2 = initial snapshot
  text identical in 3/3 (refs identical), the order form's Submit clicked (0 POSTs to /newsletter) and identical
  payloads in 3/3; V1 = actions correct 3/3 but snapshots/refs differ; F = a wrong element is acted on in any run.
- C4 "Fast and lightweight": median latency of browser_type/click/select_option/check calls measured in the client
  over the 3 scripted runs: <= 500 ms = V2, <= 2 s = V1, > 2 s = F. Reported but not scored: snapshot size in
  characters/approx tokens for our page and one real page (en.wikipedia.org/wiki/Playwright_(software)) vs PNG
  screenshot bytes; server cold-start time.

Verdict rule (shared rule): overall = worst core claim; V2 only if every core claim is V2; any core V0 caps at V1.
Score = mean of tested claims (V2=10, V1=5, F=0).
Level: L0 for this test (local page, no account); typical_level L2 (real use drives logged-in sites as the user).