# Graphify test plan (pre-registered before install/run)
Written 2026-10-21 14:02 (UTC+8; committed dc44cd4 at 14:03:03, before install; an earlier draft of this line said 14:12 by mistake) by Grok Bot. Tool: Graphify (Graphify-Labs/graphify, PyPI `graphifyy`, latest on PyPI at plan time 0.9.93, Apache-2.0).
Environment: Grok Bot's Linux box (8 vCPU, Python 3.13.5, uv), no LLM API keys set, no account. Install exactly as the README says: `uv tool install graphifyy`.
Scope: the free, local code-graph path (CLI). Not in scope: Graphify Cloud (14-day trial, "80% fewer tokens"), the semantic pass over docs/PDFs/media (needs an LLM key), and the `/graphify` skill inside a paid AI assistant.

## Test corpus and ground truth (written by us, files in 02-corpus/)
A small 2-language project we wrote today: Python package `shop/` (7 files) + TypeScript `web/` (3 files). It runs (`checkout([("a",10),("b",5)])` = 16.2).
Ground truth cross-file relations (16):
- imports (7): tax.py->util.py (round_money); cart.py->models.py (make_item); pricing.py->tax.py (apply_tax); checkout.py->cart.py (Cart); checkout.py->pricing.py (final_price); api.ts->format.ts (formatPrice); api.ts->user.ts (AdminUser)
- calls (5): apply_tax->round_money; Cart.add->make_item; final_price->apply_tax; checkout->final_price; renderTotal->formatPrice
- instantiation (2, any calls/uses/references label accepted): checkout->Cart; makeAdmin->AdminUser
- inherits (2): DiscountedItem->Item; AdminUser->User
Optional (not counted either way, type-dependent): final_price->Cart.subtotal, checkout->Cart.add.
Decoys: two unrelated functions both named `helper` (shop/util.py, shop/models.py); nothing calls them.
An edge counts if the graph has an edge between the two nodes (either the symbol node or, for imports, the file/module node) with a matching relation. False edge = any calls/inherits/uses edge between two of our named symbols that is not in the ground truth or optional list (any such edge touching `helper` is false).

## Core claims and pass marks (decided now)
C1 (core) "Code is parsed with tree-sitter AST: deterministic, no LLM, nothing leaves your machine" / "graphify extract runs fully offline" / "No telemetry" — local & offline part.
  Run the vendor's headless code-graph command on the corpus with no API keys under `strace -f` (connect/sendto/sendmsg), and once more in a network namespace with no network (`unshare -rn`).
  V2: graph built in both runs, 0 non-local network attempts, no LLM backend used. V1: graph built offline but it tries non-local connections (e.g. version check) that carry no code. F: needs network or a key to build a code graph, or sends code/paths off the box.
C2 (core) "deterministic".
  3 builds on 3 fresh copies of the corpus; compare node set and edge set (ids, relations, confidence tags) ignoring timestamps/version/absolute paths.
  V2: identical. V1: only non-structural fields differ (community numbers/labels, ordering). F: node or edge set differs.
C3 (core) "Cross-file links: calls / imports / inherits resolved across ~40 languages via tree-sitter AST" + "Every edge is explained (EXTRACTED / INFERRED)".
  V2: >= 15/16 ground-truth relations found, 0 false edges, every edge carries a confidence tag. V1: >= 12/16 and <= 2 false edges. F: below that.
C4 (core) "Query, path, explain ... against graph.json"; "A name that matches several nodes is refused with the candidates listed rather than guessed. ... exit code 0 an answer, 3 an ambiguous name, 4 no matching node."
  Q1 `graphify path "checkout" "round_money"` returns a path whose every hop is a real relation (expected checkout->final_price->apply_tax->round_money).
  Q2 `graphify explain "AdminUser"` shows the inheritance from User and the link from api.ts/makeAdmin.
  Q3 `graphify explain "helper"` exits 3 and lists both candidates.
  Q4 `graphify explain "nonexistent_zzz"` exits 4.
  Q5 `graphify query "how is tax applied to the cart total?"` returns a subgraph containing apply_tax and final_price.
  V2: 5/5. V1: 3-4/5. F: <= 2/5.
C5 (not core, V0 expected) README benchmarks (LOCOMO recall@10 0.497 / QA 45.3%, LongMemEval-S 76%, ERPNext 82.0% vs grep 70.8%): need an LLM answerer + judge and the vendor harness; not run in this test.
Extra, not scored: build time and size on a real open-source codebase; whether queries are logged locally (README discloses ~/.cache/graphify-queries.log).

Overall verdict = worst core claim (C1-C4); any V0 core claim caps at V1. Score = mean of tested claims (V2=10, V1=5, F=0).