# CLI-Anything test plan (fixed BEFORE installing or running anything from CLI-Anything)
Written: 2026-10-20 14:09:34 UTC+8. Grok Bot box: Linux x86_64, 8 vCPU, 15 GB RAM, no GPU, no Docker, Python 3.13.5, LibreOffice 25.2.3.2 (already installed from Debian), ffmpeg and poppler-utils present. No account, no key, no LLM for C1-C5.
Versions under test: whatever PyPI / the repo serve today (PyPI cli-anything-hub 0.4.1 at 14:08; repo HKUDS/CLI-Anything main, last push 2026-09-22 10:01 UTC+8, 55,397 stars, Apache-2.0; the hub package itself says MIT on PyPI).
Each CLI goes into its own fresh venv under runs/2026-10-20/tools/cli-anything/. Every command, exit code, timing and output is logged.

## Vendor claims (README https://github.com/HKUDS/CLI-Anything and CLI-Hub https://clianything.cc/ = https://hkuds.github.io/CLI-Anything/, fetched 2026-10-20 14:08-14:09 UTC+8)
C1 Hub: "`pip install cli-anything-hub` then `cli-hub install <name>` — browse, install, and manage all community-built CLIs." Commands listed: list, search, info, install, update, uninstall, launch.
C2 Real software: "Direct integration with actual software backends (Blender, LibreOffice, FFmpeg) — full professional capabilities, zero compromises" / "Direct calls to real applications for actual rendering. LibreOffice generates PDFs ... Audacity processes audio via sox. Zero compromises, Zero toy implementations."
C3 JSON: "Agent-Native Design — Built-in --json flag on every command delivers structured data for machine consumption" (also "Structured JSON output eliminates parsing complexity").
C4 State: "Persistent project state with undo/redo capabilities, plus unified REPL interface".
C5 Tests: "100% pass rate across all 2,461 tests"; LibreOffice harness listed as "158 passed (89 unit + 69 e2e)".
C6 Generation: "One Claude plugin auto-generates CLIs for ANY codebase through proven 7-phase pipeline".
Pricing: free, open source (Apache-2.0 repo). No paid tier found on the README or hub page.

## Ground truth, method and pass marks
C1: fresh venv, `pip install cli-anything-hub` (the vendor's exact command). Run cli-hub --help, list, search office, search audio, info libreoffice, install libreoffice, launch libreoffice --help, update libreoffice, uninstall + reinstall libreoffice, install audacity. Ground truth for the catalogue: the repo's registry.json + public_registry.json at main today (entry count; libreoffice and audacity must be present; search "office" must return libreoffice). Pass: V2 if every listed hub command exits 0 and does what its help says, and the installed CLIs run; V1 if install works but some hub commands fail or the list disagrees with the registry; F if installing a CLI through the hub fails.
C2: Task A (LibreOffice CLI): create a Writer document with a heading containing the unique token "Field Report 7F3A", 3 fixed paragraphs (sentences written below in 02-task-inputs.txt), and a 3x4 table with 12 fixed cell strings; export to PDF, DOCX and ODT. Also a Calc sheet with values 17, 25, 58 in A1:A3 and =SUM(A1:A3) in A4 exported to PDF and XLSX (ground truth 100). Check: pdftotext of the PDF contains all 16 text items (heading, 3 paragraphs, 12 cells) and the Calc PDF shows 100; DOCX/ODT/XLSX converted back to text by our own soffice call contain the same items; a process trace (strace -f -e execve) shows the CLI really launched soffice for PDF/DOCX/XLSX. Task B (Audacity CLI, sox backend; sox apt-installed from Debian if the CLI needs it): our own WAVs, 1.000 s 440 Hz and 2.000 s 880 Hz sine, 44.1 kHz mono 16-bit, placed back to back in one project and exported to WAV. Ground truth: 3.000 s +/-10 ms, dominant frequency 440 Hz in 0-1 s and 880 Hz in 1-3 s (+/-5 Hz, FFT). Pass: V2 if both tasks give correct output from the real backend; V1 if one task or some formats fail/are wrong; F if neither produces correct output or the backend is faked.
C3: every command we run in Tasks A/B and C4 is also run with --json (plus 3 deliberate errors per CLI: missing project file, bad option value, unknown element index). Count commands whose stdout is exactly one parseable JSON value and whose errors come back as JSON with a non-zero exit code. Pass: V2 if >= 95% of commands, V1 if 70-95%, F below 70%.
C4: LibreOffice CLI: add 3 paragraphs in separate one-shot invocations with --project, undo once, check element count 2, redo, check 3; then the same inside one REPL session (commands piped to stdin). Pass: V2 if undo/redo works across one-shot invocations and in the REPL; V1 if only in the REPL (or only one-shot); F if neither.
C5: clone the repo at main and run the LibreOffice harness tests with pytest on this box (real LibreOffice present). Pass: V2 if the suite collects 158 tests and all pass; V1 if >= 90% pass or the count differs; F below 90%. This checks one harness of the 2,461 tests, not all.
C6: needs a coding agent (Claude Code, Codex, Cursor, OpenCode, ...) with an LLM for a 7-phase build of many steps. Try only if a free path exists. Known budget: OpenRouter free account, about 50 free requests/day, about 48 already used today (Headroom rerun). Expected V0 (not testable today) unless a free agent path with enough requests turns up; reason recorded.

Overall verdict = worst of C1-C4 (using ready-made CLIs from the hub, the path the README calls "use the existing CLI ecosystem now"). C5 is rated and counts in the score but does not set the verdict; C6 is V0 if untested (left out of the score).
Access level: L0 (local software, our own synthetic files). Typical level: L1 (the CLIs read and write the user's own documents/projects).
Budget: about 30 minutes of testing.