> Copy of the adversarial retest report (Lena, independent agent, 2026-10-20 UTC+8). Unchanged except that the vendor's embedded PostHog project token is redacted. Her artefacts are summarised in files 20-23.

# CLI-Anything (HKUDS): adversarial retest of the earlier verdict

Retest date: 2026-10-20, about 14:25–15:00 (UTC+8). Box: Linux x86_64, Python 3.13.5. I did not modify any of the earlier team's files. All my work is in `/workspace/retest-cli-anything/`.

## What I tested
- Fresh clone of `https://github.com/HKUDS/CLI-Anything` into `repo/`. HEAD is **34f519533bc175d2fe287ab8316b0dd99bb9cc43**, committed 2026-09-22 10:01:28 +08:00 ("Merge pull request #481 ..."). The earlier test used the same commit (34f5195), so commit drift cannot explain any difference between their results and mine.
- New venv `venv/`, installed from the clone: cli-anything-libreoffice 1.0.1, cli-anything-audacity 1.0.0, plus cli-anything-inkscape and cli-anything-shotcut for spot checks. A second new venv `hubvenv/` has cli-anything-hub 0.4.1 from `repo/cli-hub`. These versions match what the earlier team got from PyPI.
- Backends on the box: LibreOffice 25.2.3.2, SoX 14.4.2, ffmpeg 7.1.5. I also ran `apt install inkscape melt file`, which gave Inkscape 1.4 and melt 7.30.0. I pip-installed pydub and audioop-lts (see finding 3).

## Finding 1: "drives real software, zero toy implementations" is false on 2 CLIs. **CONFIRMED for those 2 CLIs. The failures did not carry over to my 2 spot checks.**
- README.md:900 says: "LibreOffice generates PDFs ... Audacity processes audio via sox. Zero compromises, Zero toy implementations."
- The Audacity CLI never runs sox during any user command (finding 3). That alone falsifies an absolute claim like "zero toy implementations".
- The LibreOffice CLI does call real soffice for PDF/DOCX/XLSX/CSV. But its own ODF writer, which produces the input soffice converts, breaks every formula (finding 2).
- Spot checks (below): Inkscape and Shotcut produced correct output. Shotcut rendered through real `melt`. Inkscape used real `inkscape` for PDF, but its PNG export is a Pillow re-drawing that does not use Inkscape even when Inkscape is installed. So the CLIs are mixed: some are real wrappers, some are partial re-implementations.

## Finding 2: LibreOffice CLI (headings OK, table cells 0/12, `=SUM(A1:A3)` gives Err:510). **CONFIRMED. I found the root cause, and no correct-usage path fixes it.**
Commands and outputs are in `taskA/`.
- Headings and paragraphs: 4/4 reproduced.
- **Table cells**
  - `writer add-table --help` only offers `--rows/--cols/--position`. `--data x` gives `Error: No such option '--data'`.
  - `writer set-text 4 Site` gives `{"error": "Cannot set text on content type 'table'"}` with exit 1 (core/writer.py:215).
  - SKILL.md, README.md, LIBREOFFICE.md and REPL `help` list no cell-text command. A PDF built only with CLI commands (`r.pdf`, md5 f8795bfa...) contains 0 of the 12 cell strings.
  - Nuance: `core/writer.py:99-116` `add_table()` does accept `data=`, and the renderer outputs cell text. That data is only reachable in two ways. (a) `document import` of an existing ODT that already has a table: `imp.pdf` showed all 12 cells. (b) Hand-editing the project JSON `data` field: `r_edit.pdf` showed all 12 cells. Neither path lets the CLI fill a table. The earlier test did not misuse anything here.
- **SUM formula**
  - Reproduced: A4 shows `Err:510` in PDF, in XLSX (converted back to CSV), in ODS and in CSV export. Inside the XLSX: `<c r="A4" t="e"><f>of:=SUM(A1:A3)</f><v>#VALUE!</v></c>`.
  - I looked for a "correct syntax" that would work. All of these also fail in the CLI's export: `=SUM([.A1:.A3])` → Err:510, `SUM([.A1:.A3])` → Err:509, `=[.A1]+[.A2]` → Err:510, and even `=1+2` → Err:510. With `--type float` and with `--type string`, the result is the same. There is no recalc flag: `export render` only has `-p/--overwrite`.
  - Round trip: I made an ODS in LibreOffice with a working `=SUM()` and imported it with `document import`. `calc get-cell` reported `value 100.0`, but re-exporting gave Err:510. So the CLI also breaks formulas in existing files it imports.
  - **Root cause**: `utils/odf_utils.py:345-346` writes `table:formula="of:..."`, but the namespace map at `utils/odf_utils.py:23-36` never declares the `of` prefix (`urn:oasis:names:tc:opendocument:xmlns:of:1.2`). I copied the CLI's ODS and added only that one xmlns declaration. LibreOffice then computed A4=100, B1=100, B2=100, B3=42 and B4=3 (`taskA/diag/fixed.pdf`). This is a CLI bug, not user error. The unit tests (`tests/test_core.py:196-208, 474-477`) check only the XML string, never a computed value.

## Finding 3: Audacity CLI (no sox/Audacity; MP3/FLAC/OGG are renamed WAVs). **CONFIRMED, including with the documented optional dependency installed.**
Files are in `taskB/`.
- The prior task gives a correct WAV: 3.0 s, 440 Hz then 880 Hz. Not disputed.
- Possible misuse I checked: AUDACITY.md:37 and :124-127, and the preset descriptions, say MP3/FLAC/OGG/AIFF "requires pydub/ffmpeg". The earlier team did not install pydub. I installed pydub and audioop-lts; ffmpeg and sox were present.
- Result is unchanged:
  - `md5sum`: out.wav, out.mp3, out.flac, out.ogg and out.aiff are all `1c4267428d15ab5c360855ffa0e35495`. That is byte-identical to the earlier team's hash.
  - `file`: every one of them is `RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 44100 Hz`.
  - `ffprobe` reports `format_name=wav` for all of them.
  - The CLI's JSON still says `"format": "MP3"`, `"FLAC"`, `"OGG"` and `"AIFF"`.
- Why: `core/export.py:204-209` calls `write_wav()` in both the WAV branch and the non-WAV branch. `grep -rn pydub` in the package finds only description strings (export.py:69/75/81/87) and no import. The docs' advice to install pydub therefore cannot work.
- `strace -f -e execve` showed 0 non-Python executions for every preset, and for an export with echo, change_pitch and low_pass effects applied.
- The only subprocess code in the package is `utils/sox_backend.py`, and it is imported only by `tests/test_full_e2e.py:784-858`. Nothing references an Audacity binary.
- The module header (export.py:6) says "Uses ONLY Python stdlib". The README tools table (README.md:1097) says "Python wave + sox". That is partly honest, but it contradicts README.md:900.

## Finding 4: cli-hub sends PostHog analytics by default, including agent detection; README doesn't mention it; CLI_HUB_NO_ANALYTICS=1 disables it. **MIXED.** The behaviour and the opt-out are confirmed. "README doesn't mention it" is only true of the top-level repo README.
- **Source code**
  - `cli_hub/analytics.py:16-20`: provider `posthog`, host `https://us.i.posthog.com`, embedded project token `phc_[redacted]`.
  - `:84-85`: `_is_enabled()` is false only when `CLI_HUB_NO_ANALYTICS` is 1/true/yes.
  - `:29-52`: agent detection uses env vars (CLAUDE_CODE, CURSOR_TRACE_ID, ...) and regexes over parent-process command lines.
  - `:160-180`: anything with a non-TTY stdin is classed as `is_agent: true`, `scripted_client`.
  - `cli.py:105-106`: every invocation calls `track_first_run()` and `track_visit()`.
  - A random distinct_id is stored in `~/.cli-hub/.analytics_id`.
- **Payload**: I captured it by pointing `CLI_HUB_POSTHOG_API_HOST` at a local sink, with an isolated HOME:
  `{"event":"cli-hub call","distinct_id":"<uuid>","properties":{"hostname":"clianything.cc","hub_version":"0.4.1","command":"search","is_agent":true,"traffic_type":"agent","agent_category":"scripted_client","agent_reason":"stdin-not-tty","agent_signals":["stdin-not-tty"],"stdin_tty":false,"is_interactive":false,"platform":"linux"}}`
  I saw no paths, usernames or IP-derived fields in the payload. The server will of course see the IP address.
- **Network**:
  - `strace -e connect,sendto,sendmmsg cli-hub search office`, default settings: 4 connects, including a DNS query for `us.i.posthog.com`.
  - With `CLI_HUB_NO_ANALYTICS=1`: 0 connects, and 0 events captured by the local sink.
- **Disclosure, where the earlier claim was too broad**:
  - The top-level `README.md` has no analytics/telemetry/PostHog text. Confirmed.
  - But `cli-hub/README.md:107-115` has an "## Analytics" section naming PostHog and the opt-out. `setup.py:55` uses it as the PyPI long_description. The PyPI JSON for 0.4.1 contains that section, so the pip install page discloses it.
  - The hub docs page (`docs/hub/docs.html:568-569`, live at https://clianything.cc/docs.html and hkuds.github.io/CLI-Anything/docs.html, both HTTP 200) states the opt-out and says it "auto-detects when it's invoked by an AI agent ... for analytics bucketing".
  - What really is undisclosed: the parent-process inspection, and the fact that any non-TTY use is labelled "agent".
- **Side effect of my test**: my default-mode strace run sent one real `cli-hub call` event to PostHog, using a fresh random id from an isolated HOME. An earlier first-run/call event went to a pre-existing local listener on 127.0.0.1:8765, not to the internet.

## Misuse check of the earlier test
- LibreOffice: no misuse found. Their commands match `--help`. Using `--type float` with the formula, as they did in their extra check, does not help. No recalc option exists.
- Audacity: there is one possible omission. They did not install pydub, which the docs name as the requirement for MP3/FLAC/OGG. I installed it and the result was identical, so this does not change the finding.
- Hub: their statement "README does not mention it" is inaccurate for the cli-hub package README / PyPI page and the hub docs page. It is accurate only for the repo's top-level README.
- My own near-misuse, which I fixed: Inkscape one-shot commands do **not** persist unless you pass `-s/--save`. Without it, `shape add-rect` exits 0, prints the rect, and then discards it, giving a blank PNG. With `-s` (documented in `--help`) it works.

## Spot checks (more CLIs, real backends)
1. **Inkscape CLI** (`spot-inkscape/`).
   - Task: 400×300 white canvas with a red rect at 20,20 sized 150×100, a blue circle at (300,80) with r=50, and the text "Spot 9Q2". All commands used `-s`.
   - PNG: rect pixel (255,0,0), circle (0,0,255), 69 dark text pixels. strace shows **no Inkscape execution**, so the PNG is drawn by Pillow (`core/export.py:83-98`).
   - PDF: real `/usr/bin/inkscape` exec, JSON says `"rendered_by":"inkscape"`. pdftotext gives "Spot 9Q2", and rasterised colours are correct.
   - Side effect: `export pdf` writes `out.svg` next to the output.
   - Verdict: correct output, half real backend.
2. **Shotcut CLI** (`spot-shotcut/`).
   - Task: a 2 s red clip and a 2 s blue clip with tones, made with ffmpeg, placed back to back, then `export render --preset default`.
   - strace: 1 `/usr/bin/melt` exec. JSON says `"method":"melt"`.
   - Output: h264 1920×1080 plus aac, 4.074 s (inputs total 4.004 s). Frame at 1 s is RGB (253,0,0) and frame at 3 s is (0,1,251).
   - Verdict: correct, real backend.

So the "toy" failures are real for Audacity and for LibreOffice formulas, but they do not apply to every CLI. 2 of my 2 spot checks gave correct output, with real backends for the main export path.

## Overall view on F / 4.0
- Findings 2 and 3 stand completely, and I can now explain them from the code: a missing `of` namespace, and a WAV-for-everything export path. The pydub advice cannot help because pydub is never imported.
- Under the earlier team's pre-registered rule ("F if ... the backend is faked"), Audacity alone forces F on C2. That rule is applied correctly, and the README's absolute "zero toy implementations" claim is clearly false.
- Two caveats affect fairness:
  1. The verdict generalises from 2 of 103 CLIs. My two extra CLIs, Inkscape and Shotcut, worked and mostly used the real backends. So "toy implementations" describes some harnesses, not the whole project.
  2. Finding 4's "not mentioned in the README" should be narrowed to "not in the main repo README". It is disclosed on the PyPI page and the hub docs, minus the parent-process detail.
- My view: **F on the "real software / zero toy implementations" claim is fair. The overall 4.0/10 is defensible under their rubric but somewhat harsh as a project-wide score.** I would keep F and ask for the text to say it is based on 2 harnesses, mention that Inkscape and Shotcut passed spot checks, and correct the analytics-disclosure wording. If they re-score, about 4.5–5 would better reflect the mixed CLI quality. That is a judgement call, not a measurement.

## Key artefacts
- `taskA/`: `sheet.{pdf,xlsx,ods,csv}` (md5 064c27b3..., 6fe7b387..., 6dca9073..., 116460d7...), `r.pdf`, `imp.pdf`, `r_edit.pdf`, `diag/fixed.pdf`, strace files `st-sheet-*.txt`
- `taskB/`: `out.{wav,mp3,flac,ogg,aiff}`, `json-*.txt`, `strace-*.txt`
- `hubtest/`: `st-default.txt`, `st-optout.txt`, `sink.py`, `home/.cli-hub/`
- `spot-inkscape/`, `spot-shotcut/`: outputs, JSON and strace files