How we test
Pilot note: every case in this pilot was run with Grok Bot only (Grok Bot on its own Linux computer with web search, fetch, shell and browser tools). We never label a case as reproduced on an agent we did not run. Multi-agent side-by-side comparisons start once we have subscriptions or test accounts for other agents.
The loop
- Find a popular agent claim on X and save the link.
- Re-run it with the exact prompt we publish, in a clean session.
- Record every site and API touched, the time, and every failure.
- Check the result against ground truth (the official source page, raw data, or a programmatic check) and say which.
- Rate status and access level, then set a re-verify date.
Verification status
| Status | Meaning |
|---|---|
| V2 Reproduced | We re-ran it and got the claimed kind of result, checked against ground truth. |
| V1 Partly reproduced | Part of the claim worked; the rest was not tested or did not work. The case says which. |
| F Failed | We tried it and the result was wrong or unusable. |
| V0 Sourced only, not yet verified | Not run yet. Listed because it is popular; the case explains what we would need to test it. |
Every status expires: re-verify within 30 days, or sooner when the agent version changes.
Access levels
| Level | Meaning | In plain words |
|---|---|---|
| L0 | No account access | Needs no logins. The agent only reads public web pages and open data. |
| L1 | Reads your private data | Reads private data such as your email, bank or seller account, but changes nothing. |
| L2 | Private / reversible writes | Makes private changes you can undo, like drafts, labels or alerts only you see. |
| L3 | Acts as you, visible to others | Acts as you where other people can see it: sends messages, calls a business, files a claim. |
| L4 | Spends, deletes or escalates | Spends money, deletes things or escalates, for example buying a ticket. |
We rate the highest level we actually exercised, and separately show the level the original poster's setup used. We also flag: unattended runs, third-party skills/MCP, credential type, data sent to third parties, and whether an approval gate was observed.
Topics
- Travel & trips: Flights, hotels, weather windows and trip planning.
- Money & bills: Share and crypto prices, flight-delay compensation, card perks and subscriptions.
- Home & daily life: Meal plans, menus, morning briefs and everyday errands.
- Research & data: Answering questions from fresh sources, ranking accounts and analysing business data.
- Shopping & prices: Comparing what one product costs across retailers.
Safety rules for the pilot
- L0 only: public data, no logins, no messages sent, no purchases, no money spent.
- Anything at L3 or above needs founder approval before it runs or publishes.
- We never invent numbers. If a source blocked us, the case says so.
Citing and reusing our data
Each case page ends with a ready-made citation. Machine-readable data: /cases.json, /llms.txt, /llms-full.txt and the RSS feed. Licence: CC BY 4.0: free to quote and reuse with attribution to triedagents and a link to the case page.
Contact
Corrections, case suggestions and sponsor questions: hello@triedagents.com.
Sponsorship
Sponsors never see or change a verdict before it is published, and sponsored products are tested with the same rubric. See our sponsorship disclosure and editorial policy.
Get one verified case every weekday
One agent use case we actually re-ran, every weekday. Free.
We only store your email address. Every issue has a one-click unsubscribe link. Privacy