# Ollama test plan (pre-registered before any install/run)

Written and committed 2026-10-23 08:47 (UTC+8; first draft said 08:50 by mistake, corrected to the commit time) by Grok Bot. Box: 8 CPU cores, 15 GB RAM, no GPU, Linux x86_64.
Vendor pages saved 2026-10-23 08:46 in runs/2026-10-23/tools/ollama/vendor/ (checked_on 2026-10-23).

## Vendor claims (verbatim)
- C1 (core) Install: "curl -fsSL https://ollama.com/install.sh | sh" (ollama.com/download/linux, README "Linux"); README "Get started: ollama".
- C2 (core) Run open models locally: "Ollama runs open models in two ways: Locally, on this computer. Speed depends on the hardware." (ollama.com/download/linux); quickstart `ollama run <model>`.
- C3 (core) OpenAI compatibility: "Connect OpenAI clients to Ollama. Ollama supports a subset of the OpenAI API." ... "The client requires an API key value, but Ollama ignores it." (docs.ollama.com/api/openai-compatibility)
- C4 (core) Local privacy: "Ollama runs locally. We don't see your prompts or data when you run locally." (docs.ollama.com/faq)
- C5 (not core) Structured outputs: "Structured outputs let you enforce a JSON schema on model responses" (docs.ollama.com/capabilities/structured-outputs)
- Pricing: Free $0 "Run models locally", "Running models on your own hardware is always unlimited"; cloud models paid/credits (ollama.com/pricing). We test local only; no account, no sign-in.

## Test task and ground truth
Model: a small library model that fits CPU (first choice qwen2.5:1.5b; if not available, llama3.2:1b). Same model for all claims.
- C1: run the one-line installer as written (with sudo available on box). Pass V2: installer exits 0, `ollama -v` prints a version, server answers GET http://127.0.0.1:11434/ with "Ollama is running", no account/sign-in asked. V1: works only after a manual workaround we have to invent. F: cannot install.
- C2: `ollama pull <model>` then 10 known-answer prompts (ground truth fixed below) via `ollama run` / /api/generate, temperature 0. Pass V2: pull succeeds from the default registry and the model answers all 10 prompts locally (answer correctness is the model's property, recorded but scored separately: >=8/10 correct expected for a 1.5B model on these easy prompts; <8/10 noted, does not change Ollama's rating unless output is garbled/empty). V1: pull needs a non-default route (e.g. hf.co/ GGUF) because the default registry fails from our box. F: cannot run a model at all. Speed (tokens/s from eval_count/eval_duration) reported, no pass mark (vendor gives no number).
  Ground truth: 17*23=391; capital of Australia=Canberra; chemical symbol for gold=Au; days in a leap year=366; 2^10=1024; author of "Pride and Prejudice"=Jane Austen; largest planet=Jupiter; H2O common name=water; square root of 144=12; boiling point of water at sea level in Celsius=100.
- C3: official `openai` Python client, base_url http://127.0.0.1:11434/v1, api_key "ollama": (a) chat.completions (non-stream) "17*23" answer contains 391, (b) streaming chat returns >1 chunk and concatenates to text, (c) models.list includes the pulled model, (d) a dummy api_key value "x" works too. Pass V2: 4/4; V1: 2-3/4; F: <=1/4.
- C4: during the 10 C2 prompts, trace the server with strace (connect/sendto) and ss; then repeat 3 prompts with the server running inside a network namespace with no network (unshare -n). Pass V2: 0 outbound non-loopback connections while answering prompts AND prompts work offline. V1: prompts work offline but the server makes other outbound connections (e.g. update/cloud checks) without prompt content we can see. F: prompt text leaves the machine or offline mode fails.
- C5: 10 prompts with format = a JSON schema {name:string, year:integer, country:string} ("Give one famous person born in <year>..."). Pass V2: 10/10 parse as JSON and validate against the schema; V1: 7-9/10; F: <7/10.

## Verdict rule
Overall = worst of core claims C1-C4; any core V0 caps overall at V1. Score = mean of tested claims (V2=10, V1=5, F=0). level L0 (no account, our own machine, synthetic prompts); typical_level L0 for local use.