Root cause of the 45 s timeouts: the local model got the full hosted prompt (~3.1k tokens with examples). Any prompt-cache
miss meant ~40 s of CPU prefill on the single slot, and every request queued behind one paid it too.
- LOCAL_PROMPT/_local_system_prompt: terse action list, one-line examples, default page text, one line per listing
(L-refs, dollar prices). No per-visitor data in it, so the cache always hits; it only changes with the catalog.
- _local_user: look overrides (if any) + keyword/budget MATCHING LISTINGS hint + 'VISITOR: <message>'.
- max_tokens 200 for the local model; over-budget card picks dropped when some fit (_respect_budget).
- Buy vs sell examples separated (sell my own thing -> listing form, never checkout).
- 18/18 correct on typical prompts, warm median 5.9 s, max 7.7 s; cold prefill ~13 s.
- Remove the keyword-rules agent from typed chat: fallback_agent, parse_look_command (+ tables), the copy-edit parser
and _rules_check are gone. If every model fails: 'The assistant is busy, please try again in a moment.' with a Retry
button (agent 'unavailable', no actions). Clicks, forms and payments stay deterministic (fallback_event); the landing
view is a deterministic featured-listings panel; the Sell button opens the empty form.
- Local model via llama.cpp llama-server directly (ops/local-llm-watchdog.sh, loopback only, restart on exit/hang,
crash backoff, log rotation; deploy.sh ensures it runs). Lean: no vision projector, no weight repack (mmapped), ctx
6144, 1 slot, q8 KV, 6 threads at nice 10, poll 0, 1 GiB prompt cache. Idle 0% CPU, 2.9 GB RSS (0.6 GB private).
- Local requests use a JSON GBNF grammar, worked examples in the cached system prompt, and a labelled visitor message.
10/10 correct on the real prompt, warm 4-13 s.
- Primer re-primes after a model-server restart (llama-server /metrics counter) and otherwise stays idle.
- Fix: hosted calls kept the Authorization header off when a local model was configured (cfg['local'] is a dict on
hosted configs); now only the local server itself skips the key. Test added.
- Badge/chat header: 'AI: busy', 'AI: quota used up', 'AI: unavailable'; no 'built-in'/'checked by rules' texts.
- Tests: proxy checks expect the busy reply (never rules), clicks still work while busy, no-model server, local grammar/
prompt/no key; browser checks for busy + Retry; e2e_http accepts busy for typed turns unless REQUIRE_LLM=1;
test_copyedit.py replaced by test_copy_defaults.py; README updated.
