Local model: compact cached prompt so typed replies take ~6 s (fixes 45 s timeouts) #8

Merged
leora merged 2 commits from fix/local-fast-prompt into main 2026-10-08 22:43:56 +08:00

2 Commits

Author SHA1 Message Date
c8e1c162ee e2e: agent API search with a higher limit (staging accumulates test listings) 2026-10-08 22:43:54 +08:00
70f5b0525e Local model: compact prompt (1.2k tokens, same for every visitor) so typed replies take ~6 s instead of timing out
Root cause of the 45 s timeouts: the local model got the full hosted prompt (~3.1k tokens with examples). Any prompt-cache
miss meant ~40 s of CPU prefill on the single slot, and every request queued behind one paid it too.

- LOCAL_PROMPT/_local_system_prompt: terse action list, one-line examples, default page text, one line per listing
  (L-refs, dollar prices). No per-visitor data in it, so the cache always hits; it only changes with the catalog.
- _local_user: look overrides (if any) + keyword/budget MATCHING LISTINGS hint + 'VISITOR: <message>'.
- max_tokens 200 for the local model; over-budget card picks dropped when some fit (_respect_budget).
- Buy vs sell examples separated (sell my own thing -> listing form, never checkout).
- 18/18 correct on typical prompts, warm median 5.9 s, max 7.7 s; cold prefill ~13 s.
2026-10-08 22:37:57 +08:00