Local model: compact cached prompt so typed replies take ~6 s (fixes 45 s timeouts) #8

leora · 2026-10-08T22:43:55+08:00

leora commented

2026-10-08 22:43:55 +08:00

Why

The live demo's typed chat timed out (busy reply after 45 s) while the OpenRouter daily quota was used up.

Root cause: the local model got the full hosted prompt, about 3.1k tokens (schema + examples + catalog). On this CPU, prefill costs about 13 ms per token. So every prompt-cache miss cost about 40 s on the single llama-server slot, and any request queued behind it also waited.

Cache misses were frequent:

my own debug and benchmark runs, each with a fresh store and a different prefix (they were running at 22:28–22:30);
demo and staging priming different prefixes;
per-visitor look and page text sitting inside the system prompt.

Fix

Compact local prompt: LOCAL_PROMPT + _local_system_prompt is about 1.2k tokens. It contains:
- a terse action list and one-line examples;
- the default page text;
- one line per listing, with L-refs and dollar prices.
It holds no per-visitor data, so it's identical for every visitor and the cache always hits. It only changes when the catalog changes, and the primer re-warms it then.
Per-visitor part (_local_user, a few dozen tokens): the look overrides (if any), a keyword/budget MATCHING LISTINGS hint, and VISITOR: <message>.
Lean generation: max_tokens 200, no history, no tool round. Over-budget card picks are dropped when some picks fit.
Buy vs sell: the examples are separated (selling your own thing opens the listing form, never a checkout). This replaces the unfinished fix/local-sell-vs-checkout change.

Measured (gemma3:4b, 8-core CPU, -t 6, nice 10)

Demo catalog, 12 typical prompts, real handle_chat: 12/12 correct, median 5.9 s, max 6.7 s.
- logo phrase 4.5 s
- search 6.0 s
- sell form 6.7 s
- gradient 6.2 s
- compare 6.3 s
- checkout 4.1 s
- hide 3.8 s
- reset 4.0 s
- light 5.9 s
- garden 5.1 s
- cash 4.4 s
- kid/budget 6.4 s
With a custom visitor look: 6/6, median 5.8 s, max 7.7 s.
Cold prefill of the new prompt: about 13 s, once per catalog change.

gemma3:4b passes, so smaller models weren't needed.

Staging (hosted parked on the mock daily quota, real local model)

Test	Result
`e2e_http` with `REQUIRE_LLM=1`	25/25
`nl_check`	6/6 (median 6.5 s, max 8.1 s)
`browser_e2e`	16/16 (exact logo phrase)
`theme_check` with `REQUIRE_LLM=1`	26/26 (median 6.1 s, max 15.9 s)
`llm_proxy_check`	25/25
`fake_model_browser`	15/15
unit tests	17/17 + 6/6

## Why The live demo's typed chat timed out (busy reply after 45 s) while the OpenRouter daily quota was used up. **Root cause:** the local model got the full hosted prompt, about 3.1k tokens (schema + examples + catalog). On this CPU, prefill costs about 13 ms per token. So every prompt-cache miss cost about 40 s on the single llama-server slot, and any request queued behind it also waited. Cache misses were frequent: - my own debug and benchmark runs, each with a fresh store and a different prefix (they were running at 22:28–22:30); - demo and staging priming different prefixes; - per-visitor look and page text sitting inside the system prompt. ## Fix - **Compact local prompt:** `LOCAL_PROMPT` + `_local_system_prompt` is about 1.2k tokens. It contains: - a terse action list and one-line examples; - the default page text; - one line per listing, with L-refs and dollar prices. It holds no per-visitor data, so it's identical for every visitor and the cache always hits. It only changes when the catalog changes, and the primer re-warms it then. - **Per-visitor part** (`_local_user`, a few dozen tokens): the look overrides (if any), a keyword/budget `MATCHING LISTINGS` hint, and `VISITOR: <message>`. - **Lean generation:** `max_tokens` 200, no history, no tool round. Over-budget card picks are dropped when some picks fit. - **Buy vs sell:** the examples are separated (selling your own thing opens the listing form, never a checkout). This replaces the unfinished fix/local-sell-vs-checkout change. ## Measured (gemma3:4b, 8-core CPU, -t 6, nice 10) - **Demo catalog, 12 typical prompts, real `handle_chat`:** 12/12 correct, median 5.9 s, max 6.7 s. - logo phrase 4.5 s - search 6.0 s - sell form 6.7 s - gradient 6.2 s - compare 6.3 s - checkout 4.1 s - hide 3.8 s - reset 4.0 s - light 5.9 s - garden 5.1 s - cash 4.4 s - kid/budget 6.4 s - **With a custom visitor look:** 6/6, median 5.8 s, max 7.7 s. - **Cold prefill of the new prompt:** about 13 s, once per catalog change. gemma3:4b passes, so smaller models weren't needed. ## Staging (hosted parked on the mock daily quota, real local model) | Test | Result | |---|---| | `e2e_http` with `REQUIRE_LLM=1` | 25/25 | | `nl_check` | 6/6 (median 6.5 s, max 8.1 s) | | `browser_e2e` | 16/16 (exact logo phrase) | | `theme_check` with `REQUIRE_LLM=1` | 26/26 (median 6.1 s, max 15.9 s) | | `llm_proxy_check` | 25/25 | | `fake_model_browser` | 15/15 | | unit tests | 17/17 + 6/6 |

leora added 2 commits 2026-10-08 22:43:55 +08:00

Local model: compact prompt (1.2k tokens, same for every visitor) so typed replies take ~6 s instead of timing out 70f5b0525e

Root cause of the 45 s timeouts: the local model got the full hosted prompt (~3.1k tokens with examples). Any prompt-cache
miss meant ~40 s of CPU prefill on the single slot, and every request queued behind one paid it too.

- LOCAL_PROMPT/_local_system_prompt: terse action list, one-line examples, default page text, one line per listing
  (L-refs, dollar prices). No per-visitor data in it, so the cache always hits; it only changes with the catalog.
- _local_user: look overrides (if any) + keyword/budget MATCHING LISTINGS hint + 'VISITOR: <message>'.
- max_tokens 200 for the local model; over-budget card picks dropped when some fit (_respect_budget).
- Buy vs sell examples separated (sell my own thing -> listing form, never checkout).
- 18/18 correct on typical prompts, warm median 5.9 s, max 7.7 s; cold prefill ~13 s.

e2e: agent API search with a higher limit (staging accumulates test listings) c8e1c162ee

leora merged commit d7a17f32d2 into main

2026-10-08 22:43:56 +08:00

leora referenced this issue from a commit

2026-10-08 22:43:56 +08:00

Merge pull request 'Local model: compact cached prompt so typed replies take ~6 s (fixes 45 s timeouts)' (#8) from fix/local-fast-prompt into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: leora/agent-market#8