leora/agent-market
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Typed chat: free hosted models -> local model (llama-server), no keyword rules; busy reply + Retry #7

Merged
leora merged 7 commits from feat/local-llm-backup into main 2026-10-08 22:18:21 +08:00
Conversation 0 Commits 7 Files Changed 22 +847 -758
leora commented 2026-10-08 22:18:17 +08:00
Owner
Copy Link

What

Typed chat now uses OpenRouter free models → local model on this box → nothing else. The keyword-rules agent is removed from typed chat.

  • No rule-based replies for typed messages. fallback_agent, parse_look_command (+ colour/gradient/section tables), the copy-edit parser and _rules_check are deleted. If every model fails (or none is configured): "The assistant is busy, please try again in a moment." with a Retry button that resends the same message (agent: "unavailable", no actions, badge AI: busy).
  • Clicks, forms and payments stay deterministic (fallback_event, unchanged). The landing view is a deterministic featured-listings panel, and the Sell button opens the empty form.
  • Local model: gemma3:4b (Q4_K_M) on llama.cpp llama-server, run directly (no Ollama serve). ops/local-llm-watchdog.sh:
    • listens on 127.0.0.1:11434 only;
    • restarts the server on exit (with a crash backoff) or after 3 failed health checks;
    • rotates its log, and deploy.sh makes sure it's running (--ensure, idempotent).
  • Lean settings (resource pressure): no vision projector, weights mmapped and not repacked, -c 6144 -np 1, q8 KV cache, flash attention, -t 6 -tb 6 at nice 10, --poll 0, --cache-ram 1024, --swa-full (so Gemma's prompt cache is reused).
  • Reliability for a 4B model without a rules safety net:
    • a JSON GBNF grammar;
    • worked examples in the cached system prompt;
    • a labelled visitor message;
    • short catalog refs (L1, L2, ...) mapped back to real ids, because the model miscopied 8-hex ids.
  • Primer re-primes after a model-server restart (/metrics counter). Otherwise it only reads /metrics every 30 s, so the server idles at 0% CPU.
  • Bug fixed: with a local model configured, hosted calls were sent without the API key (cfg["local"] is a dict on hosted configs). Only the local server skips the key now. Test added.
  • README updated: chain, busy reply, llama-server setup, resources, benchmark.

Measured on this box (8 vCPU, no GPU, 16 GB)

  • Idle: 0% CPU, 2.9 GB RSS (0.6 GB private + 2.4 GB mmapped model file). It was about 10 GB under Ollama.

  • Answering: about 6 cores at nice 10 for 4–14 s, RSS 3.0 GB.

  • Warm latency per reply (real chain, typed):

    Reply Latency
    logo phrase 5 s
    hide / reset / light / list 4–6 s
    search 7–9 s
    compare 8–10 s
    checkout 6.5 s
    listing form 11–13 s
    cash meetup 14 s
    "show me everything" about 22 s

    Cold (first turn before priming): 40–60 s.

  • Other services: stayed responsive during about 10 min of local-model load (294 probes each). Port 8788 had a median of 1.6 ms (max 34 ms). Sigil on 8789 had a median of 1.4 ms (max 11 ms). Gitea had a median of 1.1 ms (max 30 ms); one refused probe came from an unrelated Gitea restart (SIGTERM).

Tests

  • tests/llm_proxy_check.sh: 25/25. It covers the busy reply for every failure mode, clicks while busy, the no-model server, and that local requests use the grammar and never carry the hosted key.
  • tests/fake_model_browser.sh: 15/15. It covers busy + Retry (resends once, no duplicate bubble), the quota badge and the local badge.
  • Unit tests: test_agent_units 13/13 (refs, primer restart detection), test_copy_defaults 6/6 (replaces test_copyedit.py), test_theme OK.
  • Staging, real local model (hosted parked on the mock daily-quota 429):
    Test Result
    e2e_http with REQUIRE_LLM=1 25/25
    nl_check 6/6 (median 8.5 s)
    browser_e2e 16/16 (incl. the exact logo phrase)
    theme_check with REQUIRE_LLM=1 26/26 (median 10.5 s)
## What Typed chat now uses **OpenRouter free models → local model on this box → nothing else**. The keyword-rules agent is removed from typed chat. - **No rule-based replies for typed messages.** `fallback_agent`, `parse_look_command` (+ colour/gradient/section tables), the copy-edit parser and `_rules_check` are deleted. If every model fails (or none is configured): **"The assistant is busy, please try again in a moment."** with a **Retry** button that resends the same message (`agent: "unavailable"`, no actions, badge `AI: busy`). - **Clicks, forms and payments stay deterministic** (`fallback_event`, unchanged). The landing view is a deterministic featured-listings panel, and the Sell button opens the empty form. - **Local model: gemma3:4b (Q4_K_M) on llama.cpp `llama-server`**, run directly (no Ollama serve). `ops/local-llm-watchdog.sh`: - listens on 127.0.0.1:11434 only; - restarts the server on exit (with a crash backoff) or after 3 failed health checks; - rotates its log, and `deploy.sh` makes sure it's running (`--ensure`, idempotent). - **Lean settings** (resource pressure): no vision projector, weights mmapped and not repacked, `-c 6144 -np 1`, q8 KV cache, flash attention, `-t 6 -tb 6` at nice 10, `--poll 0`, `--cache-ram 1024`, `--swa-full` (so Gemma's prompt cache is reused). - **Reliability for a 4B model without a rules safety net:** - a JSON GBNF grammar; - worked examples in the cached system prompt; - a labelled visitor message; - short catalog refs (L1, L2, ...) mapped back to real ids, because the model miscopied 8-hex ids. - **Primer** re-primes after a model-server restart (`/metrics` counter). Otherwise it only reads `/metrics` every 30 s, so the server idles at 0% CPU. - **Bug fixed:** with a local model configured, hosted calls were sent **without the API key** (`cfg["local"]` is a dict on hosted configs). Only the local server skips the key now. Test added. - README updated: chain, busy reply, llama-server setup, resources, benchmark. ## Measured on this box (8 vCPU, no GPU, 16 GB) - **Idle:** 0% CPU, 2.9 GB RSS (0.6 GB private + 2.4 GB mmapped model file). It was about 10 GB under Ollama. - **Answering:** about 6 cores at nice 10 for 4–14 s, RSS 3.0 GB. - **Warm latency per reply** (real chain, typed): | Reply | Latency | |---|---| | logo phrase | 5 s | | hide / reset / light / list | 4–6 s | | search | 7–9 s | | compare | 8–10 s | | checkout | 6.5 s | | listing form | 11–13 s | | cash meetup | 14 s | | "show me everything" | about 22 s | Cold (first turn before priming): 40–60 s. - **Other services:** stayed responsive during about 10 min of local-model load (294 probes each). Port 8788 had a median of 1.6 ms (max 34 ms). Sigil on 8789 had a median of 1.4 ms (max 11 ms). Gitea had a median of 1.1 ms (max 30 ms); one refused probe came from an unrelated Gitea restart (SIGTERM). ## Tests - `tests/llm_proxy_check.sh`: 25/25. It covers the busy reply for every failure mode, clicks while busy, the no-model server, and that local requests use the grammar and never carry the hosted key. - `tests/fake_model_browser.sh`: 15/15. It covers busy + Retry (resends once, no duplicate bubble), the quota badge and the local badge. - Unit tests: `test_agent_units` 13/13 (refs, primer restart detection), `test_copy_defaults` 6/6 (replaces `test_copyedit.py`), `test_theme` OK. - **Staging, real local model** (hosted parked on the mock daily-quota 429): | Test | Result | |---|---| | `e2e_http` with `REQUIRE_LLM=1` | 25/25 | | `nl_check` | 6/6 (median 8.5 s) | | `browser_e2e` | 16/16 (incl. the exact logo phrase) | | `theme_check` with `REQUIRE_LLM=1` | 26/26 (median 10.5 s) |
leora added 7 commits 2026-10-08 22:18:17 +08:00 
- Remove the keyword-rules agent from typed chat: fallback_agent, parse_look_command (+ tables), the copy-edit parser
  and _rules_check are gone. If every model fails: 'The assistant is busy, please try again in a moment.' with a Retry
  button (agent 'unavailable', no actions). Clicks, forms and payments stay deterministic (fallback_event); the landing
  view is a deterministic featured-listings panel; the Sell button opens the empty form.
- Local model via llama.cpp llama-server directly (ops/local-llm-watchdog.sh, loopback only, restart on exit/hang,
  crash backoff, log rotation; deploy.sh ensures it runs). Lean: no vision projector, no weight repack (mmapped), ctx
  6144, 1 slot, q8 KV, 6 threads at nice 10, poll 0, 1 GiB prompt cache. Idle 0% CPU, 2.9 GB RSS (0.6 GB private).
- Local requests use a JSON GBNF grammar, worked examples in the cached system prompt, and a labelled visitor message.
  10/10 correct on the real prompt, warm 4-13 s.
- Primer re-primes after a model-server restart (llama-server /metrics counter) and otherwise stays idle.
- Fix: hosted calls kept the Authorization header off when a local model was configured (cfg['local'] is a dict on
  hosted configs); now only the local server itself skips the key. Test added.
- Badge/chat header: 'AI: busy', 'AI: quota used up', 'AI: unavailable'; no 'built-in'/'checked by rules' texts.
- Tests: proxy checks expect the busy reply (never rules), clicks still work while busy, no-model server, local grammar/
  prompt/no key; browser checks for busy + Retry; e2e_http accepts busy for typed turns unless REQUIRE_LLM=1;
  test_copyedit.py replaced by test_copy_defaults.py; README updated.
A 4B model miscopied 8-hex listing ids (picked a tutoring id for a kids' bike). The local prompt now shows L-refs, the
server maps them back before tools/hydration (unknown refs are dropped as before), tool results are aliased the same
way, and refs are scrubbed from replies. Hosted models still see real ids.
leora merged commit 11994b9e73 into main 2026-10-08 22:18:21 +08:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
leora
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leora/agent-market#7
No description provided.
Delete Branch "feat/local-llm-backup"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?