Real hosted LLM via OpenRouter free models, with fallback chain and per-IP rate limit #2

leora · 2026-10-08T19:29:54+08:00

leora commented

2026-10-08 19:29:54 +08:00

Switches the demo from the rule-based agent to a hosted LLM, using OpenRouter's free models.

Model chain
nvidia/nemotron-3-super-120b-a12b:free → apodex/apodex-1.1-mini:free → google/gemma-4-31b-it:free → rule-based agent

Changes

Key: read from a file (AGENT_MARKET_LLM_API_KEY_FILE) that lives outside the repo. It never reaches the browser or the logs. deploy.sh sources ~/.config/agent-market/llm.env and <env>.env, which hold no secrets.
Fallback chain:
- A model that returns 429 gets skipped for 60 s.
- There's a hard deadline of 12 s per model attempt and 28 s for the whole chain.
- Invalid JSON or an error moves on to the next model, and the last resort is the rule-based agent. The page never breaks.
Rate limit: a per-IP token bucket on /api/chat, 12/min with a burst of 6. It returns 429 with Retry-After. CF-Connecting-IP is trusted only from loopback.
Prompt: rewritten so the model picks ids from the catalog and returns one panel in a single call. Listing ids are still checked against the store.
Events: free text goes to the LLM. UI events (clicks, form submits, payment results) and the landing view stay deterministic, which keeps them instant and saves free-tier quota. AGENT_MARKET_LLM_EVENTS=1 routes events to the LLM too.
Frontend:
- The header badge shows the model that actually answered, e.g. AI: nvidia/nemotron-3-super-120b-a12b.
- The chat header shows latency.
- The listing form respects the payment options the agent prefilled, e.g. cash only.

Model benchmark (scored on valid UI actions and latency)

nemotron-3-super: 8/8 valid, about 2–3 s.
apodex-1.1-mini: 6/6 valid, about 2 s.
gemma-4-31b and gemma-4-26b: 429 upstream.
inkling: 403, limited to agentic harnesses.
nemotron-3.5-lightning: 24–55 s and non-JSON.

Tests on staging (:8795)

e2e_http.py: 23/23 passed.
browser_e2e.js: 11/11 passed.
nl_check.py against the real model: 6/6 passed, median 2.1 s.
llm_proxy_check.sh: 6/6 passed. It covers the 429 to next-model path, falling back to the rule-based agent when all models fail, the cooldown, the key never reaching the browser, and the per-IP limit.

Switches the demo from the rule-based agent to a hosted LLM, using OpenRouter's free models. **Model chain** `nvidia/nemotron-3-super-120b-a12b:free` → `apodex/apodex-1.1-mini:free` → `google/gemma-4-31b-it:free` → rule-based agent **Changes** - **Key:** read from a file (`AGENT_MARKET_LLM_API_KEY_FILE`) that lives outside the repo. It never reaches the browser or the logs. `deploy.sh` sources `~/.config/agent-market/llm.env` and `<env>.env`, which hold no secrets. - **Fallback chain:** - A model that returns 429 gets skipped for 60 s. - There's a hard deadline of 12 s per model attempt and 28 s for the whole chain. - Invalid JSON or an error moves on to the next model, and the last resort is the rule-based agent. The page never breaks. - **Rate limit:** a per-IP token bucket on `/api/chat`, 12/min with a burst of 6. It returns 429 with `Retry-After`. `CF-Connecting-IP` is trusted only from loopback. - **Prompt:** rewritten so the model picks ids from the catalog and returns one panel in a single call. Listing ids are still checked against the store. - **Events:** free text goes to the LLM. UI events (clicks, form submits, payment results) and the landing view stay deterministic, which keeps them instant and saves free-tier quota. `AGENT_MARKET_LLM_EVENTS=1` routes events to the LLM too. - **Frontend:** - The header badge shows the model that actually answered, e.g. `AI: nvidia/nemotron-3-super-120b-a12b`. - The chat header shows latency. - The listing form respects the payment options the agent prefilled, e.g. cash only. **Model benchmark** (scored on valid UI actions and latency) - nemotron-3-super: 8/8 valid, about 2–3 s. - apodex-1.1-mini: 6/6 valid, about 2 s. - gemma-4-31b and gemma-4-26b: 429 upstream. - inkling: 403, limited to agentic harnesses. - nemotron-3.5-lightning: 24–55 s and non-JSON. **Tests on staging (:8795)** - `e2e_http.py`: 23/23 passed. - `browser_e2e.js`: 11/11 passed. - `nl_check.py` against the real model: 6/6 passed, median 2.1 s. - `llm_proxy_check.sh`: 6/6 passed. It covers the 429 to next-model path, falling back to the rule-based agent when all models fail, the cooldown, the key never reaching the browser, and the per-IP limit.

leora added 2 commits 2026-10-08 19:29:54 +08:00

OpenRouter free-model LLM: key from file, model fallback chain with 429 cooldown and hard deadlines, per-IP chat rate limit, model badge, deterministic UI events b1a43c897c

tests: natural-language checks against the real model; browser test waits for LLM replies; README LLM docs 584593cb0a

leora added 1 commit 2026-10-08 19:29:55 +08:00

tests: natural-language checks against the real model; browser test waits for LLM replies; README LLM docs 584593cb0a

leora merged commit c925544fe9 into main

2026-10-08 19:29:59 +08:00

leora referenced this issue from a commit

2026-10-08 19:30:00 +08:00

Merge pull request 'Real hosted LLM via OpenRouter free models, with fallback chain and per-IP rate limit' (#2) from feat/openrouter-llm into main

leora referenced this pull request

2026-10-08 19:31:05 +08:00