Merge pull request 'Real hosted LLM via OpenRouter free models, with fallback chain and per-IP rate limit' (#2) from feat/openrouter-llm into main

README.md

@@ -49,17 +49,39 @@ The server stays small on purpose. It does three things:
 | anything else | refused (`not_a_test_card`) |
 
 ## LLM configuration
-The model has to be hosted. The server reads its config from the environment only, and the key never reaches the browser:
+The model has to be hosted. The server talks to any OpenAI-compatible `/chat/completions` endpoint, and the
+key stays on the server: it never goes to the browser, never into the repo, and never into a URL.
 
 | Variable | Meaning |
 |---|---|
-| `AGENT_MARKET_LLM_API_KEY` | key for any OpenAI-compatible `/chat/completions` endpoint |
+| `AGENT_MARKET_LLM_API_KEY_FILE` | path to a file holding the key (preferred), e.g. `~/.config/agent-market/openrouter.key` (mode 600) |
-| `AGENT_MARKET_LLM_BASE_URL` | e.g. `https://api.x.ai/v1` |
+| `AGENT_MARKET_LLM_API_KEY` | the key itself, if a file can't be used |
-| `AGENT_MARKET_LLM_MODEL` | model name |
+| `AGENT_MARKET_LLM_BASE_URL` | e.g. `https://openrouter.ai/api/v1` |
-| `XAI_API_KEY` / `OPENAI_API_KEY` | shortcuts with default base URL and model |
+| `AGENT_MARKET_LLM_MODELS` | comma-separated fallback chain, tried in order |
+| `AGENT_MARKET_LLM_TIMEOUT` / `_BUDGET` | hard cap per model attempt (12 s) and for the whole chain (28 s) |
+| `AGENT_MARKET_LLM_COOLDOWN` | how long a model that returned 429 gets skipped (60 s) |
+| `AGENT_MARKET_CHAT_RPM` / `_BURST` | per-IP token bucket on `/api/chat` (12/min, burst 6). `CF-Connecting-IP` is trusted only from loopback, i.e. the tunnel. |
 
-With no key set, the demo runs a **rule-based fallback agent**. The page labels this in the header badge and the
+`deploy.sh` loads `~/.config/agent-market/llm.env` and `~/.config/agent-market/<env>.env` if they exist. Both
-chat header. It emits the same UI actions, so the whole flow still works.
+sit outside the repo. The box runs OpenRouter free models:
+
+`nvidia/nemotron-3-super-120b-a12b:free` → `apodex/apodex-1.1-mini:free` → `google/gemma-4-31b-it:free` → rule-based agent
+
+The order came from a benchmark of natural-language prompts, scored on valid UI actions and latency. In that run:
+- **nemotron-3-super**: 8/8 valid, about 2–3 s.
+- **apodex-1.1-mini**: 6/6 valid, about 2 s.
+- **gemma-4-31b and gemma-4-26b**: rate-limited upstream (429).
+- **inkling**: 403, because it's limited to agentic harnesses.
+- **nemotron-3.5-lightning**: 24–55 s, and returned non-JSON.
+
+If a model returns 429, it times out, or it returns invalid JSON, the server moves on to the next model in the chain. When every model fails,
+the labelled rule-based agent answers, so the page never breaks. The header badge shows the model that answered.
+
+**Free text goes to the LLM. Structured UI events stay deterministic.** Button clicks, form submits and payment results
+each have exactly one outcome, so the server handles them deterministically. That keeps them instant and saves free-tier quota.
+The landing view doesn't call the model either. Set `AGENT_MARKET_LLM_EVENTS=1` to send events to the LLM as well.
+
+With no LLM configured, the demo runs the **rule-based fallback agent**. The page labels this.
 
 ## Run
 ```bash
@@ -82,12 +104,14 @@ curl -s $BASE/api/actions -H 'X-Agent-Id: agent:my-bot' -H 'Content-Type: applic
 ## Tests
 ```bash
 python3 tests/e2e_http.py http://127.0.0.1:8795           # HTTP end-to-end (23 checks)
-bash tests/llm_proxy_check.sh                             # LLM proxy path against a fake LLM
+bash tests/llm_proxy_check.sh                             # fake LLM: 429 chain, cooldown, fallback, key handling, rate limit
+python3 tests/nl_check.py http://127.0.0.1:8795           # real model: 6 natural-language prompts
 NODE_PATH=/path/to/node_modules node tests/browser_e2e.js http://127.0.0.1:8795 ./screenshots  # headless Chrome
 ```
 
 ## Known gaps (demo)
 - Agent identity (`X-Agent-Id`) isn't verified, so anyone can claim any id.
 - There's no real messaging between buyer and seller for cash meetups.
-- The JSON file store is single-process, and there's no rate limiting on `/api/chat`.
+- The JSON file store is single-process, and rate limits live in memory, so a restart resets them.
+- Free OpenRouter models have small daily request caps and get rate-limited upstream. Once those run out, the rule-based agent answers.
 - The mock payment provider is the only one. A real provider would need its own test keys and webhook secret.

deploy.sh

@@ -15,6 +15,13 @@ mkdir -p "$ROOT"
 if [ ! -d "$ROOT/app/.git" ]; then git clone -q "$REPO" "$ROOT/app"; fi
 cd "$ROOT/app" && git fetch -q origin && git checkout -q --detach "origin/$REF" 2>/dev/null || git checkout -q --detach "$REF"
 echo "deploying $ENVN @ $(git rev-parse --short HEAD) on :$PORT"
+# Optional config outside the repo (never committed): shared LLM settings, then per-env overrides.
+# llm.env holds e.g. AGENT_MARKET_LLM_API_KEY_FILE / _BASE_URL / _MODELS (a path to the key, never the key itself).
+CONF=$HOME/.config/agent-market
+set -a
+[ -f "$CONF/llm.env" ] && . "$CONF/llm.env"
+[ -f "$CONF/$ENVN.env" ] && . "$CONF/$ENVN.env"
+set +a
 PID=$(ss -ltnp 2>/dev/null | grep ":$PORT " | grep -o 'pid=[0-9]*' | cut -d= -f2 || true)
 [ -n "$PID" ] && kill "$PID" && sleep 0.5
 AGENT_MARKET_PORT=$PORT AGENT_MARKET_ENV=$ENVN AGENT_MARKET_DATA=$ROOT/data \

market/agent.py

@@ -4,56 +4,40 @@ Returns {reply, actions} where actions is a list of UI action objects from a fix
 component set the frontend knows how to render.
 """
 from __future__ import annotations
-import json, os, re, urllib.request, urllib.error
+import concurrent.futures, json, os, re, time, urllib.request, urllib.error
 from .store import new_id, now
 
 FEE_BPS = int(os.environ.get("AGENT_MARKET_FEE_BPS", "500"))  # 5.00%
 FEE_RATE = FEE_BPS / 10000.0
 
 UI_SCHEMA = """
-You are Market Guide, the concierge for Agent Market (a TEST DEMO marketplace).
+You are Market Guide, the AI agent that drives the page of Agent Market, a TEST DEMO marketplace
-You control the page by returning JSON UI actions the frontend renders.
+where people and AI agents buy and sell products and services. All listings are fake test data and
-Never invent IDs. Only use listing ids from SEARCH_RESULTS or CREATE_LISTING results.
+all payments are TEST MODE ONLY (no real money).
-Currency amounts are integer cents. Always mention that payments are TEST MODE ONLY.
-Online purchases add a facilitation fee of FEE_PERCENT. Cash-in-person deals have $0 fee.
 
-Reply ONLY with a single JSON object of this shape (no markdown fences):
+You control what the page shows by returning UI actions. Reply with ONE JSON object and nothing else
-{
+(no markdown, no prose outside JSON):
-  "reply": "short helpful message to the user",
+{"reply": "<short friendly message, max 60 words>", "actions": [ ... ]}
-  "actions": [ /* 0 or more of the actions below */ ]
-}
 
-Allowed actions (pick only from this set):
+Allowed actions (use only these types; listing ids must come from CATALOG below, never invent them):
-1. {"type":"show_hero","payload":{"title":"...","subtitle":"...","bullets":["..."]}}
+- {"type":"show_cards","payload":{"heading":"...","listing_ids":["lst_..."]}}            list matching items
-2. {"type":"show_cards","payload":{"heading":"...","listing_ids":["lst_..."]}}
+- {"type":"show_compare","payload":{"heading":"...","listing_ids":["lst_...","lst_..."]}} side-by-side, 2-4 ids
-3. {"type":"show_listing_form","payload":{"defaults":{"title":"","kind":"product","price":0,"description":"","category":"","area":"","payment_options":["online","cash"]}}}
+- {"type":"show_checkout","payload":{"listing_id":"lst_..."}}       online purchase; the panel shows price + fee
-4. {"type":"show_compare","payload":{"listing_ids":["lst_...","lst_..."],"heading":"..."}}
+- {"type":"show_cash_meetup","payload":{"listing_id":"lst_...","suggested_place":"...","suggested_time":"..."}}  pay cash in person, $0 fee
-5. {"type":"show_checkout","payload":{"listing_id":"lst_...","quantity":1}}
+- {"type":"show_listing_form","payload":{"defaults":{"title":"...","kind":"product|service","price":<integer cents>,"description":"...","category":"...","area":"Demo Town","payment_options":["online","cash"]}}}
-6. {"type":"show_cash_meetup","payload":{"listing_id":"lst_...","suggested_place":"...","suggested_time":"..."}}
+     use when the user wants to sell/list something; prefill everything you can infer, the user presses Publish
-7. {"type":"show_receipt","payload":{"order_id":"ord_..."}}
+     ("cash only" => payment_options ["cash"]; "$120" => price 12000)
-9. {"type":"clear_panel","payload":{}}
+- {"type":"show_hero","payload":{"title":"...","subtitle":"...","bullets":["..."]}}       welcome / help
-8. {"type":"toast","payload":{"level":"info|ok|warn","text":"..."}}
+- {"type":"toast","payload":{"level":"info|ok|warn","text":"..."}}
 
-You may also ask the server to run tools by including a "tools" array. The server
+Rules:
-executes them and you get the results before the final reply. Tools:
+- Interpret needs, not just keywords: e.g. "get my kid to school" => a kids' bike; "garden" => lawn/yard services.
-- {"name":"search","args":{"q":"...","kind":"product|service|any","limit":6}}
+- Respect budgets (prices in CATALOG are integer cents: 5500 = $55.00).
-- {"name":"get_listing","args":{"id":"lst_..."}}
+- If nothing fits, say so and show the closest options with show_cards.
-- {"name":"create_listing","args":{"title":"...","kind":"product|service","price":cents,"description":"...","category":"...","area":"...","payment_options":["online","cash"],"seller":"demo-seller","seller_type":"human","emoji":"📦"}}
+- Online payment adds a FEE_PERCENT facilitation fee paid by the buyer; cash in person has no fee. Mention this when relevant.
-- {"name":"start_checkout","args":{"listing_id":"...","buyer":"demo-buyer","quantity":1}}
+- Usually return exactly one main panel action. Keep "reply" short and do not repeat the whole catalog in text.
-- {"name":"start_cash_deal","args":{"listing_id":"...","buyer":"demo-buyer","place":"...","time":"..."}}
+- Optional: if you truly need a server action first, add "tools":[{"name":"search","args":{"q":"..."}}] (also
-- {"name":"complete_cash_deal","args":{"deal_id":"cash_..."}}
+  create_listing, start_checkout, start_cash_deal); you will get TOOL_RESULTS and then answer. Prefer not to.
-
-Card details never reach you: the checkout panel posts them straight to the test payment
-endpoint and then sends you a UI_EVENT {"type":"payment_result",...}. React with show_receipt.
-UI_EVENT messages are clicks/forms from the page (JSON). Handle them:
- view_listing -> show_cards with that id; buy_online -> start_checkout then show_checkout;
- buy_cash -> start_cash_deal then show_cash_meetup; compare -> show_compare;
- sell -> show_listing_form; submit_listing -> create_listing then show_cards with the new id;
- payment_result -> show_receipt; cash_completed -> complete_cash_deal then show_receipt.
-
-Prefer one clear panel at a time. For "show me bikes" run search then show_cards.
-For "I want to buy X" show_checkout. For "meet in person" show_cash_meetup.
-For "I want to sell" show_listing_form. Keep reply under 80 words.
 """.replace("FEE_PERCENT", f"{FEE_RATE*100:.1f}%")
 
 
@@ -407,113 +391,202 @@ def fallback_agent(store, message: str, history, actor: str = "demo-buyer"):
 
 # ---------- LLM path ----------
 
-def _llm_config():
+LLM_TIMEOUT = float(os.environ.get("AGENT_MARKET_LLM_TIMEOUT", "12"))        # hard cap per model attempt
-    """Hosted LLM config from env only (OpenAI-compatible chat completions). Never logs the key."""
+LLM_BUDGET = float(os.environ.get("AGENT_MARKET_LLM_BUDGET", "28"))          # hard cap across the whole chain
-    key_env = None
+_pool = concurrent.futures.ThreadPoolExecutor(max_workers=8, thread_name_prefix="llm")
+COOLDOWN_429 = float(os.environ.get("AGENT_MARKET_LLM_COOLDOWN", "60"))
+_cooldown = {}  # model -> unix time until which we skip it after a 429
+LAST = {"model": None, "at": None, "error": None}
+
+
+class LLMError(Exception):
+    def __init__(self, kind, detail=""):
+        super().__init__(f"{kind}: {detail}")
+        self.kind = kind
+
+
+def _read_key():
+    """Key from a file path (preferred) or env. Never logged, never returned to clients."""
+    path = os.environ.get("AGENT_MARKET_LLM_API_KEY_FILE")
+    if path:
+        try:
+            with open(os.path.expanduser(path)) as f:
+                k = f.read().strip()
+            if k:
+                return k
+        except OSError:
+            pass
     for env in ("AGENT_MARKET_LLM_API_KEY", "XAI_API_KEY", "OPENAI_API_KEY"):
         if os.environ.get(env):
-            key_env = env
+            return os.environ[env]
-            break
-    if not key_env:
     return None
-    defaults = {
+
-        "XAI_API_KEY": ("https://api.x.ai/v1", "grok-3-mini"),
+
-        "OPENAI_API_KEY": ("https://api.openai.com/v1", "gpt-4o-mini"),
+def _llm_config():
-        "AGENT_MARKET_LLM_API_KEY": ("", ""),
+    """Hosted LLM config (OpenAI-compatible chat completions) with an ordered model fallback chain."""
-    }[key_env]
+    if not _read_key():
-    base = os.environ.get("AGENT_MARKET_LLM_BASE_URL") or defaults[0]
+        return None
-    model = os.environ.get("AGENT_MARKET_LLM_MODEL") or defaults[1]
+    base = os.environ.get("AGENT_MARKET_LLM_BASE_URL") or ""
-    if not base or not model:
+    if not base:
+        base = "https://api.x.ai/v1" if os.environ.get("XAI_API_KEY") else ("https://api.openai.com/v1" if os.environ.get("OPENAI_API_KEY") else "")
+    models = [m.strip() for m in (os.environ.get("AGENT_MARKET_LLM_MODELS") or os.environ.get("AGENT_MARKET_LLM_MODEL") or "").split(",") if m.strip()]
+    if not base or not models:
         return None
     return {"provider": base.split("//")[-1].split("/")[0], "url": base.rstrip("/") + "/chat/completions",
-            "model": model, "env": key_env}
+            "models": models, "model": models[0]}
 
 
-def _call_llm(cfg, messages):
+def _call_llm(cfg, model, messages, json_mode=True):
-    key = os.environ[cfg["env"]]
+    payload = {"model": model, "messages": messages, "temperature": 0.2, "max_tokens": 900}
-    body = json.dumps({
+    if json_mode:
-        "model": cfg["model"],
+        payload["response_format"] = {"type": "json_object"}
-        "messages": messages,
+    req = urllib.request.Request(cfg["url"], data=json.dumps(payload).encode(), method="POST", headers={
-        "temperature": 0.2,
+        "Authorization": "Bearer " + _read_key(),
-        "response_format": {"type": "json_object"},
-    }).encode()
-    req = urllib.request.Request(cfg["url"], data=body, method="POST", headers={
-        "Authorization": f"Bearer {key}",
         "Content-Type": "application/json",
+        "X-Title": "Agent Market demo",
     })
-    with urllib.request.urlopen(req, timeout=45) as r:
+    try:
-        return json.loads(r.read().decode())
+        with urllib.request.urlopen(req, timeout=LLM_TIMEOUT) as r:
+            data = json.loads(r.read().decode())
+    except urllib.error.HTTPError as e:
+        body = e.read().decode(errors="replace")[:300]
+        if e.code == 429:
+            raise LLMError("rate_limited", body)
+        if e.code == 400 and json_mode and "response_format" in body:
+            return _call_llm(cfg, model, messages, json_mode=False)
+        raise LLMError(f"http_{e.code}", body)
+    except (urllib.error.URLError, TimeoutError, OSError) as e:
+        raise LLMError("timeout_or_network", type(e).__name__)
+    if isinstance(data, dict) and data.get("error"):
+        err = data["error"]
+        code = err.get("code") if isinstance(err, dict) else None
+        raise LLMError("rate_limited" if code == 429 else "provider_error", str(err)[:200])
+    try:
+        return data["choices"][0]["message"].get("content") or ""
+    except (KeyError, IndexError, TypeError):
+        raise LLMError("bad_response", str(data)[:200])
 
 
 def _parse_agent_json(text: str):
-    text = (text or "").strip()
+    text = re.sub(r"<think>.*?</think>", "", text or "", flags=re.S).strip()
-    if text.startswith("```"):
+    text = re.sub(r"^```(?:json)?\s*|\s*```$", "", text).strip()
-        text = re.sub(r"^```(?:json)?\s*", "", text)
+    try:
-        text = re.sub(r"\s*```$", "", text)
+        obj = json.loads(text)
-    return json.loads(text)
+    except ValueError:
+        m = re.search(r"\{.*\}", text, re.S)
+        if not m:
+            raise LLMError("not_json", text[:120])
+        try:
+            obj = json.loads(m.group(0))
+        except ValueError:
+            raise LLMError("not_json", text[:120])
+    if not isinstance(obj, dict):
+        raise LLMError("not_json", "top-level not an object")
+    return obj
 
 
-def llm_agent(store, message: str, history, cfg):
+def _catalog(store):
-    # Round 1: may request tools
+    return [{"id": L["id"], "title": L["title"], "kind": L["kind"], "price_cents": L["price"],
-    catalog = [{"id": L["id"], "title": L["title"], "kind": L["kind"], "price": L["price"],
+             "category": L.get("category"), "about": (L.get("description") or "")[:110],
-                "tags": L.get("tags"), "payment_options": L.get("payment_options")}
+             "seller_type": L.get("seller_type"), "pay": L.get("payment_options")}
-               for L in store.all("listings") if L.get("status") == "active"]
+            for L in store.all("listings") if L.get("status") == "active"][:60]
-    sys = UI_SCHEMA + "\n\nCURRENT_LISTINGS_SUMMARY:\n" + json.dumps(catalog)[:4000]
+
-    msgs = [{"role": "system", "content": sys}]
+
-    for h in (history or [])[-8:]:
+def _llm_once(store, cfg, model, message, history):
+    sys_prompt = UI_SCHEMA + "\nCATALOG (all active listings):\n" + json.dumps(_catalog(store))[:9000]
+    msgs = [{"role": "system", "content": sys_prompt}]
+    for h in (history or [])[-6:]:
         if h.get("role") in ("user", "assistant") and h.get("content"):
-            msgs.append({"role": h["role"], "content": str(h["content"])[:1000]})
+            msgs.append({"role": h["role"], "content": str(h["content"])[:600]})
     msgs.append({"role": "user", "content": message})
-
+    content = _call_llm(cfg, model, msgs)
-    raw1 = _call_llm(cfg, msgs)
+    plan = _parse_agent_json(content)
-    content1 = raw1["choices"][0]["message"]["content"]
+    tool_results = []
-    plan = _parse_agent_json(content1)
+    tools = [t for t in (plan.get("tools") or []) if isinstance(t, dict)][:3]
-    tools = plan.get("tools") or []
-    tool_results = run_tools(store, tools) if tools else []
-
     if tools:
-        msgs.append({"role": "assistant", "content": content1})
+        tool_results = run_tools(store, tools)
-        msgs.append({"role": "user", "content": "TOOL_RESULTS:\n" + json.dumps(tool_results)[:6000] +
+        msgs += [{"role": "assistant", "content": content},
-                     "\nNow return the final {reply, actions} JSON. Do not include tools."})
+                 {"role": "user", "content": "TOOL_RESULTS:\n" + json.dumps(tool_results)[:6000] +
-        raw2 = _call_llm(cfg, msgs)
+                  "\nNow return the final {\"reply\",\"actions\"} JSON only."}]
-        content2 = raw2["choices"][0]["message"]["content"]
+        plan = _parse_agent_json(_call_llm(cfg, model, msgs))
-        plan = _parse_agent_json(content2)
+    actions = [a for a in (plan.get("actions") or []) if isinstance(a, dict) and a.get("type") in ALLOWED_ACTIONS]
+    reply = str(plan.get("reply") or "").strip()
+    if not reply and not actions:
+        raise LLMError("empty_plan", "")
+    return {"reply": (reply or "Here you go.")[:600], "actions": actions, "tools": tool_results}
 
-    actions = plan.get("actions") or []
+
-    # sanitize action types
+def llm_agent(store, message, history, cfg):
-    allowed = ALLOWED_ACTIONS
+    """Try each model in the chain; skip models cooling down after a 429. Raises if all fail."""
-    actions = [a for a in actions if isinstance(a, dict) and a.get("type") in allowed]
+    errors = []
-    return {
+    now_t = time.time()
-        "reply": str(plan.get("reply") or "Done.")[:500],
+    chain = [m for m in cfg["models"] if _cooldown.get(m, 0) <= now_t] or cfg["models"][-1:]
-        "actions": actions,
+    start = time.time()
-        "agent": cfg["provider"],
+    for model in chain:
-        "model": cfg["model"],
+        t0 = time.time()
-        "tools": tool_results,
+        remaining = LLM_BUDGET - (t0 - start)
-    }
+        if remaining < 3:
+            errors.append({"model": model, "error": "skipped_budget", "ms": 0})
+            continue
+        try:
+            fut = _pool.submit(_llm_once, store, cfg, model, message, history)
+            try:
+                out = fut.result(timeout=min(LLM_TIMEOUT, remaining))
+            except concurrent.futures.TimeoutError:
+                raise LLMError("timeout", f">{min(LLM_TIMEOUT, remaining):.0f}s")
+            out.update(agent="llm", provider=cfg["provider"], model=model,
+                       latency_ms=int((time.time() - t0) * 1000), fallbacks=errors)
+            LAST.update(model=model, at=time.time(), error=None)
+            return out
+        except LLMError as e:
+            if e.kind == "rate_limited":
+                _cooldown[model] = time.time() + COOLDOWN_429
+            errors.append({"model": model, "error": e.kind, "ms": int((time.time() - t0) * 1000)})
+        except Exception as e:  # never let the page break
+            errors.append({"model": model, "error": type(e).__name__, "ms": int((time.time() - t0) * 1000)})
+    LAST.update(error=errors[-1]["error"] if errors else "no_model", at=time.time())
+    raise LLMError("all_models_failed", json.dumps(errors))
+
+
+# Structured UI events (button clicks, form submits, payment results) have one obvious outcome,
+# so they are handled deterministically: fast, reliable, and they don't spend free-tier LLM quota.
+# Free-text messages go to the LLM. Set AGENT_MARKET_LLM_EVENTS=1 to route events to the LLM too.
+LLM_EVENTS = os.environ.get("AGENT_MARKET_LLM_EVENTS") == "1"
 
 
 def handle_chat(store, message: str, history=None, event=None, actor="demo-buyer"):
     actor = (actor or "demo-buyer")[:60]
+    t0 = time.time()
+    if event and (event.get("type") == "landing" or not LLM_EVENTS):
+        if event.get("type") == "landing":
+            out = fallback_agent(store, "show me everything", [], actor)
+        else:
+            out = fallback_event(store, event, actor)
+        out.update(agent="ui-event", latency_ms=int((time.time() - t0) * 1000))
+        return out
     cfg = _llm_config()
     if cfg:
-        text = message
+        text = message if not event else "UI_EVENT " + json.dumps(event)[:2000] + f"\n(actor/buyer id: {actor})"
-        if event:
-            text = "UI_EVENT " + json.dumps(event)[:2000] + f"\n(actor/buyer id: {actor})"
         try:
             return llm_agent(store, text, history or [], cfg)
         except Exception as e:
             out = fallback_event(store, event, actor) if event else fallback_agent(store, message, history or [], actor)
-            out["agent"] = "fallback"
+            out.update(agent="fallback", llm_error=getattr(e, "kind", type(e).__name__),
-            out["llm_error"] = type(e).__name__
+                       fallbacks=json.loads(str(e).split(": ", 1)[1]) if getattr(e, "kind", "") == "all_models_failed" else [],
+                       latency_ms=int((time.time() - t0) * 1000))
+            out["reply"] = "(The AI model is busy right now, so the simple built-in agent answered.) " + out["reply"]
             return out
     out = fallback_event(store, event, actor) if event else fallback_agent(store, message, history or [], actor)
-    out["agent"] = "fallback"
+    out.update(agent="fallback", latency_ms=int((time.time() - t0) * 1000))
     return out
 
 
 def agent_status():
     cfg = _llm_config()
     if cfg:
-        return {"mode": "llm", "provider": cfg["provider"], "model": cfg["model"]}
+        now_t = time.time()
+        return {"mode": "llm", "provider": cfg["provider"], "model": cfg["models"][0], "chain": cfg["models"],
+                "cooling_down": [m for m in cfg["models"] if _cooldown.get(m, 0) > now_t],
+                "last_model": LAST["model"], "last_error": LAST["error"], "ui_events": "llm" if LLM_EVENTS else "deterministic"}
     return {"mode": "fallback", "provider": "deterministic", "model": None,
-            "note": "No hosted LLM key configured (set AGENT_MARKET_LLM_API_KEY + AGENT_MARKET_LLM_BASE_URL + AGENT_MARKET_LLM_MODEL, or XAI_API_KEY / OPENAI_API_KEY). Using the labelled rule-based fallback agent."}
+            "note": "No hosted LLM configured (set AGENT_MARKET_LLM_API_KEY_FILE or AGENT_MARKET_LLM_API_KEY, plus AGENT_MARKET_LLM_BASE_URL and AGENT_MARKET_LLM_MODELS). Using the labelled rule-based fallback agent."}

server.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """Agent Market demo server. Tiny on purpose: static files, an LLM proxy (key stays server-side),
 test-mode payment endpoints, and a small JSON action API shared by the chat agent and outside agents."""
-import json, os, re, secrets, sys, urllib.parse
+import json, os, re, secrets, sys, threading, time, urllib.parse
 from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
 
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
@@ -16,6 +16,28 @@ ENV_LABEL = os.environ.get("AGENT_MARKET_ENV", "demo")
 STATIC = os.path.join(os.path.dirname(os.path.abspath(__file__)), "static")
 WEBHOOK_SECRET = os.environ.get("AGENT_MARKET_WEBHOOK_SECRET") or secrets.token_hex(16)  # per-process if unset
 MAX_BODY = 64 * 1024
+CHAT_RPM = int(os.environ.get("AGENT_MARKET_CHAT_RPM", "12"))      # per-IP chat requests per minute
+CHAT_BURST = int(os.environ.get("AGENT_MARKET_CHAT_BURST", "6"))
+_buckets = {}  # ip -> [tokens, last_ts]
+_bucket_lock = threading.Lock()
+
+
+def rate_ok(ip):
+    """Token bucket per client IP. Returns (allowed, retry_after_seconds)."""
+    if CHAT_RPM <= 0:
+        return True, 0
+    rate = CHAT_RPM / 60.0
+    with _bucket_lock:
+        now_t = time.time()
+        tokens, last = _buckets.get(ip, (float(CHAT_BURST), now_t))
+        tokens = min(float(CHAT_BURST), tokens + (now_t - last) * rate)
+        if tokens >= 1:
+            _buckets[ip] = (tokens - 1, now_t)
+            return True, 0
+        _buckets[ip] = (tokens, now_t)
+        if len(_buckets) > 5000:
+            _buckets.clear()
+        return False, int((1 - tokens) / rate) + 1
 
 store = Store(DATA_DIR)
 
@@ -196,13 +218,24 @@ class H(BaseHTTPRequestHandler):
         a = self.headers.get("X-Agent-Id") or (body or {}).get("actor") or "guest"
         return re.sub(r"[^A-Za-z0-9:_\-. ()]", "", str(a))[:60] or "guest"
 
-    def _json(self, code, obj):
+    def _client_ip(self):
+        # Only trust proxy headers when the peer is the local tunnel (cloudflared) on loopback.
+        peer = self.client_address[0]
+        if peer in ("127.0.0.1", "::1"):
+            fwd = self.headers.get("CF-Connecting-IP") or (self.headers.get("X-Forwarded-For") or "").split(",")[0].strip()
+            if fwd:
+                return fwd
+        return peer
+
+    def _json(self, code, obj, extra_headers=None):
         b = json.dumps(obj).encode()
         self.send_response(code)
         self.send_header("Content-Type", "application/json")
         self.send_header("Content-Length", str(len(b)))
         self.send_header("Cache-Control", "no-store")
         self.send_header("Access-Control-Allow-Origin", "*")
+        for k, v in (extra_headers or {}).items():
+            self.send_header(k, v)
         self.end_headers()
         self.wfile.write(b)
 
@@ -279,6 +312,11 @@ class H(BaseHTTPRequestHandler):
             return self._json(200, self._webhook(body))
         actor = self._actor(body)
         if path == "/api/chat":
+            ok, retry = rate_ok(self._client_ip())
+            if not ok:
+                return self._json(429, {"error": "rate_limited", "retry_after": retry, "agent": "rate-limit",
+                                        "reply": f"You're sending messages quickly. Please wait about {retry}s and try again.",
+                                        "actions": []}, {"Retry-After": str(retry)})
             msg = str(body.get("message") or "")[:2000]
             ev = body.get("event") if isinstance(body.get("event"), dict) else None
             hist = body.get("history") if isinstance(body.get("history"), list) else []

static/app.js

@@ -33,6 +33,12 @@
     return r.json();
   }
 
+  function setModel(model, ms) {
+    const short = (model || "").replace(/:free$/, "");
+    $("#agentBadge").textContent = "AI: " + short;
+    $("#agentBadge").title = (model || "") + " via OpenRouter (free tier)";
+    $("#chatMode").textContent = `AI agent · ${short}${ms ? " · " + (ms / 1000).toFixed(1) + "s" : ""}`;
+  }
   async function talk({ message, event, echo }) {
     openChat();
     if (echo) say("user", echo);
@@ -44,7 +50,9 @@
       if (event) history.push({ role: "user", content: "UI_EVENT " + JSON.stringify(event) });
       history.push({ role: "assistant", content: out.reply || "" });
       say("bot", out.reply || "(no reply)");
-      $("#chatMode").textContent = out.agent === "fallback" ? "rule-based fallback agent (no LLM key)" : `LLM agent · ${out.model || out.agent}`;
+      if (out.agent === "llm") setModel(out.model, out.latency_ms);
+      else if (out.agent === "fallback") $("#chatMode").textContent = "built-in fallback agent answered (AI model busy)";
+      else if (out.agent === "rate-limit") toast(out.reply, "warn");
       render(out.actions || []);
     } catch (e) { thinking.remove(); say("sys", "Agent error: " + e.message); }
   }
@@ -74,6 +82,7 @@
     },
     show_listing_form(p) {
       const d = p.defaults || {};
+      const payOpts = Array.isArray(d.payment_options) && d.payment_options.length ? d.payment_options : ["online", "cash"];
       const f = el("form", { class: "form", "data-component": "listing-form" },
         el("label", { class: "full" }, "Title", el("input", { name: "title", required: "", value: d.title || "", placeholder: "e.g. Vintage desk chair" })),
         el("label", {}, "Type", (() => { const s = el("select", { name: "kind" }, el("option", { value: "product" }, "Product"), el("option", { value: "service" }, "Service")); s.value = d.kind || "product"; return s; })()),
@@ -81,8 +90,8 @@
         el("label", {}, "Category", el("input", { name: "category", value: d.category || "" })),
         el("label", {}, "Area (approximate)", el("input", { name: "area", value: d.area || "Demo Town" })),
         el("label", { class: "full" }, "Description", el("textarea", { name: "description", rows: "3", required: "" }, d.description || "")),
-        el("label", {}, el("span", {}, el("input", { type: "checkbox", name: "online", checked: "" }), ` Accept online payment (${feeBps / 100}% fee, paid by buyer)`)),
+        el("label", {}, el("span", {}, el("input", { type: "checkbox", name: "online", checked: payOpts.includes("online") ? "" : null }), ` Accept online payment (${feeBps / 100}% fee, paid by buyer)`)),
-        el("label", {}, el("span", {}, el("input", { type: "checkbox", name: "cash", checked: "" }), " Accept cash in person (free)")),
+        el("label", {}, el("span", {}, el("input", { type: "checkbox", name: "cash", checked: payOpts.includes("cash") ? "" : null }), " Accept cash in person (free)")),
         el("div", { class: "full" }, el("button", { class: "btn", type: "submit" }, "Publish test listing")));
       f.addEventListener("submit", (e) => {
         e.preventDefault();
@@ -201,12 +210,12 @@
 
   api("/api/status").then((s) => {
     feeBps = s.fee_bps; $("#feePct").textContent = (feeBps / 100).toFixed(feeBps % 100 ? 1 : 0) + "%";
-    $("#agentBadge").textContent = s.agent.mode === "llm" ? `LLM: ${s.agent.model}` : "agent: rule-based fallback";
+    if (s.agent.mode === "llm") setModel(s.agent.last_model || s.agent.model);
-    $("#agentBadge").title = s.agent.note || "";
+    else { $("#agentBadge").textContent = "agent: rule-based fallback"; $("#agentBadge").title = s.agent.note || ""; }
   });
   // Landing: show featured listings via the agent.
   (async () => {
-    const out = await api("/api/chat", { message: "show me everything", actor });
+    const out = await api("/api/chat", { event: { type: "landing" }, actor });
     render(out.actions || []);
     say("bot", "Hi! I run this page. Ask me to find, compare, sell, or buy. Payments are test mode only.");
   })();

tests/browser_e2e.js

@@ -16,13 +16,19 @@ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
   await page.waitForSelector(".card", { timeout: 15000 });
   check("landing renders agent-chosen cards", (await page.$$(".card")).length >= 4);
   const badge = await page.$eval("#agentBadge", (n) => n.textContent);
-  check("agent mode badge shown", /fallback|LLM/.test(badge), badge);
+  check("agent mode badge shown", /fallback|AI:|LLM/.test(badge), badge);
 
+  // ask(): type into the chat popup and wait for the agent's reply (LLM replies take a few seconds)
+  const ask = async (text) => {
+    const n = (await page.$$(".msg.bot")).length;
+    await page.type("#chatInput", text);
+    await page.keyboard.press("Enter");
+    await page.waitForFunction((n) => document.querySelectorAll(".msg.bot").length > n, { timeout: 60000 }, n);
+    await sleep(300);
+  };
   // chat search
   await page.click("#chatLauncher");
-  await page.type("#chatInput", "show me bikes");
+  await ask("show me bikes");
-  await page.keyboard.press("Enter");
-  await page.waitForFunction(() => [...document.querySelectorAll("#stage h2")].some((h) => /bike/i.test(h.textContent)), { timeout: 15000 });
   const titles = await page.$$eval("#stage .card h3", (ns) => ns.map((n) => n.textContent));
   check("chat search renders bike cards", titles.length > 0 && titles.every((t) => /bike/i.test(t)), titles.join(" | "));
   if (SHOTS) { await sleep(600); await page.screenshot({ path: SHOTS + "/1-landing-chat-search.png" }); }
@@ -49,9 +55,7 @@ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
   check("test payment confirms, receipt shows fee", rfee === money(feeC), rfee);
 
   // cash meetup: first card offering cash
-  await page.type("#chatInput", "show me everything");
+  await ask("show me everything that I can pay for in cash");
-  await page.keyboard.press("Enter");
-  await page.waitForFunction(() => /everything/.test((document.querySelector("#stage h2") || {}).textContent || ""), { timeout: 15000 });
   const cashId = await page.evaluate(() => { const c = [...document.querySelectorAll("#stage .card")].find((c) => c.querySelector("button.cash")); return c && c.dataset.listing; });
   await page.click(`.card[data-listing="${cashId}"] button.cash`);
   await page.waitForSelector("[data-component=cash-meetup]", { timeout: 15000 });
@@ -73,8 +77,7 @@ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
   check("listing form publishes a new card", true);
 
   // compare
-  await page.type("#chatInput", "compare tutoring");
+  await ask("compare the two tutoring options");
-  await page.keyboard.press("Enter");
   await page.waitForSelector("[data-component=compare]", { timeout: 15000 });
   check("compare view renders", (await page.$$("[data-component=compare] th")).length >= 3);
 

tests/llm_proxy_check.sh

@@ -1,20 +1,34 @@
 #!/usr/bin/env bash
-# Exercises the hosted-LLM proxy path with a local fake OpenAI-compatible server (no real key needed).
+# LLM proxy path against a local fake OpenAI-compatible server (no real key needed):
+# tool call -> hydrated cards, key stays server-side, 429 fallback chain, rule-based last resort, per-IP rate limit.
 set -euo pipefail
 cd "$(dirname "$0")/.."
-D=$(mktemp -d)
+D=$(mktemp -d); KEYF=$D/key; printf 'test-dummy-key' > $KEYF; chmod 600 $KEYF
 python3 tests/mock_llm.py 18798 & M=$!
-AGENT_MARKET_PORT=18799 AGENT_MARKET_DATA=$D AGENT_MARKET_LLM_API_KEY=test-dummy-key \
+start() { # port models rpm
-  AGENT_MARKET_LLM_BASE_URL=http://127.0.0.1:18798/v1 AGENT_MARKET_LLM_MODEL=mock-model python3 server.py >/dev/null 2>&1 & S=$!
+  AGENT_MARKET_PORT=$1 AGENT_MARKET_DATA=$D/d$1 AGENT_MARKET_LLM_API_KEY_FILE=$KEYF AGENT_MARKET_CHAT_RPM=$3 AGENT_MARKET_CHAT_BURST=3 \
-trap 'kill $M $S 2>/dev/null; rm -rf $D' EXIT
+  AGENT_MARKET_LLM_BASE_URL=http://127.0.0.1:18798/v1 AGENT_MARKET_LLM_MODELS=$2 python3 server.py >/dev/null 2>&1 & echo $!; }
-sleep 1.2
+S1=$(start 18799 busy-model,mock-model 0); S2=$(start 18800 busy-model,busy-model-2 0); S3=$(start 18801 mock-model 2)
-OUT=$(curl -s localhost:18799/api/chat -H 'Content-Type: application/json' -d '{"message":"show me bikes"}')
+trap 'kill $M $S1 $S2 $S3 2>/dev/null; rm -rf $D' EXIT
+sleep 1.5
+chat() { curl -s localhost:$1/api/chat -H 'Content-Type: application/json' -d '{"message":"show me bikes"}'; }
+OUT=$(chat 18799)
 echo "$OUT" | python3 -c '
 import json,sys; d=json.load(sys.stdin)
-assert d["agent"]!="fallback", d
+assert d["agent"]=="llm" and d["model"]=="mock-model", d
+assert d["fallbacks"][0]=={"model":"busy-model","error":"rate_limited","ms":d["fallbacks"][0]["ms"]}, d["fallbacks"]
 types=[a["type"] for a in d["actions"]]
-assert types==["show_cards"], types  # unknown component from the LLM is dropped
+assert types==["show_cards"], types
 assert all("bike" in l["title"].lower() for l in d["actions"][0]["payload"]["listings"])
-print("PASS LLM proxy path: tool call -> hydrated cards; disallowed component dropped")'
+print("PASS 429 on first model -> second model answered; cards hydrated; disallowed component dropped")'
 if echo "$OUT" | grep -q test-dummy-key; then echo "FAIL key leaked to browser"; exit 1; else echo "PASS key never sent to browser"; fi
-test "$(curl -s localhost:18798/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["auth_ok"])')" -ge 2 && echo "PASS server sent key to LLM upstream only"
+test "$(curl -s localhost:18798/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["auth_ok"])')" -ge 2 && echo "PASS key read from file and sent upstream only"
+chat 18800 | python3 -c '
+import json,sys; d=json.load(sys.stdin)
+assert d["agent"]=="fallback" and d["llm_error"]=="all_models_failed" and d["actions"], d
+print("PASS all models rate-limited -> rule-based agent still renders", [a["type"] for a in d["actions"]])'
+curl -s localhost:18800/api/status | python3 -c '
+import json,sys; a=json.load(sys.stdin)["agent"]; assert set(a["cooling_down"])=={"busy-model","busy-model-2"}, a; print("PASS 429ed models put on cooldown")'
+CODES=$(for i in 1 2 3 4 5; do curl -s -o /dev/null -w '%{http_code} ' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.9' -d '{"event":{"type":"landing"}}'; done)
+OTHER=$(curl -s -o /dev/null -w '%{http_code}' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.10' -d '{"event":{"type":"landing"}}')
+[[ "$CODES" == "200 200 200 429 429 " && "$OTHER" == "200" ]] && echo "PASS per-IP rate limit (burst 3: $CODES; other IP $OTHER)" || { echo "FAIL rate limit: $CODES / $OTHER"; exit 1; }

tests/mock_llm.py

@@ -12,6 +12,10 @@ class H(BaseHTTPRequestHandler):
         b = json.dumps(SEEN).encode(); self.send_response(200); self.end_headers(); self.wfile.write(b)
     def do_POST(self):
         body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
+        if body.get("model") in ("busy-model", "busy-model-2"):
+            b = b'{"error":{"message":"rate-limited upstream","code":429}}'
+            self.send_response(429); self.send_header("Content-Type", "application/json"); self.end_headers(); self.wfile.write(b)
+            return
         if self.headers.get("Authorization") == "Bearer test-dummy-key":
             SEEN["auth_ok"] += 1
         last = body["messages"][-1]["content"]

tests/nl_check.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+"""Natural-language checks against the real hosted model. Usage: nl_check.py BASE_URL
+Each prompt should be answered by the LLM (not the fallback) with the expected panel."""
+import json, sys, time, urllib.request, urllib.error
+BASE = (sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8795").rstrip("/")
+fee_bps = json.load(urllib.request.urlopen(BASE + "/api/status"))["fee_bps"]
+CASES = [
+    ("I need something to get my kid to school, under $100", "show_cards", lambda p: any("kid" in l["title"].lower() for l in p["listings"]) and all(l["price"] <= 10000 for l in p["listings"])),
+    ("what can I get for my garden?", "show_cards", lambda p: any("lawn" in l["title"].lower() for l in p["listings"])),
+    ("list my old guitar for $120, cash only", "show_listing_form", lambda p: p["defaults"].get("price") == 12000 and p["defaults"].get("payment_options") == ["cash"] and "guitar" in p["defaults"].get("title", "").lower()),
+    ("compare the two tutoring options", "show_compare", lambda p: len(p["listings"]) == 2 and all("tutoring" in l["title"].lower() for l in p["listings"])),
+    ("I'd like to buy the ceramic lamp and pay online", "show_checkout", lambda p: p["intent"]["fee_cents"] == round(p["intent"]["subtotal_cents"] * fee_bps / 10000) and p["intent"]["fee_bps"] == fee_bps),
+    ("can I just pay cash for the lawn mowing when they come round?", "show_cash_meetup", lambda p: p["deal"]["fee_cents"] == 0),
+]
+ok_n, times, history = 0, [], []
+for prompt, want, pred in CASES:
+    t = time.time()
+    req = urllib.request.Request(BASE + "/api/chat", data=json.dumps({"message": prompt, "history": [], "actor": "guest-nl-test"}).encode(),
+                                 headers={"Content-Type": "application/json"})
+    try:
+        out = json.load(urllib.request.urlopen(req, timeout=90))
+    except urllib.error.HTTPError as e:
+        out = json.loads(e.read() or b"{}")
+    dt = time.time() - t
+    acts = [a for a in out.get("actions", []) if a["type"] == want]
+    good = out.get("agent") == "llm" and bool(acts) and pred(acts[0]["payload"])
+    ok_n += good; times.append(dt)
+    extra = ""
+    if acts and want == "show_checkout":
+        it = acts[0]["payload"]["intent"]; extra = f" price ${it['subtotal_cents']/100:.2f} fee ${it['fee_cents']/100:.2f} total ${it['total_cents']/100:.2f}"
+    if acts and want == "show_cash_meetup":
+        extra = f" cash fee ${acts[0]['payload']['deal']['fee_cents']/100:.2f}"
+    if acts and want in ("show_cards", "show_compare"):
+        extra = " -> " + " | ".join(l["title"] for l in acts[0]["payload"]["listings"])
+    if acts and want == "show_listing_form":
+        extra = " -> " + json.dumps(acts[0]["payload"]["defaults"])
+    print(f"{'PASS' if good else 'FAIL'} [{out.get('agent')}/{(out.get('model') or '').split('/')[-1]} {dt:.1f}s"
+          f"{' fb=' + ','.join(f['error'] for f in out.get('fallbacks', [])) if out.get('fallbacks') else ''}] {prompt!r}\n     {[a['type'] for a in out.get('actions', [])]}{extra}\n     reply: {out.get('reply', '')[:200]!r}")
+times.sort()
+print(f"\n{ok_n}/{len(CASES)} passed; median {times[len(times)//2]:.1f}s, max {times[-1]:.1f}s")
+sys.exit(0 if ok_n == len(CASES) else 1)