leora/agent-market
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Rate-limit only LLM-bound chat turns' (#3) from fix/rate-limit-llm-only into main

Browse Source
This commit is contained in:
leora 2026-10-08 19:32:09 +08:00
commit ad29b9c93a
3 changed files with 8 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -60,7 +60,7 @@ key stays on the server: it never goes to the browser, never into the repo, and
| `AGENT_MARKET_LLM_MODELS` | comma-separated fallback chain, tried in order | | `AGENT_MARKET_LLM_MODELS` | comma-separated fallback chain, tried in order |
| `AGENT_MARKET_LLM_TIMEOUT` / `_BUDGET` | hard cap per model attempt (12 s) and for the whole chain (28 s) | | `AGENT_MARKET_LLM_TIMEOUT` / `_BUDGET` | hard cap per model attempt (12 s) and for the whole chain (28 s) |
| `AGENT_MARKET_LLM_COOLDOWN` | how long a model that returned 429 gets skipped (60 s) | | `AGENT_MARKET_LLM_COOLDOWN` | how long a model that returned 429 gets skipped (60 s) |
| `AGENT_MARKET_CHAT_RPM` / `_BURST` | per-IP token bucket on `/api/chat` (12/min, burst 6). `CF-Connecting-IP` is trusted only from loopback, i.e. the tunnel. | | `AGENT_MARKET_CHAT_RPM` / `_BURST` | per-IP token bucket on `/api/chat` turns that reach the LLM (12/min, burst 6). Deterministic UI events aren't limited. `CF-Connecting-IP` is trusted only from loopback, i.e. the tunnel. |
`deploy.sh` loads `~/.config/agent-market/llm.env` and `~/.config/agent-market/<env>.env` if they exist. Both `deploy.sh` loads `~/.config/agent-market/llm.env` and `~/.config/agent-market/<env>.env` if they exist. Both
sit outside the repo. The box runs OpenRouter free models: sit outside the repo. The box runs OpenRouter free models:

4
server.py
View File

@ -312,7 +312,9 @@ class H(BaseHTTPRequestHandler):
return self._json(200, self._webhook(body)) return self._json(200, self._webhook(body))
actor = self._actor(body) actor = self._actor(body)
if path == "/api/chat": if path == "/api/chat":
ok, retry = rate_ok(self._client_ip()) # Only free-text turns reach the LLM; deterministic UI events (clicks, payments) are not limited.
llm_bound = not isinstance(body.get("event"), dict) or A.LLM_EVENTS
ok, retry = rate_ok(self._client_ip()) if llm_bound else (True, 0)
if not ok: if not ok:
return self._json(429, {"error": "rate_limited", "retry_after": retry, "agent": "rate-limit", return self._json(429, {"error": "rate_limited", "retry_after": retry, "agent": "rate-limit",
"reply": f"You're sending messages quickly. Please wait about {retry}s and try again.", "reply": f"You're sending messages quickly. Please wait about {retry}s and try again.",

6
tests/llm_proxy_check.sh
View File

@ -29,6 +29,8 @@ assert d["agent"]=="fallback" and d["llm_error"]=="all_models_failed" and d["act
print("PASS all models rate-limited -> rule-based agent still renders", [a["type"] for a in d["actions"]])' print("PASS all models rate-limited -> rule-based agent still renders", [a["type"] for a in d["actions"]])'
curl -s localhost:18800/api/status | python3 -c ' curl -s localhost:18800/api/status | python3 -c '
import json,sys; a=json.load(sys.stdin)["agent"]; assert set(a["cooling_down"])=={"busy-model","busy-model-2"}, a; print("PASS 429ed models put on cooldown")' import json,sys; a=json.load(sys.stdin)["agent"]; assert set(a["cooling_down"])=={"busy-model","busy-model-2"}, a; print("PASS 429ed models put on cooldown")'
CODES=$(for i in 1 2 3 4 5; do curl -s -o /dev/null -w '%{http_code} ' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.9' -d '{"event":{"type":"landing"}}'; done) CODES=$(for i in 1 2 3 4 5; do curl -s -o /dev/null -w '%{http_code} ' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.9' -d '{"message":"show me bikes"}'; done)
OTHER=$(curl -s -o /dev/null -w '%{http_code}' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.10' -d '{"event":{"type":"landing"}}') OTHER=$(curl -s -o /dev/null -w '%{http_code}' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.10' -d '{"message":"show me bikes"}')
EV=$(curl -s -o /dev/null -w '%{http_code}' localhost:18801/api/chat -H 'Content-Type: application/json' -H 'CF-Connecting-IP: 203.0.113.9' -d '{"event":{"type":"landing"}}')
[[ "$EV" == "200" ]] && echo "PASS UI events (no LLM call) not rate-limited" || { echo "FAIL event limited: $EV"; exit 1; }
[[ "$CODES" == "200 200 200 429 429 " && "$OTHER" == "200" ]] && echo "PASS per-IP rate limit (burst 3: $CODES; other IP $OTHER)" || { echo "FAIL rate limit: $CODES / $OTHER"; exit 1; } [[ "$CODES" == "200 200 200 429 429 " && "$OTHER" == "200" ]] && echo "PASS per-IP rate limit (burst 3: $CODES; other IP $OTHER)" || { echo "FAIL rate limit: $CODES / $OTHER"; exit 1; }