An autonomous agent fans out dozens of LLM calls per task. Nemo Router keeps a long run alive when a provider degrades, caps each agent with a per-key budget, and logs every tool call for audit.
One agent, one key, one budget
Fallback chains survive a provider blip
A 402 stops a runaway loop
Every step in the request log
models on Google Vertex AI
Agents are unpredictable by design. The gateway makes them dependable: a run that survives a provider blip, a budget it cannot exceed, and a log of everything it did.
Agents fan out dozens of LLM calls per task. One provider hiccup halfway through a run shouldn’t kill it. The fallback chain retries the next link transparently so the agent keeps going.
A buggy agent loop can burn credits fast. Give each agent its own virtual key with a per-key budget — when it runs out, the gateway returns a 402 instead of spending without a ceiling.
When an agent does something surprising, you need the trail. Every LLM call — planning, tool selection, reflection — lands in the request log with model, latency, cost, and result.
Planning wants a strong reasoning model; a quick tool-argument extraction wants a fast one. Pick the model per call from the catalog — no provider account, no SDK swap.
Each agent carries its own virtual key. Every planning and tool-selection step is a separate LLM call — routed, failover-protected, budget-checked, and logged.
Agent run flow
Agent key
sk-nemo-... · per agent
One virtual key per agent — its own budget and rate limit.
Plan + tool calls
many /chat/completions
Each reasoning + tool-selection step is a separate call.
Routing + failover
fallback chain
A degraded provider triggers the next link — the run survives.
Budget check
reserve + settle
Out of budget → 402, before the call. No runaway spend.
Logged per step
request log
Model, latency, cost, result — replay the whole run.
The budget check is a reserve-then-settle: credits are reserved before the call and settled at the real cost after. If an agent is out of budget, it gets a 402 — never a surprise overspend.
Fallback chains
The longer the run, the higher the chance one call hits a degraded provider. With an ordered fallback chain per model group, a 5xx or timeout triggers the next link transparently. The agent loop keeps iterating; the failover is recorded for you to review later.
Fallback during a run
Nemo Router speaks the OpenAI API, so any agent framework that targets the OpenAI SDK works with a base-URL and key change. These snippets are generated from the same SDK examples the playground uses — give each agent its own key and the per-key budget does the rest.
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | top_p=1, |
| 15 | messages=[ |
| 16 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 17 | ], |
| 18 | extra_body={ |
| 19 | # "nemo_cache": False, # Uncomment to skip cache |
| 20 | }, |
| 21 | ) |
| 22 | |
| 23 | print(response.choices[0].message.content) |
Issue one virtual key per agent — spend, rate limits, and the request log all scope to that key.
Reliable, bounded, auditable
Auto-failover, per-agent budgets, and a full tool-call log — unlocked on every plan, behind one NemoRouter key.