Router Settings
Choose a routing strategy, set fallbacks, retries, caching, and per-model weights
Last updated
Router Settings (/router-settings) controls how NemoRouter picks a deployment for each request and what it does when one fails. It's available to owners and admins.
Routing strategy
Point a router alias at a set of candidate models and choose how each request resolves to one of them:
| Strategy | Behavior |
|---|---|
| Cost | Routes to the cheapest model in the set (by your list price) |
| Latency | Routes to the model with the lowest recent p95 latency for your org |
| Weighted | Splits traffic across the set by the weights you assign each model |
Each alias is per-org and resolves at request time — change the set or the strategy in the dashboard, no redeploy. A concrete model you name directly is never re-routed (routing is opt-in).
Fallback chains
Define what happens when a model fails. There are two trigger types, each with its own ordered chain:
- On Error — fires on 5xx, network errors, or generic exceptions
- On Rate Limit — fires on HTTP 429 or context-window-exceeded
For each chain, set a primary model and an ordered list of fallback models. Reorder by priority (drag or arrow keys) and delete chains you no longer need. Duplicate primaries within the same trigger type are rejected.
Retries & timeouts
| Setting | Scope | Default |
|---|---|---|
| Number of retries | per org, per-request override | 3 |
| Timeout (seconds) | per org, per-request override | 60 |
On a retry, the provider's retry-after hint is honored.
Per-model weights
For nemo-auto requests, assign each model a 0–100% weight to control the traffic split. The header shows the running total and flags it when the weights don't sum to 100%. Set weights with a slider or numeric input per model.
Response caching
Caching is on by default. Override it per request with the nemo_cache: false header.
Saving changes
Changes aren't live until you click Save Changes. An "unsaved changes" badge appears while the form is dirty, and Reset reverts to the last saved state after a confirmation. Saved changes reach live inference within about 30 seconds.
Next steps
- Model Catalog — The models you route between
- A/B Testing — Split traffic across variants for experiments
- Analytics & Reports — Benchmark models to inform your strategy
FAQ
Who on my team can change router settings?
Router Settings is available to owners and admins. Members and viewers see the page read-only — the Save and edit controls are disabled with a "Requires admin or owner" hint, so they can review the configuration but not change it.
Do I have to change my application code to use routing?
No. Routing is opt-in: point a router alias at a set of candidate models and call that alias. A concrete model you name directly is never re-routed. You change the model set or strategy in the dashboard and it takes effect at request time — no redeploy and no SDK change.
What's the difference between the Cost, Latency, and Weighted strategies?
Cost routes each request to the cheapest model in the set by your list price. Latency routes to the model with the lowest recent p95 latency for your org. Weighted splits traffic across the set by the percentage weight you assign each model.
What happens when a model fails mid-request?
Failover is automatic — you don't have to configure anything. If the model you asked for fails, NemoRouter retries your request on an equivalent model from a different provider and returns that result. Nothing changes in your code, and the response tells you it happened via the x-nemo-model-fallback header (requested->served). A same-provider retry would just fail with the provider, so the default always crosses providers.
You can still define your own fallback chains — a primary model and an ordered list of fallbacks — when you want specific models in a specific order. Your chain replaces the default for that model.
Two triggers are available when you define a chain: On Error (5xx, network errors, generic exceptions) and On Rate Limit (HTTP 429 or context-window-exceeded). Note that both chains are combined and tried on any failure — they are not isolated per trigger. If you need a model tried only on rate limits and never on errors, that isn't expressible today; list it in whichever chain you prefer and expect it to be attempted on both.
How many times will a request retry, and can I change it?
The defaults are 3 retries and a 60-second timeout, both set per org in Router Settings. They are not overridable per request — num_retries, timeout, and fallbacks sent in a request body are ignored, so your org's policy always wins and one caller can't opt out of it. When a provider returns a retry-after hint, NemoRouter honors it.
Retries happen against the requested model first; the fallback engages once they're exhausted. With the defaults that means a hard provider outage can hold a request for up to ~60 seconds before the fallback serves. Lower timeout in Router Settings if you'd rather fail over sooner.
How do the per-model weights work?
Per-model weights apply to nemo-auto requests: assign each model a 0–100% value to control the traffic split. The header shows the running total and flags it when the weights don't add up to 100%, so you can fix it before saving.
Does picking the Cost strategy actually lower my bill?
The Cost strategy sends each request to the cheapest eligible model by list price, so you're billed for whichever model actually runs, reported on the x-nemo-request-cost response header. Router Settings only decides which model handles a request — your spend ceilings live separately under Budget Controls and are enforced independently.
Is response caching on by default, and how do I turn it off for a request?
Yes, caching is on by default. To skip the cache for a specific request, send the nemo_cache: false header on that call; the org-wide default stays unchanged.
Do my changes take effect the moment I edit them?
No — edits aren't live until you click Save Changes, and an "unsaved changes" badge appears while the form is dirty. Reset reverts to the last saved state after a confirmation. Once saved, the new strategy, fallbacks, retries, and weights resolve at request time (within about 30 seconds) with no redeploy.