A voice app lives or dies on latency. Nemo Router steers each conversational turn to the quickest healthy endpoint, keeps the line alive with transparent failover, and logs every turn for observability.
One turn of a live conversation
p50 added — LLM inference dominates
Steered to the quickest healthy endpoint
The caller never hears an error
p50 / p99 latency per model
Voice is the least forgiving LLM workload — every turn is on a clock and a caller is listening. Nemo Router gives that turn fast routing, transparent failover, and observability.
A voice app lives or dies on response time — silence on the line is the failure mode. Latency-based routing steers each conversational turn to the quickest healthy endpoint using live signal.
A provider blip during a live call cannot be a dead air moment. The fallback chain retries the next link transparently so the turn still completes and the caller hears a response, not an error.
When a call feels slow, you need the turn. Every turn lands in the request log with the model, latency, token counts, and cost — and latency metrics surface p50 and p99 per model.
Realtime assistants want a fast, capable model — and the freedom to switch as faster models ship. The catalog exposes every model behind one key; pick the one with the latency profile you need.
Nemo Router is the LLM hop between transcription and synthesis. The turn routes for latency, streams back token-by-token, and lands in the log with p50 / p99 metrics.
Voice turn flow
Caller speaks
speech-to-text upstream
Your speech layer transcribes the turn to text.
Turn request
POST /v1/chat/completions
The transcript becomes a streaming chat request.
Latency route
quickest healthy endpoint
Live latency signal picks the fastest model deployment.
Stream the reply
token-by-token
Tokens flow to your text-to-speech with no buffering.
Turn logged
latency metrics
Model, latency, cost per turn — p50 / p99 tracked.
Your speech-to-text and text-to-speech layers stay yours. Nemo Router gives the reasoning turn in between low-latency routing, failover, and a logged record.
Latency-based routing
A model deployment that was fastest a minute ago may not be now. Latency-based routing uses live latency signal to pick the quickest healthy endpoint for each turn. The gateway itself adds about 95 ms at p50 — the dominant factor is always LLM inference, never the proxy.
Turn-by-turn latency
A voice turn is a streaming chat request — transcript in, tokens out to your speech layer. These snippets come from the same SDK examples the playground uses; enable streaming and tokens flow as they generate.
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | top_p=1, |
| 15 | messages=[ |
| 16 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 17 | ], |
| 18 | extra_body={ |
| 19 | # "nemo_cache": False, # Uncomment to skip cache |
| 20 | }, |
| 21 | ) |
| 22 | |
| 23 | print(response.choices[0].message.content) |
Set stream: true — Nemo Router proxies the token stream with no hot-path buffering.
Fast turns, no dead air
Latency-based routing, transparent failover, and turn-level observability — all unlocked on every plan.