Pick the model per task
Fast model for inline autocomplete, strong reasoning model for refactors and test generation — set the model per request from one catalog, no SDK swap.
Use Case · Code Generation
Coding assistants with model choice and per-seat budgets.
A coding tool mixes fast autocomplete with deep reasoning. Nemo Router lets you pick the model per task from one catalog, route latency-sensitive calls to the quickest endpoint, and cap spend per developer seat.
code-assistant · seat key
One catalog, the right model per task
Inline completiongemini-2.5-flash
Refactor / testsgemini-2.5-pro
Route strategylatency
Streamingtoken-by-token
Seat budget$31 / $80
Provider confignone
model choicelow-latencyper-seat budget
- Model catalog
- 97+
- Gateway overhead
- 95 ms
- Per-seat spend
- Budgeted
- Every call
- Logged
models on Anthropic, Google & OpenAI
p50 — LLM inference dominates
One key per developer, capped
Model, latency, and cost per request
Why Nemo for code
The right model, fast, and within budget
A coding assistant has two competing needs — speed for completion, depth for reasoning — and cost that scales with every seat. Nemo Router handles all three behind one key.
Low-latency completion
A developer is waiting on every keystroke. Latency-based routing steers each request to the model with the lowest recent p95 for your org; streaming is proxied token-by-token.
Per-seat budgets
One virtual key per developer with a hard 402 ceiling — cost-per-seat is a number, not a guess, and one seat can never hog capacity (per-key RPM/TPM).
Every generation logged
When a suggestion is wrong or slow, filter the request log by the developer’s key — model, latency, tokens, and real cost per call. A/B test two models on real traffic.
How it works
An editor request, end to end
Each developer carries a seat key. Completion and reasoning requests route to the model the task needs, stream back token-by-token, and land in the log attributed to that seat.
Code-assistant request flow
Editor request
completion or chat
A keystroke completion or a refactor prompt from the IDE.
Seat key
sk-nemo-... · per developer
Budget and rate limit scope to the developer’s key.
Model + latency route
catalog · latency
Fast model for completion, strong model for reasoning.
Stream back
token-by-token
Streaming proxied transparently to the editor.
Logged per seat
request log
Model, latency, tokens, cost — attributed to the seat.
The gateway adds about 95 ms at p50 — LLM inference is the dominant latency factor. Streaming is proxied with no hot-path buffering.
The code
Set the model per request
A coding assistant just sets the model field per call — fast for completion, strong for reasoning. These snippets come from the same SDK examples the playground uses; change the model string and the catalog does the rest.
Install
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash-lite", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | top_p=1, |
| 15 | messages=[ |
| 16 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 17 | ], |
| 18 | extra_body={ |
| 19 | # "nemo_cache": False, # Uncomment to skip cache |
| 20 | }, |
| 21 | ) |
| 22 | |
| 23 | print(response.choices[0].message.content) |
One key reaches every model in the catalog — no per-model provider account to manage.
FAQ
Common code-assistant questions
Model choice, low latency, per-seat budgets
Build a coding assistant your finance team can read
Pick the model per task, route for latency, and cap spend per developer — all unlocked on every plan.
OpenAI-compatible — works with any IDE extension that targets the OpenAI SDK.