Nemo Routernemorouter.aiWhat to Log (and Not Log) on an LLM Gateway
Request logs are your debugger and your liability at once. Here is a practical policy for LLM logging — what to capture, what to redact, how long to keep it, and the four logging levels to choose from.
Request logs are the first thing you reach for when something breaks and the last thing you want leaked. They're simultaneously your best debugging tool and a concentrated store of exactly the sensitive data you promised customers you'd protect. Getting LLM logging right is about resolving that tension deliberately — not logging everything and hoping, and not logging nothing and flying blind.
What's actually in an LLM log?
A single request log can contain the prompt (often user data), the completion (often generated about user data), the model, token counts, cost, latency, the key that made the call, and request metadata. Some of that is pure operational gold; some of it is a breach waiting to happen. The job is to separate them.
| Field | Value to you | Risk |
|---|---|---|
| Model, tokens, cost, latency | High — debugging + analytics | Low |
| Key id, org, timestamp | High — attribution | Low |
| Prompt content | High — reproduce bugs | High — user PII |
| Completion content | Medium — quality review | High — derived PII |
| Requester IP / headers | Medium — abuse triage | High — identifies users |
The insight: the low-risk fields carry most of the operational value. You can run cost, latency, and usage analytics on metadata alone. Content is only needed for a narrower set of tasks (reproducing a specific bug, quality review), and that's exactly the data you should think hardest about retaining.
The four logging levels
Rather than one global "logging on/off," NemoRouter exposes a content-logging policy with four levels, set per org:
- Zero — metadata only, no prompt/completion content stored. Maximum privacy; you keep cost/latency/usage.
- Metadata — structured fields, still no raw content.
- PII-redacted — content is stored after PII detection strips emails, phones, cards, SSNs.
- Full — everything, for teams who need complete reproduction and accept the trade.
The right default for most teams is PII-redacted or zero: you keep enough to operate, without warehousing raw customer data you'd rather not be holding when a subject-access or deletion request arrives.
A log is data you now own
Anything you log, you are now responsible for: securing it, retaining it lawfully, and deleting it on request. "We log full prompts forever" is a sentence that sounds harmless until it's in a breach disclosure. Log the minimum that keeps you operational.
Redact at write time, not read time
PII redaction has to happen before the log row is written. Redacting on display is theater — the raw value is already at rest in your store, discoverable by anyone with database access and exposed in any backup. NemoRouter's redaction runs in the logging path, so a PII-redacted policy means the sensitive substring never lands in the log table at all. (Same principle as the inline guardrails on the request path.)
This also covers a subtle leak: request metadata can carry the requester's IP. For non-admins, that field is stripped from the metadata copy too — there's no point redacting the obvious field and shipping the same value in a sidecar.
How long should you keep logs?
Match retention to purpose:
- Operational logs (debugging, abuse triage): short — days to a few weeks. After that they're risk without reward.
- Usage / cost aggregates: keep the rollups long-term (they're not sensitive), drop the raw rows.
- Financial records: retained per your accounting/compliance obligations — but these are ledger entries, not prompt content.
NemoRouter defaults to a bounded retention on request logs and keeps financial records per obligation. The principle: keep aggregates, expire raw content.
Reading logs without re-leaking
The Request Logs view is paginated, searchable, and filterable by model/status/time — built so you can find the one failing request without dumping the whole table. Access is role-scoped, and content visibility follows the logging policy. You can debug a 500 on a specific key without that workflow exposing every customer's prompts to every viewer.
The takeaway
Log the metadata generously and the content carefully. Default to zero or PII-redacted, redact at write time so the secret never lands, expire raw content while keeping aggregates, and scope access by role. Done this way, request logs stay what they should be — a debugger you're glad to have — instead of a liability you forgot you were carrying. Configure your policy in Data Policy.