Observability for LLM Systems: Metrics That Matter

LLM applications fail differently from normal software: they degrade silently. Latency dashboards stay green while answer quality falls off a cliff. Here are the metrics that actually matter for LLM systems, and how to track them.

The Three Layers of LLM Observability

1. System metrics — latency, error rate, throughput (necessary, not sufficient)
2. Model metrics — token usage, cost, model/prompt versions serving traffic
3. Quality metrics — faithfulness, relevance, citation correctness, refusal rate

Most teams stop at layer one. The incidents that hurt live in layer three.

Traces Before Metrics

A single LLM request touches query understanding, retrieval, ranking, prompt assembly, generation, and post-processing. Without distributed tracing (OpenTelemetry works fine), you cannot localize a quality drop. Every span should record its inputs' fingerprints: index version, prompt template version, model ID, retrieval parameters.

with tracer.start_as_current_span("retrieval") as span:
    span.set_attribute("index.version", index.version)
    span.set_attribute("retrieval.k", k)
    hits = retriever.search(query, k=k)
    span.set_attribute("retrieval.hit_count", len(hits))

Quality Metrics Worth Tracking

• Faithfulness — is the answer supported by the retrieved context? (LLM-judge on a sampled % of traffic)
• Context precision/recall — did retrieval surface the right passages?
• Citation correctness — do cited sources actually contain the claim?
• Abstention rate — how often does the system correctly say "I don't know"? A sudden drop usually means hallucinations are up.
• Human feedback rate and reasons — thumbs down clustered by failure type.

Cost as an Observability Signal

Track tokens and cost per request, per tenant, per pipeline stage. Cost anomalies are often the first symptom of a bug: a runaway agent loop, a context window that doubled, a retry storm.

Alerting That Doesn't Cry Wolf

Alert on trends against baselines, not absolute thresholds: faithfulness 7-day rolling average dropping >5%, abstention rate halving overnight, cost per request doubling. Route quality alerts to whoever owns prompts and indexes — not just on-call infra.

The Weekly Quality Review

Metrics find regressions; humans find causes. A 30-minute weekly review of the worst-rated traces — clustered by query class — consistently produces the highest-ROI backlog items: a parsing bug here, a missing document source there, a prompt instruction that conflicts with a new model version.

Minimal Viable Setup

1. OpenTelemetry tracing across all stages with version attributes
2. Sampled LLM-judge scoring (1–5% of traffic) for faithfulness and relevance
3. Cost per request dashboard segmented by tenant
4. User feedback wired to full traces
5. Golden-dataset evaluation on every deploy

Track what truly matters, and your LLM system stops being a black box and starts being an engineering system you can improve deliberately.