P
Back to Case Studies

Evaluation & Observability Stack for LLM Apps

Built a unified stack to evaluate, monitor, and improve LLM application quality in production — traces, metrics, and human feedback in one loop.

TracesMetricsFeedback SaaS / Technology
2.5x
Quality Improvement
60%
Faster Debugging
95%
Feedback Coverage

Business Problem

LLM applications degrade silently: latency dashboards stay green while answer quality falls. There was no way to localize a quality drop to query understanding, retrieval, ranking, or generation.

Architecture Overview

Distributed tracing across every pipeline stage, sampled LLM-judge scoring in production, human feedback wired to full traces, and a weekly quality review that turns failures into backlog items.

  1. 1 Stage-level tracing (OTel spans)
  2. 2 Version attributes (index, prompt, model)
  3. 3 Sampled LLM-judge scoring
  4. 4 Human feedback capture
  5. 5 Quality dashboards + alerts
  6. 6 Weekly failure clustering → backlog

All components are deployed with private networking, scoped IAM roles, and encryption at rest and in transit.

Key Design Decisions

  • Traces before metrics. Without stage-level tracing you cannot localize a quality drop; metrics alone say "something is wrong" without saying where.
  • Alert on trends, not thresholds. Rolling-baseline alerts (faithfulness −5% week-over-week) avoid crying wolf while catching drift.
  • Feedback tied to full traces. A thumbs-down is only actionable when it carries the query, retrieved chunks, prompt, and versions that produced it.

Technology Stack

OpenTelemetryPrometheusGrafanaLangSmithPython

My Role

  • Designed the tracing schema and version attributes.
  • Built the sampled judging pipeline and quality dashboards.
  • Set up alerting baselines and the weekly review process.

Measurable Impact

  • 2.5x Quality Improvement
  • 60% Faster Debugging
  • 95% Feedback Coverage

Challenges & How I Solved Them

  • Judge cost at scale: Sampling 1–5% of traffic with stratification by query class kept costs low with statistical power.
  • PII in traces: Field-level redaction before spans leave the boundary.
  • Alert fatigue: Trend-based baselines replaced brittle absolute thresholds.

Lessons Learned

  • Cost anomalies are often the first symptom of a bug.
  • The highest-ROI improvements come from clustering real failures.
  • Abstention-rate drops usually mean hallucinations are up.