LLMOps in Production: Evals, Tracing, Cost and Guardrails

LLM applications break the assumptions of classic MLOps. You usually do not train the model; you assemble behaviour from a prompt, retrieved context, tools and a hosted or self-hosted model that can change underneath you. Outputs are free text, so "accuracy" needs an evaluation design. Each request has a direct, variable cost in tokens, and an agent can make dozens of model and tool calls to answer one question. LLMOps is the set of practices that makes such systems reliable, measurable and affordable: versioning prompts like code, regression-testing them, tracing every step, tracking cost and latency, guarding inputs and outputs, and learning from users.

Intuition: running a call centre staffed by brilliant temps

Picture a call centre whose agents are extremely capable temporary staff. Each one is briefed with a written script (the prompt), a binder of documents (retrieved context) and a phone list (tools). They can be swapped for a different agency's staff overnight (a model upgrade). You would want: version control on the scripts, a test set of tricky calls to rehearse before any change, call recordings to review what happened (traces), a cost meter per call, rules about what may never be said (guardrails), a backup agency for when the first one is overloaded (fallbacks), and a way for customers to rate the call (feedback). That list is this lesson.

Prompts and configuration as versioned artifacts

In an LLM app, the "model" that users experience is the combination of:

• model identifier, pinned to a dated snapshot or version where the provider offers one, not a floating alias;
• system prompt and templates;
• decoding parameters (temperature, max tokens, stop sequences);
• tool definitions and schemas;
• retrieval configuration (embedding model, index version, chunking, top-k);
• guardrail and routing configuration.

Treat that bundle as a single versioned release. Store prompts in the repository (or in a prompt registry that records immutable versions), review changes in pull requests, give each bundle a version ID, and log that ID with every request. Then any production trace can be reproduced, and any regression can be traced to the change that caused it. A prompt edit is a code change: it can break things, so it goes through the same evaluation gate.

Evaluation pipelines and regression tests

An eval suite is a versioned dataset of inputs with a way to grade outputs. Build it from three sources: hand-written cases covering the product's core tasks and policies, production traces (especially failures and user complaints), and adversarial cases from red-teaming. Grade with the cheapest reliable method:

Zheng et al. (2023) showed that strong LLM judges can agree with human preferences at levels comparable to agreement between humans on their benchmark, while documenting position and verbosity biases. Shankar et al. (2024) found that evaluation criteria themselves shift as people grade outputs. The practical lesson: write grading rubrics as binary or small-scale questions, measure judge agreement with human labels on a sample, and revisit the rubric as you learn.

Run the suite in CI on every change to the release bundle, and on a schedule against production (to catch provider-side model changes). Block the release if a must-pass subset (safety, policy, critical workflows) has any failure, or if the overall score drops by more than you can attribute to noise.

Worked example: is a regression real?

A 200-case suite. The current release passes 172 cases (86%); a candidate with a shorter, cheaper prompt passes 164 (82%). Is it worse, or is this noise? Because both versions ran on the same cases, use a paired test. Of the 200 cases, 14 passed before and fail now; 6 failed before and pass now; the rest are unchanged ($172 - 14 + 6 = 164$, consistent).

McNemar's test looks only at the $b = 14$ and $c = 6$ discordant cases. Under the null hypothesis of no difference, each discordant case is equally likely to go either way, so the smaller count follows a Binomial$(20, 0.5)$. The exact two-sided p-value is

$p = 2 \times P(X \le 6) = 2 \times \frac{\sum_{k=0}^{6} \binom{20}{k}}{2^{20}} = 2 \times \frac{60{,}460}{1{,}048{,}576} = 0.115.$

So the 4-point drop is not statistically significant at the 5% level. But it is not reassuring either: read the 14 newly failing cases. If several share a cause (say, the shorter prompt lost an instruction about refunds), it is a real regression the suite is too small to prove. Two lessons: grow the suite where failures cluster, and also run each case several times at your production temperature, because LLM outputs vary between runs.

Tracing and observability

A single user request in an agentic app might involve query rewriting, a retrieval call, three LLM calls, two tool calls and a guardrail check. When the answer is wrong or slow, you need the full tree. Tracing records each step as a span (name, start and end time, attributes, parent span), and the spans of one request form a trace.

OpenTelemetry (OTel) is the open standard for traces, metrics and logs. Its GenAI semantic conventions define standard attribute names for model calls, agent steps and tool executions, so that different instrumentation libraries and observability back ends can understand each other's data. As of mid-2026 every GenAI convention was still marked "Development" rather than stable, and attribute names have changed between versions (for example, an earlier gen_ai.system attribute was replaced by gen_ai.provider.name). Check the current specification before you build dashboards on specific names. Attributes you will typically find include:

• gen_ai.operation.name (for example chat, embeddings, execute_tool, invoke_agent);
• gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model;
• gen_ai.request.temperature, gen_ai.request.max_tokens;
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens;
• gen_ai.response.finish_reasons.

Capturing full prompt and completion text is opt-in in the conventions, because it can contain personal or confidential data. Decide deliberately what to record, redact sensitive fields, and set retention limits.

import time
from opentelemetry import trace

tracer = trace.get_tracer("support-bot")
RELEASE = "support-bot@2026-11-03.2"              # prompt+model+config bundle version

def traced_chat(client_call, model, messages, **params):
    # client_call: your provider SDK call returning (text, input_tokens, output_tokens)
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("app.release", RELEASE)
        for k in ("temperature", "max_tokens"):
            if k in params:
                span.set_attribute(f"gen_ai.request.{k}", params[k])
        t0 = time.perf_counter()
        text, n_in, n_out = client_call(model=model, messages=messages, **params)
        span.set_attribute("gen_ai.usage.input_tokens", n_in)
        span.set_attribute("gen_ai.usage.output_tokens", n_out)
        span.set_attribute("app.latency_ms", (time.perf_counter() - t0) * 1000)
        return text

This uses only the stable OTel tracing API; without an SDK and exporter configured, the calls are harmless no-ops. Many providers' SDKs and agent frameworks have instrumentation packages that emit these attributes automatically, and several LLM observability products accept OTel data.

Key metrics to derive from traces: requests, errors and refusals per release; latency split into time to first token (TTFT, what users feel in streaming UIs) and total time; tokens and cost per request, per user and per feature; tool-call counts and failure rates; agent step counts (runaway loops show up here); cache hit rate; guardrail trigger rates; and user feedback rates.

Cost and latency engineering

Cost per request is

$C = n_{\text{in}} \cdot p_{\text{in}} + n_{\text{out}} \cdot p_{\text{out}},$

where $n_{\text{in}}, n_{\text{out}}$ are input and output token counts and $p_{\text{in}}, p_{\text{out}}$ the per-token prices. Output tokens usually cost several times more than input tokens and are generated sequentially, so they dominate latency too.

Worked example: prompt caching

Illustrative prices (not any specific provider's): $3 per million input tokens, $15 per million output tokens, and cached input read at 10% of the normal input price. A support assistant sends 4,000 input tokens per request, of which a 3,000-token prefix (system prompt plus tool definitions) is identical for every request, and generates 500 output tokens.

Without caching:

• Input: $4{,}000 \times \text{\textdollar}3/10^6 = \text{\textdollar}0.0120$
• Output: $500 \times \text{\textdollar}15/10^6 = \text{\textdollar}0.0075$
• Total: $0.0195 per request; $19,500 per million requests.

With the 3,000-token prefix cached:

• Cached input: $3{,}000 \times \text{\textdollar}0.30/10^6 = \text{\textdollar}0.0009$
• Uncached input: $1{,}000 \times \text{\textdollar}3/10^6 = \text{\textdollar}0.0030$
• Output: $0.0075
• Total: $0.0114 per request; $11,400 per million, a 42% saving.

Caching also cuts time to first token, because the provider skips recomputing the cached prefix. Real pricing differs by provider: some charge extra to write the cache, caches expire after minutes, and the prefix must match exactly, so put stable content first and variable content (the user's question, retrieved chunks) last.

Other levers:

• Response (semantic) caching: return a stored answer for an identical or near-identical request. Effective for FAQs; risky when answers depend on user-specific data or change over time. Key on the release version and scope by user or tenant.
• Model routing: send easy requests to a small, cheap model and hard ones to a large model (see the routing lesson).
• Trimming context: fewer, better retrieved chunks; summarised conversation history.
• Capping output: sensible max_tokens, concise output formats.
• Batch APIs for non-urgent work, which some providers offer at a discount.

Guardrails, rate limits and fallbacks

Guardrails are checks around the model: input checks (prompt-injection heuristics, PII detection, topic restrictions) and output checks (schema validation, policy classifiers, grounding checks against retrieved sources, secret and PII redaction). Prefer deterministic validation where possible, and remember that no classifier stops prompt injection completely; the stronger defence is limiting what the model can do (least-privilege tools, human confirmation for consequential actions). The structured-outputs lesson goes deeper; the OWASP Top 10 for LLM Applications is a useful threat checklist.

Rate limits exist at the provider (requests and tokens per minute) and should exist in your own app (per user and per tenant, to stop abuse and runaway costs). Add per-request budgets for agents: maximum steps, maximum tokens, maximum wall-clock time.

Fallbacks keep the product working when a provider is slow or down:

import random, time

class TransientError(Exception):
    pass                                        # map 429s, 5xx and timeouts to this

def call_with_fallback(providers, request, max_retries=3, base_delay=0.5):
    # providers: ordered list of (name, fn); fn(request) -> response or raises
    for name, fn in providers:
        for attempt in range(max_retries):
            try:
                return name, fn(request)
            except TransientError:
                delay = base_delay * (2 ** attempt) * random.uniform(0.5, 1.5)
                time.sleep(delay)               # exponential backoff with jitter
    raise RuntimeError("all providers failed")  # degrade gracefully upstream

Fallback models must be evaluated too: run the eval suite against the fallback path, because a different model may need a different prompt. Respect Retry-After headers when providers send them, set timeouts on every call, and use a circuit breaker so a failing provider is skipped for a while rather than retried by every request.

Human feedback

Collect explicit feedback (thumbs up/down, a reason picker, free-text comments) and implicit signals (the user copies the answer, edits a draft heavily, retries, abandons, escalates to a human agent). Log feedback with the trace ID and release version. Use it to: find failure clusters, add failing cases to the eval suite, calibrate LLM judges, and, where you fine-tune, build preference data. Feedback is biased (unhappy users click more often, most users click nothing), so do not treat the thumbs-up rate as a quality metric on its own; pair it with sampled expert review.

Red-teaming cadence

Red-teaming is deliberately trying to make the system fail: jailbreaks, prompt injection through retrieved documents or tool outputs, data exfiltration, harmful or off-policy content, and abuse of tools. A workable cadence: a focused exercise before launch; automated adversarial suites in CI on every release (model-generated attacks, as in Perez et al., 2022, scale this); a broader manual exercise each quarter or at any major change (new model, new tool, new data source); and immediate additions to the suite after any incident. NIST's Generative AI Profile (NIST AI 600-1) lists risk categories you can map tests to.

A production readiness checklist

• Release bundle pinned: model snapshot, prompts, tools, retrieval and parameters, with a version ID logged on every request.
• Eval suite in CI with must-pass cases, a significance-aware comparison, and scheduled runs against production.
• LLM judges validated against human labels.
• Tracing across every model, retrieval and tool call; content capture decided, redaction and retention set.
• Dashboards and alerts on errors, p95 latency and TTFT, cost per request and per tenant, agent step counts, guardrail triggers.
• Budgets: per-request token and step limits; per-user and per-tenant rate limits; spend alerts.
• Guardrails: schema validation, policy checks, least-privilege tools, confirmations for consequential actions.
• Resilience: timeouts, retries with backoff, circuit breaker, an evaluated fallback model, and a graceful degraded mode.
• Caching strategy with correct scoping.
• Feedback capture tied to traces, with a weekly triage loop into the eval suite.
• Red-team results reviewed; incident runbook and owner on call.
• Documentation: a model or system card covering intended use and limits.

Common pitfalls

• Using a floating model alias, so behaviour changes without a deploy.
• "Vibe checks" instead of an eval suite; changing prompts directly in production.
• Trusting an LLM judge that nobody has compared with human ratings.
• Logging full prompts with personal data indefinitely, or logging nothing and being unable to debug.
• Reading average latency instead of p95 and TTFT.
• Untested fallback paths that produce worse or unsafe answers.
• Agents without step or token budgets that loop and burn money.
• Semantic caches that serve one user's personalised answer to another.

In practice

Start small: pin the model, put prompts in version control, write 50 good eval cases, and instrument every call with OpenTelemetry. Grow the suite from production failures every week. Add caching and routing once you can measure cost per request. Review dashboards for cost, latency and quality per release, and gate every change, including "just a prompt tweak", on the eval suite.