Scope
How you operate a GenAI app after the first demo works: the lifecycle as a closed loop (scope → prototype → evaluate → deploy → monitor → iterate), the three observability pillars (traces/logs/metrics) adapted for LLMs via the OpenTelemetry GenAI semantic conventions (
gen_ai.*), prompt/completion capture and the privacy rules around it, token/cost tracking, eval-in-production, drift, prompt versioning, CI/CD for prompts, and the tooling landscape. This is “keep the system honest in production”; for measuring quality see 13 - Evaluation & Evals, for spend/latency see 15 - Cost, Latency & Performance.
The lifecycle is a loop, not a line — evals and production monitoring feed signal back into scope and prototype.
Related: Gen AI Engineer, 13 - Evaluation & Evals, 15 - Cost, Latency & Performance, 17 - Deployment & Serving
1. The concept: LLMOps is MLOps with a non-deterministic, opaque component
Classic MLOps assumes a model you trained: you own the weights, version the dataset, retrain on drift, and a fixed input gives a fixed output. A GenAI app inverts most of that. The “model” is usually a vendor API you don’t control, the output is non-deterministic (same prompt, different text), quality is subjective (no single label to check against), and your real “code” is often a prompt + context + tool wiring that changes faster than any model. LLMOps is the discipline that makes this operable: observe every call, evaluate quality continuously, version the prompts, and watch for drift in behavior — not just in metrics.
The unit of work shifts from “a prediction” to “a trace” — a tree of LLM calls, retrievals, and tool executions that together produce one user-facing answer. Observability has to capture that whole tree, because the bug is usually between the calls (wrong context retrieved, tool returned junk, agent looped), not inside any single token.
Mental model
MLOps watches a number drift (accuracy, AUC). LLMOps watches a behavior drift — and behavior only shows up if you captured the trace, scored the output, and can replay it against a versioned prompt. No traces, no LLMOps.
2. The lifecycle as a loop
| Stage | What you do | What feeds back |
|---|---|---|
| Scope | define task, success criteria, guardrails, budget | production failure modes reshape scope |
| Prototype | prompt + context + model + tools; get it working once | — |
| Evaluate | offline evals on a dataset (LLM-judge, rules, human) — see 13 - Evaluation & Evals | eval gaps reveal missing prototype features |
| Deploy | ship behind flags / staged rollout — see 17 - Deployment & Serving | — |
| Monitor | traces, cost, latency, online evals, drift alerts | the core feedback source |
| Iterate | new prompt version / model / retrieval → back to Evaluate | closes the loop |
The line that separates teams that ship from teams that thrash is whether the loop is instrumented. If production captures traces and scores them, monitoring feeds scope directly: you see the actual failure modes (ambiguous queries, missing context, jailbreaks) and your next iteration targets them. Without instrumentation the loop is open — you ship, users complain, you guess.
Evals and monitoring are the same signal at different times
An offline eval runs your dataset through a candidate before deploy. An online eval scores live traffic after deploy. Same scorers, same prompt-version key — so a regression caught in production can be pinned to the exact prompt change that caused it and re-tested offline.
3. The three pillars, adapted for LLMs
Standard observability is traces, logs, metrics. For GenAI each gains LLM-specific content:
| Pillar | Generic meaning | GenAI-specific content |
|---|---|---|
| Traces | request flow across services | the call tree: invoke_agent → chat → execute_tool → retrieval; nested spans with model, params, token counts |
| Logs | discrete events | the actual prompt/completion message content (opt-in — see §5), tool arguments + returns |
| Metrics | aggregatable numbers | tokens in/out, cost, latency (incl. time-to-first-token), error/refusal rate, online eval scores |
The trace is the high-value pillar for LLMs because a single user turn fans out into many model and tool calls, and the failure is usually in the shape of that tree.
4. OpenTelemetry GenAI semantic conventions (gen_ai.*)
OpenTelemetry (OTel) is the vendor-neutral standard for emitting telemetry. The GenAI semantic conventions define a common vocabulary — gen_ai.* attribute names — so a trace looks the same whether it lands in Langfuse, Phoenix, Datadog, or your own collector. Instrument once at the app layer, swap backends freely. That portability is the whole pitch.
HIGH CHURN — the semconv is still "Development", not stable
As of 2026 the GenAI semantic conventions carry a Development stability badge — not stable. Attribute names are actively changing and the spec was even moved out of the main semantic-conventions repo into its own
semantic-conventions-genairepo. Concrete breakages already shipped:
gen_ai.system→ deprecated, replaced bygen_ai.provider.name.gen_ai.prompt/gen_ai.completion(and the oldgen_ai.{role}.messageevents) → deprecated in favor ofgen_ai.input.messages/gen_ai.output.messages.Pin your instrumentation version, use the
OTEL_SEMCONV_STABILITY_OPT_INenv var (e.g.gen_ai_latest_experimental) to control which version you emit, and expect attribute renames between releases. Don’t hard-code dashboard queries against names you can’t re-map.
Core attributes you’ll see on a chat span:
| Attribute | Meaning |
|---|---|
gen_ai.operation.name | the operation: chat, embeddings, execute_tool, … (see §5) |
gen_ai.provider.name | provider/format flavor (replaces gen_ai.system) |
gen_ai.request.model / gen_ai.response.model | requested vs. actually-served model |
gen_ai.request.temperature, gen_ai.request.max_tokens | sampling params (see 02 - Decoding, Sampling & Reasoning Controls) |
gen_ai.usage.input_tokens / gen_ai.usage.output_tokens | token counts → cost/latency basis (§6) |
gen_ai.input.messages / gen_ai.output.messages | the prompt/completion content (opt-in, §5) |
gen_ai.tool.name, gen_ai.tool.call.id | on execute_tool spans |
5. Span taxonomy: the call tree
gen_ai.operation.name (all Development status) names what a span is:
| Operation | Span for |
|---|---|
chat | a chat-completion LLM call (the common one) |
text_completion | legacy completion-style call |
generate_content | multimodal generation |
embeddings | an embedding call |
execute_tool | one tool/function invocation |
invoke_agent / create_agent | agent lifecycle |
retrieval | a RAG retrieval step |
These nest into the trace tree. A typical agent turn:
invoke_agent (root: the user turn)
├── chat (LLM decides to call a tool)
│ └── gen_ai.usage.input_tokens=812, output_tokens=37
├── execute_tool search_docs (gen_ai.tool.name="search_docs")
│ └── retrieval (the vector lookup inside the tool)
├── chat (LLM reads tool result, answers)
│ └── gen_ai.usage.output_tokens=240
└── (online eval score attached as span attribute / event)
Reading the tree top-down is how you debug: a wrong answer with a good chat but an empty retrieval is a retrieval bug, not a prompt bug. Same answer with a never-fired execute_tool is a tool-selection bug.
6. Prompt/completion capture & privacy
Message content (gen_ai.input.messages / gen_ai.output.messages) is the most useful and most dangerous thing to log: it’s how you debug “why did it say that”, but it’s raw user input and model output — exactly where PII and secrets live.
Content capture is OPT-IN by default
The OTel GenAI conventions do not capture prompt/completion content by default. You enable it explicitly — e.g.
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. Treat that flag as a privacy decision, not a debug convenience.
Never log secrets, PII, or raw tokens
Prompts routinely contain user PII, and tool arguments can contain API keys or DB rows. Before content lands in a trace store: redact/mask PII, strip credentials and bearer tokens, and respect data-residency. Logs persist and fan out to vendors — a leaked prompt is a leaked secret. This is a security control, not a nicety. See 14 - Safety, Security & Risk.
Sample, don't capture-all
Full content on 100% of traffic is expensive and risky. Capture content on a sample (plus all errored/low-eval traces), keep metrics/token-counts on everything.
7. Token & cost tracking
Every chat/embeddings span should carry gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Multiply by per-model price → per-trace cost; aggregate by user, route, prompt-version, or model. This is the raw material for budgets, alerts, and model-downgrade decisions.
- Watch cached-vs-uncached input tokens — prompt caching changes the price of an input token dramatically; a cost metric that ignores cache state misleads.
- Reasoning models hide thinking tokens in output usage — your cost line can spike with no visible output-length change.
- Tie thresholds and per-route budgets to 15 - Cost, Latency & Performance; cost and latency share the same token + span data, so instrument once.
8. Eval-in-production (online evals)
Offline evals gate a deploy; online evals score a sample of live traffic continuously. Run the same scorers (LLM-judge, rule-based, heuristics) over production traces and attach the score to the span. This gives a quality time-series alongside cost/latency — the thing classic monitoring can’t see.
LLM-judge in production is itself a cost + drift source
A judge model is another LLM call — it costs tokens and can drift when its provider updates. Sample (don’t judge every trace), pin the judge model version, and periodically re-validate judges against human labels. See 13 - Evaluation & Evals.
9. Drift
Drift in GenAI is subtle because there’s no label stream. Three signals, in order of trustworthiness:
| Signal | What it catches | Trap |
|---|---|---|
| Input drift | users asking new kinds of things (new topics, languages, longer queries) | benign on its own — users always change |
| Eval-score drop | output quality falling on online evals | the signal that actually matters |
| Embedding drift | distribution shift in input/output embeddings | noisy; fires on harmless change |
Alert on input-drift AND eval-score-drop jointly — not embedding-drift alone
Embedding-drift alarms in isolation are mostly false positives: the input distribution always moves, and most movement is harmless. The actionable alert is “inputs shifted and quality dropped” — that’s a real regression (model update, prompt rot, new attack pattern). Eval-score-drop alone catches silent provider model swaps even with stable inputs. Wiring embedding-drift straight to PagerDuty just trains the team to ignore it.
Also watch for silent model updates: a vendor pins your alias to a new snapshot, behavior shifts, inputs unchanged — only an online eval catches this.
10. Prompt versioning & experiment tracking
Prompts are code that lives outside your repo if you let it. Don’t.
- Version every prompt with an ID + hash; stamp
prompt.versiononto each trace so any output is traceable to the exact template that produced it. - Decouple prompt edits from deploys via a prompt registry (most tooling offers one) — but the trade-off is the prompt is no longer in your git history/PR review. Pick deliberately; for high-risk prompts keep them in-repo.
- Experiment tracking: when you change a prompt/model, run the candidate against a fixed eval dataset, record scores per version, and compare. This is the offline half of §8 and the gate for §11.
11. CI/CD for prompts
Treat a prompt change like a code change:
PR changes prompt v12 → v13
→ CI runs eval dataset against v13
→ assert: eval score ≥ baseline AND cost/latency within budget
→ block merge on regression
→ deploy behind flag → online eval on sample → promote or roll back
The key difference from code CI: the gate is a statistical eval threshold, not a pass/fail unit test — outputs are non-deterministic, so you assert on aggregate score over a dataset, not exact strings. (Same lesson as the determinism myth in 02 - Decoding, Sampling & Reasoning Controls.)
12. Tooling landscape
| Tool | License / model | Notable for | Convention |
|---|---|---|---|
| Langfuse | open-source (MIT), self-host | prompt management, evals, datasets; prompt-centric | OTel GenAI + OTLP |
| LangSmith | hosted (LangChain) | tight LangChain/LangGraph integration, LangGraph Studio | OTel-compatible |
| Arize Phoenix | open-source | strong tracing + eval, auto-instruments many SDKs | OpenInference (not gen_ai.*) |
| Helicone | Apache-2.0 proxy | cheap drop-in request/response capture via proxy | proxy-level |
| Weave (W&B) | hosted | fits teams already on W&B; strong eval harness | OTel-compatible |
Phoenix uses OpenInference, not OTel
gen_ai.*Arize Phoenix is OpenTelemetry-native (rides OTLP) but emits its own OpenInference semantic conventions, which are not the same attribute names as
gen_ai.*. Spans are compatible at the transport level but you can’t blindly querygen_ai.usage.input_tokensagainst Phoenix data — the attribute keys differ. Know which convention your backend speaks before you write dashboards or portability assumptions.
Portability is the exit strategy
Instrument at the app layer against OTel GenAI conventions (e.g. OpenLLMetry) and any OTLP backend — Langfuse, Phoenix, LangSmith, Datadog — accepts it. That decouples you from any single vendor. Just remember the convention is still Development (§4), so the “standard” you’re betting on is moving.
13. Pros / cons / when
| Pros of full LLMOps instrumentation | debug the call tree not the guesswork; quality time-series via online evals; cost/drift caught early; prompt changes gated like code; vendor-portable traces |
| Cons / cost | trace volume + storage cost; content capture is a privacy liability; online evals add their own LLM spend; the semconv churns under you; another stack to operate |
| Use when | anything past a prototype that real users touch; agents/RAG/tools (the trace tree is the only way to debug); regulated or PII-bearing domains (you need the audit trail done right) |
| Skip / defer when | one-off scripts, internal throwaway demos — but stamp prompt versions even then; it’s cheap insurance |
14. Caveats & risk mitigation
- The semconv will rename things on you. Pin instrumentation versions, use
OTEL_SEMCONV_STABILITY_OPT_IN, keep a mapping layer between raw attributes and dashboards. Re-check after every OTel GenAI release. - Content capture is a breach waiting to happen. It’s opt-in for a reason — redact PII, strip secrets, sample rather than capture-all, honor data residency. See 14 - Safety, Security & Risk.
- Drift alerts that fire on embedding-drift alone become noise. Gate alerts on input-drift and eval-score-drop together; alert on eval-score-drop alone to catch silent model swaps.
- Online evals cost money and drift themselves. Sample, pin the judge version, re-validate against humans periodically.
- Prompt in a registry leaves your git history. You gain hot-swap, you lose PR review — choose per prompt’s risk.
- Token/cost metrics that ignore cache state and hidden reasoning tokens mislead. Capture cached-vs-uncached and remember reasoning output usage. Link 15 - Cost, Latency & Performance.
- Phoenix ≠
gen_ai.*. Confirm your backend’s convention before assuming attribute names are portable.
15. In practice
# Stamp a versioned, cost-aware, PII-safe span on each LLM call.
from opentelemetry import trace
tracer = trace.get_tracer("genai.app")
with tracer.start_as_current_span("chat") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", "anthropic") # NOT gen_ai.system (deprecated)
span.set_attribute("gen_ai.request.model", model_id)
span.set_attribute("prompt.version", prompt_v) # traceable to exact template
resp = client.messages.create(model=model_id, messages=msgs, max_tokens=1024)
# cost basis — works for budgets, dashboards, drift
span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
# CONTENT: opt-in only, and redacted. Never raw PII/secrets.
if CAPTURE_CONTENT: # OTEL_..._CAPTURE_MESSAGE_CONTENT
span.set_attribute("gen_ai.input.messages", redact(msgs))
span.set_attribute("gen_ai.output.messages", redact(resp))
# Later: an online-eval job samples these spans, scores them,
# and attaches the score → quality time-series + joint drift alert.Sources
- OpenTelemetry — GenAI semantic conventions repo (Development status)
- OpenTelemetry — GenAI attribute registry (
gen_ai.*) - OpenTelemetry — GenAI agent & framework spans
- OpenTelemetry blog — Inside the LLM Call: GenAI Observability (2026)
- Langfuse — Phoenix/Arize alternatives & OTel convention support
- Arize Phoenix — OpenInference conventions
- traceloop/openllmetry —
gen_ai.prompt/gen_ai.completiondeprecation