OpenTelemetry for AI Products: A Product Manager's Guide

What OpenTelemetry Is and Why AI Products Need It

OpenTelemetry is a vendor-neutral observability framework maintained by the Cloud Native Computing Foundation. It standardizes how distributed systems emit three types of signals: traces (records of a request flowing through your system), metrics (aggregate measurements like latency and error rate), and logs (structured event records). Before OTel, every monitoring vendor had its own SDK, which meant switching providers required rewriting all your instrumentation. OTel solves that by giving you one SDK that can export to any backend: Datadog, Grafana, Honeycomb, Jaeger, or your own pipeline.

For AI products, OTel matters for a reason standard web services do not face: the model itself is a black box that produces variable latency, variable token usage, and variable output quality. A typical API call to your database either succeeds in 2ms or fails. An LLM call might succeed in 800ms on Monday, 4,200ms on Tuesday (because the provider is under load), and succeed in 1,100ms Wednesday but produce a hallucinated response. Tracking success or failure alone does not tell you enough. You need traces that capture the full context of each inference call: what prompt was sent, which model version responded, how many tokens were consumed, how long each phase took, and what the model returned.

Why standard APM tools miss AI-specific failures

Latency without context

A 4s p95 shows in your dashboard, but you cannot tell if it is the retrieval step, the reranker, or the model generating 2,000 tokens when 200 would have done.

Errors without prompt state

A 429 rate limit error tells you the provider rejected a call. It does not tell you which feature, which user segment, or which prompt template triggered the burst.

No quality signal

Traditional APM tracks HTTP status codes. A 200 from an LLM API can still mean the model refused the request, generated a factually wrong answer, or hit a guardrail and returned a canned disclaimer.

Multi-step invisibility

A RAG pipeline has 5 to 8 steps: query rewriting, embedding, retrieval, reranking, generation, and citation extraction. Standard traces show one span for the whole thing.

The Three OTel Signals and How They Map to AI Products

Understanding the three signal types lets you have an informed conversation with your engineering team about what to instrument. You do not need to write the code, but you do need to define which product questions each signal should answer.

Traces

What it is: A trace records the end-to-end journey of a single request. Each step is a 'span' with a start time, end time, and metadata. In a RAG pipeline, you would have separate spans for query embedding, vector search, context assembly, LLM call, and response post-processing.

PM use: Traces answer 'what happened to request X.' Use them for debugging slow P99 latency, understanding which pipeline step fails most often, and investigating individual user-reported issues. Your on-call engineer needs traces to diagnose incidents faster.

A user reports: the assistant gave the wrong answer about our refund policy. A trace shows the retrieval step pulled the wrong document version. Without traces, this is a 2-hour debugging session. With them, it is 5 minutes.

Metrics

What it is: Metrics are aggregate measurements over time: p50/p95/p99 latency, tokens per request, cost per 1,000 requests, error rate by endpoint, cache hit rate. They are pre-aggregated, cheap to store, and fast to query.

PM use: Metrics answer 'how is the system performing overall.' Use them for your real-time dashboards, SLA monitoring, cost tracking, and capacity planning. Metrics are what your product health scorecard is built on.

You notice in your weekly review that average tokens per request climbed 40% over two weeks. A metric trend surfaces this; a trace explains which prompt change caused it.

Logs

What it is: Logs are structured event records tied to a specific moment in time. For AI products, the most important logs are prompt logs: the exact text sent to the model, including the system prompt and any injected context.

PM use: Logs answer 'what was the exact state when this happened.' Use them for prompt audits, compliance evidence, output quality review, and training data curation. Prompt logs are also your dataset for offline evaluation.

A regulated industry customer asks for an audit trail of all prompts sent on their behalf during a specific date range. If you have structured prompt logs, this is a query. If you do not, it is a compliance gap.

The AI-Specific OTel Attributes That Change Everything

Standard OTel spans capture HTTP method, URL, status code, and duration. Those four fields are not enough for AI. The OpenTelemetry Semantic Conventions for Generative AI define a set of standard attributes specifically for LLM calls. Knowing these lets you specify what your engineering team should emit.

gen_ai.request.model

Which exact model version handled this request. Essential when you are running A/B tests across models, or when a provider silently upgrades their API and output quality changes.

gen_ai.usage.input_tokens

Tokens sent to the model. Combined with pricing, this is your per-request cost driver. Trending up means prompts are getting longer, which could be a RAG retrieval problem or a system prompt bloat.

gen_ai.usage.output_tokens

Tokens the model generated. High output token counts are expensive and slow. If this is spiking, you might be missing a max_tokens constraint somewhere.

gen_ai.request.temperature

The sampling temperature used. Useful when debugging inconsistency issues: different features may use different temperatures and that difference matters for reliability.

gen_ai.response.finish_reason

Why the model stopped generating: 'stop' (normal), 'length' (hit max_tokens), 'content_filter' (blocked by safety layer), 'tool_calls' (function calling). A high rate of 'length' is a truncation bug.

gen_ai.system

The AI provider (openai, anthropic, cohere). Required if you are using multiple providers or a routing layer. Lets you compare latency and error rates across providers.

Beyond these standard attributes, add custom attributes for your product context: the feature name that triggered the call, the user tier, the prompt template ID, and the RAG document source. These are what let you filter traces by product dimension rather than just by technical dimension.

Build AI Products the Right Way

The AI PM Masterclass covers how to architect observable AI systems, define the right metrics, and ship AI features that perform reliably in production.

The Three Instrumentation Conversations to Have Before Launch

You do not write OTel code, but you do define the requirements. Three conversations with your engineering team will determine whether your observability is useful at launch or something you scramble to retrofit after the first production incident.

Conversation 1: What are our SLOs, and which traces prove we are meeting them?

A Service Level Objective is a target for your system's reliability: 95% of requests complete within 2 seconds, or the model responds in under 800ms at p95. Without an SLO, you do not know what to alert on. With one, you know exactly which spans to instrument and which thresholds to watch. Start with user-facing latency and end-to-end success rate. Define what 'success' means for your AI feature: is a guardrail refusal a success or a failure from the user's perspective?

Conversation 2: What is our prompt logging policy, and what do we filter for privacy?

Full prompt logging is powerful but dangerous. User prompts may contain PII, health information, or confidential business data. Before you log prompts, define: what goes into the log (full prompt, just the metadata, a hash), who has read access, how long logs are retained, and how you handle deletion requests. In regulated industries, your compliance team must be in this conversation. Some teams log only system prompts plus query intent categories, not the raw user text. This is a product decision with legal implications.

Conversation 3: How do we connect technical traces to product outcomes?

The most powerful observability connects a trace to a downstream business event. A user runs a query, gets a response, and then either converts to a paid plan or churns. If you only instrument the technical layer, you can see that request succeeded. If you connect the trace ID to your product analytics event, you can see that response quality predicted conversion. This requires engineering to emit a trace ID that your analytics layer also receives. It is not hard, but it must be decided before both systems are built.

Common Instrumentation Mistakes and How to Avoid Them

Mistake: Tracing only the LLM call

Fix: Trace the full pipeline. In a RAG system, the model call is often 30% of total latency. Retrieval and reranking frequently cause more variance. If you only trace the generation step, you miss 70% of your optimization surface.

Mistake: Using sampling rates set for web APIs

Fix: Standard web APM tools sample 1% to 10% of traces to reduce cost. AI requests are lower volume but higher value per trace. Start at 100% sampling in development and staging. In production, sample by feature tier: trace everything for premium users, sample 20% for free tier.

Mistake: Logging raw prompts without consent policy

Fix: Write a data retention policy before writing logging code. Define PII handling, deletion rights, and access controls. A prompt log of a healthcare AI without these policies is a liability.

Mistake: No alert routing from OTel signals

Fix: Instrumentation that no one acts on is decoration. For every SLO you define, create a corresponding alert. Route latency alerts to the on-call engineer and cost spike alerts to the PM. Without routing, your dashboards are a museum.

Mistake: Skipping semantic conventions

Fix: Using custom attribute names instead of the OTel GenAI semantic conventions means you cannot use standard tooling, dashboards, or libraries. The conventions exist; use them.

Your OTel Readiness Checklist Before Shipping to Production

Use this as a pre-launch gate. Missing any of these makes your first production incident significantly harder to diagnose.