Continuous AI Evaluation: Building a Living Quality Pipeline in Production

Why One-Time Evals Are Not Enough

Traditional software quality is largely static: version 1.2 behaves identically on the same inputs as it did the day it shipped, unless a developer changes the code. AI products are not like this. The same prompt sent to the same API endpoint can return different outputs a month later because the model provider updated the model weights, the serving infrastructure, or the sampling behavior without changing the API version.

GPT-4o Turbo updated in April 2025 and changed JSON output formatting in ways that silently broke downstream parsers for thousands of applications. Teams without regression evals running in production found out through user reports, not monitoring dashboards. This is the most common AI quality failure mode: not a crash, not an error code, just quietly wrong outputs that persist for days or weeks before anyone notices.

Distribution shift

Your eval set was built on early user queries. Six months later, users are asking different things, in different phrasing, in different volumes. Evals that passed at launch are testing a population that no longer represents your production traffic.

Silent provider updates

Model providers update weights, system prompts, safety filters, and output formatting without changing API version or notifying customers. Your product can degrade without a single line of your code changing.

Prompt drift

Engineers and PMs update system prompts to fix one issue and introduce regressions in others. Without a regression suite running against the live prompt, you only know about the issue you fixed.

Edge case accumulation

Production surfaces edge cases no pre-launch test anticipated. Without a process for converting these failures into eval cases, they stay as one-off bug reports that never become systematic protection.

The Four-Layer Evaluation Stack

Continuous evaluation is not a single pipeline. It is four distinct evaluation layers running at different cadences with different signal types. Teams that collapse all four into a single "eval run" get a noisy number that is hard to act on. Teams that run them separately get actionable signal at the right frequency.

Layer 1: Correctness regression suite

Every deploy, every model config change

What it does: A fixed set of golden input/output pairs where the correct answer is deterministic. Checks that the model still produces the right output on cases you have already solved. Runs in minutes. Should fail loudly and block deploys.

Coverage: 30 to 50 cases, weighted toward your highest-traffic query types and your historical failure cases.

Layer 2: Production sample scoring

Daily

What it does: A random 1 to 5% sample of production requests, scored automatically using an LLM judge or rule-based criteria against your quality rubric. Produces a daily quality score that you can trend over time.

Coverage: Statistically significant sample of real production traffic. The key metric is trend, not absolute score.

Layer 3: Human expert review

Weekly or biweekly

What it does: A domain expert reviews 20 to 30 cases flagged by Layer 2 as borderline (scored 3 to 4 out of 5, not clearly good or clearly bad). Human review calibrates the LLM judge and surfaces systematic issues that the automated scorer misses.

Coverage: Borderline cases from Layer 2. Not a random sample: borderline cases have the highest marginal information value.

Layer 4: Adversarial and edge case mining

Monthly or after major incidents

What it does: A structured effort to find cases where the system fails: stress testing with unusual inputs, probing policy boundaries, testing the cases your users will eventually send that you haven't anticipated. Outputs become new Layer 1 regression cases.

Coverage: Typically 50 to 100 cases per session, added to the regression suite as they are confirmed.

Sampling and Traffic Selection

Not all production traffic is equally valuable to evaluate. Random sampling is the right starting point, but stratified sampling produces better signal at lower cost. The goal is to maximize coverage of distinct failure modes per evaluation dollar, not to process as many requests as possible.

Stratify by query type

Maintain a query taxonomy and ensure each stratum is represented in your daily sample. If 70% of your traffic is type A but A rarely fails, sampling uniformly will undersample your high-risk types. Weight your sample toward types that have historically generated failures.

Oversample low-confidence outputs

If your model exposes a confidence score or if you have a proxy (response length, hedging language, refusal rate), oversample the outputs where confidence is lowest. These are the cases most likely to contain errors and the cases most valuable for human review.

Capture the full context

Log enough context to reproduce the failure: the system prompt version, the model ID, the temperature setting, the full conversation history for multi-turn sessions, and the timestamp. Without this, you cannot diagnose root cause when an issue surfaces.

Respect PII handling

Do not log raw user input that contains PII into your evaluation store without scrubbing. Use a PII detection layer before storage. Your eval pipeline should have the same data handling standards as your production pipeline.

Learn Evaluation Strategy in the AI PM Masterclass

The masterclass covers how to design and operationalize AI evaluation pipelines, taught live by a Salesforce Sr. Director PM who has built them in production.

Automated Scoring: Human-in-the-Loop Without the Bottleneck

Human review is the gold standard for evaluation quality, but it does not scale to production traffic volume. Automated scoring using a separate LLM as a judge has become the standard approach for high-volume production eval, with human review reserved for the cases where automated scoring is uncertain.

An effective automated scoring pipeline has three components: a structured rubric, a judge model, and a calibration process. The rubric defines what "good" means for your specific use case, the judge model scores each output against that rubric, and calibration validates that the judge's scores correlate with human expert judgments on the same cases.

1. Define a structured rubric

A rubric is a 3 to 5 dimensional scoring framework, each dimension rated 1 to 5. Example dimensions: factual accuracy, completeness, appropriate tone, adherence to format constraints, absence of policy violations. The rubric should be specific to your use case, not generic. A customer support AI has different quality dimensions than a code review assistant.

2. Prompt the judge model with both the rubric and the output

Send the original request, your system prompt, and the model output to a judge model (typically a frontier model that is different from the model being evaluated, to avoid self-grading bias). Ask it to score each dimension and explain its reasoning. Chain-of-thought improves judge accuracy significantly on borderline cases.

3. Aggregate dimension scores into a composite

Weight dimensions by their importance to your use case. A compliance violation in a legal AI might be a hard failure regardless of other scores. A slightly off-tone response in a creative tool might not matter at all. Build your composite to reflect your actual quality priorities, not a simple average.

4. Calibrate the judge against human labels

Run the judge on 100 to 200 cases where you also have human expert labels. Measure agreement rate and Spearman correlation between judge scores and human scores. Recalibrate the judge prompt if agreement is below 80% on the dimensions that matter most.

5. Flag low-confidence scores for human review

When the judge scores a case in the middle range (typically 3 on a 1 to 5 scale), route it to human review rather than trusting the score. Middle scores have the lowest judge accuracy and the highest information value for calibration.

Regression Detection and Alerting

A regression is a statistically significant drop in your composite quality score compared to a baseline. Detecting regressions requires a baseline, a comparison method, and a decision rule for when to alert. Without a defined decision rule, teams either ignore dashboards entirely or generate so many false positives that alerts lose meaning.

Establish a rolling baseline

Your baseline is the rolling 7-day average composite score from the previous period. Not a historical benchmark from launch, but a recent window that reflects current distribution. AI product quality naturally drifts as usage patterns change, and comparing to a launch-era baseline produces misleading regression signals.

Use statistical significance, not raw deltas

A drop of 0.3 points on a 5-point scale sounds meaningful but might be noise in a daily sample of 200 requests. Use a two-sample t-test or Mann-Whitney U test to distinguish signal from sampling variance. Alert only when the drop is statistically significant at p < 0.05.

Define severity tiers

Tier 1 (critical, page the on-call team): any score below a hard threshold (e.g., below 2.5 on key dimensions), or a regression affecting a safety-critical dimension like policy compliance. Tier 2 (urgent, Slack alert to PM and engineering lead): a statistically significant 5% or greater drop in composite score over 48 hours. Tier 3 (informational, weekly summary): a gradual downward trend visible over 7 or more days.

Connect alerts to the on-call rotation

Evaluation alerts with no clear owner get ignored. Wire Tier 1 and Tier 2 alerts into your existing incident management system with a default assignee who has authority to roll back a model config change or pin a prior model version.

Closing the Loop: From Signal to Sprint

Evaluation that does not feed back into product improvement is operational theater. The last and most important step in continuous evaluation is a defined process for converting evaluation findings into sprint priorities. Without this process, evaluation data accumulates in dashboards and the team continues shipping the same failure modes indefinitely.

The recommended cadence for AI product teams:

Daily (15 minutes): Eval triage

Review the daily quality score and any alerts. If a regression is detected, confirm whether it is real or noise. If real, open a P1 or P2 incident and assign root cause investigation. If noise, note it in the evaluation log and adjust the alert threshold if needed.

Weekly (30 minutes): Failure pattern review

Review the week's human-reviewed cases and identify any patterns: are failures clustering around a query type, a user cohort, a particular time of day, or a feature? Patterns become the input to the following week's sprint prioritization. Assign a category and impact estimate to each pattern.

Biweekly: Regression suite maintenance

Add any confirmed new failure cases to Layer 1 as golden examples. Review the regression suite for cases that are no longer testing real risk (e.g., edge cases that were fixed and have not recurred in six months). A bloated regression suite slows down deploys without proportional safety benefit.

Monthly (60 minutes): Adversarial red team session

Run a structured session where one or two team members actively try to break the system. Document what they find. Add confirmed failures to the regression suite. Review whether the existing evaluation rubric still matches current quality priorities, and update dimensions if the product has evolved.

The key mindset shift

Pre-launch evaluation asks "is this good enough to ship?" Continuous evaluation asks "is this still as good as it was yesterday?" The second question is harder to answer and more important to keep answering. The teams whose AI products maintain quality at scale are the ones who treat evaluation as a permanent operational function, not a launch gate they passed once.