AI PM TEMPLATES

AI Fine-Tuning Project Brief Template: Scope, Data, and Success Metrics in One Document

By Institute of AI PM·16 min read·Jul 10, 2026

TL;DR

Most fine-tuning projects fail because the PM handed off a vague ask — "make the model sound more like us" — instead of a structured brief. A fine-tuning brief is different from a standard PRD: it specifies the training data sourcing plan, quality criteria, baseline evaluation methodology, and acceptance thresholds before anyone writes a line of code. This template gives you a four-part document that covers business context, data requirements, success criteria, and rollout risks. Copy it, fill it in, and use it to align your ML team, data team, and stakeholders before a single GPU spins up.

The AI PM Minute

One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.

No fluff. Unsubscribe anytime.

Why Fine-Tuning Needs Its Own Brief

A standard PRD documents what a feature should do. A fine-tuning brief documents what a model should learn, how you will measure whether it learned it, and what data will teach it. These are fundamentally different things. A PRD describes deterministic behavior: "clicking this button submits the form." A fine-tuning brief describes probabilistic behavior: "when given a support ticket, the model should respond in a tone that matches our brand voice 85% of the time as judged by a panel of customer service leads."

Fine-tuning projects that skip a proper brief tend to fail in one of three ways. First, the ML team builds the wrong thing because the PM could not articulate what "better" actually means. Second, the data team collects training examples that are plentiful but wrong, because no one defined what a good example looks like before collection began. Third, the project ships a model that performs well on the eval set but fails in production because the eval set did not represent real user inputs.

The brief is also a forcing function for the PM. Writing it reveals whether fine-tuning is actually the right approach. Many tasks that feel like fine-tuning problems are actually prompt engineering problems, RAG problems, or model selection problems. If you cannot write a clear brief, that is information: the problem definition is not sharp enough to start a fine-tuning project yet.

When to use this template

Use this brief when you have decided fine-tuning is the right approach and are now moving from decision to execution. If you are still evaluating whether to fine-tune at all, read the LLM Fine-Tuning guide first, then return here.

Part 1: Business Context and Objectives

The first section of your brief answers the question every ML engineer will have before spending weeks on a project: "why are we doing this and what does success look like for the business?"

Problem statement (2-3 sentences)

State the specific behavior gap you are trying to close. Not 'improve model quality' but 'the model responds to customer refund requests in a generic tone that does not match our brand voice, leading to a 23% escalation rate on AI-handled tickets.'

Example: Our AI support agent uses the base Claude Sonnet vocabulary and tone, which our customer satisfaction data shows reads as corporate and cold. Brand voice surveys indicate 67% of AI-handled tickets would have been preferred human-handled. Fine-tuning on approved support transcripts should close this gap.

Baseline you are improving

Define the current model, prompt, and behavior you are measuring from. Include the version or date so there is no ambiguity about what you are comparing against.

Example: Baseline: Claude Sonnet 5, June 30 2026 system prompt v4.2, brand voice score of 54/100 on 200-sample human eval.

Target metric with threshold

State the specific metric, measurement method, and the minimum threshold at which you would ship. This must be measurable before training begins.

Example: Brand voice score of 75/100 or higher on the 200-sample held-out eval set, as judged by the same panel of 3 customer service leads using the rubric in Appendix A.

Business outcome linked to target

Connect the model metric to a business metric that stakeholders understand. This prevents the project from being deprioritized when ML capacity is constrained.

Example: Reaching 75 brand voice score is projected to reduce AI ticket escalation rate from 23% to under 12%, freeing 2.4 FTE equivalent hours per day in tier-1 support.

Alternatives considered

Document why you chose fine-tuning over prompt engineering, RAG, or a different base model. This shows due diligence and prevents revisiting the decision mid-project.

Example: Prompt engineering (tested 6 variants over 2 weeks) moved brand voice from 54 to 61 — insufficient. RAG on brand guidelines gave inconsistent results across ticket types. Base model switch to Grok 4.5 evaluated; brand voice score was 49/100. Fine-tuning on Sonnet 5 is the remaining lever.

Part 2: Data Requirements

Data requirements are the section most PMs skip or underspecify. The ML team can usually figure out the architecture. They cannot figure out what a "good" training example looks like without precise PM guidance. Fill this section out before any data collection begins.

Volume target

State the minimum number of training examples required. For instruction fine-tuning of most tasks, 200 to 1,000 high-quality examples outperform 10,000 mediocre ones. If you are not sure, start with 500 and plan to iterate.

Format specification

Define the input-output pair format. Supervised fine-tuning requires (prompt, ideal completion) pairs. Specify whether completions should be full responses or partial continuations, and the expected average length.

Quality criteria

Write a rubric for what makes an example good versus bad. For each criterion (tone, accuracy, length, format), describe what passing and failing look like. This rubric will be used by whoever labels the data.

Data sources

List where examples will come from: existing transcripts, human-written examples, LLM-generated synthetic data, or a combination. For each source, note any privacy, legal, or IP constraints that must be resolved before collection.

Train / validation / test split

Specify the split and how the held-out test set will be kept separate from training. The test set must not be used to make training decisions — it is reserved for the final acceptance evaluation only.

Negative examples

Define what the model should NOT do as clearly as what it should do. Include examples of failure modes in the brief so annotators know what to reject. DPO training requires explicit negative pairs.

Data quality beats data volume

Fine-tuning amplifies patterns in your training data. If your data contains 15% examples with inconsistent tone, the model learns inconsistent tone. Invest in a 30-minute labeler calibration session before any data collection begins. Everyone touching the data should agree on the quality rubric before collecting a single example.

Run Fine-Tuning Projects Like a Pro

The AI PM Masterclass teaches you how to scope, run, and evaluate fine-tuning projects end to end — including how to work with ML teams, build eval frameworks, and ship fine-tuned models as product features.

Part 3: Success Criteria and Evaluation Plan

The evaluation plan is the most important section of the brief, and the one most often left vague. "We will know it is working when the model sounds better" is not an evaluation plan. A real evaluation plan specifies who evaluates, with what rubric, on what sample, and what threshold triggers a ship or a no-ship decision.

Primary metric and threshold

The single number that determines whether the project ships. Define it before training starts. Changing the metric after seeing results invalidates the evaluation. Example: human brand voice score >= 75/100 on 200-sample test set.

Regression guard metrics

Fine-tuning on a narrow task often degrades the model on adjacent tasks it was not trained on. Define two to four capability checks that must not regress below a floor. Example: factual accuracy on product FAQ questions must stay above 90% (baseline: 94%). If the fine-tuned model drops below 90%, the project fails regardless of brand voice score.

Human evaluation protocol

For quality dimensions that cannot be measured automatically (tone, brand voice, empathy), specify: who evaluates (role, seniority, number of evaluators), the rubric they use, the scale (1-5, 1-10, pass/fail), the inter-rater agreement threshold, and how disagreements are resolved.

Automated evaluation coverage

Specify which quality dimensions will be checked automatically (format compliance, response length, profanity filter, latency, specific keyword presence or absence) and the tools used. Automated evals run continuously in production; human evals run at milestone checkpoints.

A/B test plan

Define the traffic split, duration, user population, and primary metric for the production A/B test. Include the minimum detectable effect and statistical power. Fine-tuning rarely justifies a full production rollout without a controlled A/B test, especially on customer-facing features.

Part 4: Timeline, Risks, and Rollout

Fine-tuning projects have a different timeline profile from feature development. Data collection and annotation are often slower than the training run itself. The brief should include a realistic phase plan so stakeholders understand where the time actually goes.

Phase 1: Data collection and annotation

2 to 4 weeks

Source training examples, run labeler calibration session, collect and annotate to quality rubric, spot-check 10% of annotations, build train/val/test split.

Phase 2: Training run

1 to 5 days

Fine-tuning run on selected base model, hyperparameter logging, checkpoint evaluation on validation set, iteration if validation metric has not converged.

Phase 3: Evaluation

1 to 2 weeks

Human eval on held-out test set, automated regression checks, latency and cost benchmarking, safety review, decision gate: ship or iterate.

Phase 4: A/B rollout

2 to 4 weeks

Shadow mode deployment (log outputs, do not serve to users), gradual traffic ramp, primary metric monitoring, rollback plan in place, full rollout if A/B results meet threshold.

Risk register (add to brief)

Data collection takes longer than planned

Start data collection before committing to a ship date. Never schedule a training run before the data is collected and spot-checked.

Fine-tuned model degrades on regression checks

Define regression floors before training. If they are hit, the decision to iterate or descope is made against pre-agreed criteria, not in a high-pressure conversation.

Base model provider updates the base model mid-project

Pin the base model version and test fine-tuning on that pinned version. Never fine-tune on 'latest.'

Evaluation reveals the task was underspecified

Run a small-scale pilot (50 to 100 examples) before full data collection. If the pilot eval reveals ambiguity in the quality rubric, resolve it before scaling.

Common Fine-Tuning Brief Mistakes

Mistake 1: Writing the evaluation plan after seeing the training results

You will unconsciously design an eval that confirms the results you have. The held-out test set loses its value the moment anyone looks at it before finalizing the evaluation criteria.

Mistake 2: Targeting a metric you cannot measure

'Make the model more empathetic' is not a measurable target. Either define a rubric that lets humans score empathy consistently, or choose a proxy metric that is measurable. If you cannot measure it, you cannot know if fine-tuning worked.

Mistake 3: Underestimating data quality investment

Teams routinely spend 3x more time on data than the training run. A brief that allocates one week to data and two weeks to training is planning to fail. Reverse those estimates and you are closer to reality.

Mistake 4: Fine-tuning on a model version without pinning it

If the base model provider silently updates their model after you begin data collection, your training run produces a model fine-tuned on a different foundation than your eval assumed. Always pin the version.

Mistake 5: Skipping the rollback plan

The brief should include the specific steps and decision owner for reverting to the unfine-tuned model. If the brief does not include this, it will not be done when needed, and it will be needed.

Build AI Products That Perform in Production

The AI PM Masterclass covers how to scope, run, and ship fine-tuning projects as product initiatives — not research experiments. Learn from a Salesforce Sr. Director PM who has built these pipelines in production.

→ LLM Fine-Tuning: When It's Worth It and When It's Not → LoRA and PEFT Explained: The PM's Guide to Parameter-Efficient Fine-Tuning → AI Data Labeling Brief Template: How to Commission High-Quality Training Data → AI Evaluation Design Guide: How to Build Evals That Actually Predict Production Quality

Before you go: get the AI PM Minute