Data Labeling at Scale: Building High-Quality Training Data
TL;DR
Most AI teams overinvest in model architecture and underinvest in labeling quality. In practice, moving from a good model with noisy labels to the same model with clean labels improves performance more than switching from a good model to a great model. This guide covers the four labeling approaches (in-house, crowdsourced, expert, and LLM-assisted), how to measure and enforce labeling quality, and the operational decisions — like when to invest in more labels vs. better labels — that separate successful data teams from ones that burn budget on data that doesn't help.
Why Labeling Quality Determines Model Quality
Every supervised ML model learns from labeled examples. The model's job is to find patterns in the training data and generalize them to new inputs. If the labels are inconsistent, ambiguous, or wrong, the model learns the wrong patterns. This is the “garbage in, garbage out” principle, but its implications go deeper than most teams appreciate.
Research consistently shows that label noise has a multiplicative effect on model error. A 5% label error rate doesn't produce a 5% drop in model accuracy — it produces a 10–15% drop because the model actively learns from the incorrect examples. On hard-to-classify examples near decision boundaries, noisy labels cause the model to learn the wrong boundary entirely.
Inconsistent labels create confused models
If two identical inputs have different labels in your training data, the model learns to hedge between them. This manifests as low-confidence predictions and high error rates on similar inputs. Inconsistency is worse than systematic bias because the model can't learn a clean pattern from contradictory signal.
Ambiguous labeling guidelines compound over time
If your labeling instructions say 'classify as urgent if the customer seems frustrated,' different labelers will interpret 'frustrated' differently. This creates systematic disagreement that grows as you add more labelers. The fix isn't more labelers — it's clearer guidelines with concrete examples of what does and doesn't count.
Label errors on edge cases are the most expensive
Easy examples are usually labeled correctly by everyone. The value of high-quality labeling is concentrated in the hard examples — the ones near decision boundaries where the model is most uncertain. If you're going to invest in label quality, focus your quality assurance on the examples where labelers disagree, not the ones where they all agree.
Evaluation sets need higher quality than training sets
A noisy training set hurts model performance. A noisy evaluation set makes you unable to measure model performance accurately. Your test set labels need to be near-perfect (>98% accuracy). If your evaluation set has 10% label noise, you can't distinguish between a model improvement and label noise — you'll make wrong decisions about model changes.
The 4 Labeling Approaches
There is no single best labeling approach. Each has different cost structures, quality profiles, and operational requirements. Most mature labeling operations use a mix of approaches, with the blend shifting as the project matures.
In-house labeling teams
Your own employees label data, either as their primary role or as part of their existing job. In-house labelers understand your domain, your product, and your taxonomy deeply. They produce the highest-quality labels but at the highest cost per label ($0.50–$5.00 per item depending on complexity). Best for: complex domain-specific tasks, building gold standard evaluation sets, and bootstrapping a new labeling taxonomy. In-house teams also generate the institutional knowledge about edge cases that makes labeling guidelines effective.
Trade-off: Highest quality and domain knowledge, but slowest to scale and most expensive per label. Use for high-stakes labels and guideline development.
Crowdsourced labeling
Platforms like Scale AI, Labelbox, Amazon Mechanical Turk, and Toloka provide access to large pools of labelers who work on microtasks. Cost per label drops to $0.05–$0.50, and you can scale to millions of labels quickly. The challenge is quality: crowdsource labelers don't understand your domain, may be incentivized to work fast rather than accurately, and require extensive quality controls. Best for: high-volume labeling of relatively simple tasks (image classification, sentiment analysis, entity recognition) where you can verify quality through redundancy.
Trade-off: Cheapest and fastest to scale, but requires heavy quality assurance investment. Never use for evaluation sets without expert verification.
Expert labeling
Domain experts (doctors labeling medical images, lawyers labeling contract clauses, engineers labeling code quality) provide labels that require specialized knowledge. Cost is the highest at $5–$50+ per item, and throughput is the lowest because expert time is scarce. But for domains where correct labeling requires years of training, there is no substitute. Best for: medical, legal, financial, and scientific classification tasks where a non-expert label is worthless. Use experts to label evaluation sets and a sample of training data, then use those expert-labeled examples to train and calibrate cheaper labeling approaches.
Trade-off: Only option for specialized domains, but prohibitively expensive at scale. Use strategically for gold standards and calibration.
LLM-assisted labeling
Using LLMs (GPT-4, Claude, Gemini) to generate labels, either fully automatically or as a pre-labeling step where humans review and correct. Cost per label is $0.01–$0.10, with high throughput. Quality depends on task complexity: LLMs perform well on tasks that require general language understanding but poorly on tasks requiring domain expertise or subjective judgment. Best for: bootstrap labeling when you have no labeled data, pre-labeling to accelerate human review (reduces human review time by 40–60%), and generating labels for the long tail of categories where you have few training examples.
Trade-off: Fast and cheap, but quality degrades on domain-specific or subjective tasks. Always validate LLM labels against expert-labeled ground truth before trusting them for training.
Labeling Quality Assurance
Quality assurance for labeling isn't optional — it's the difference between training data that improves your model and training data that makes it worse. The core challenge is that you can't easily measure label quality without having correct labels to compare against, which creates a circularity problem. Here are the proven methods.
Inter-annotator agreement (IAA)
Have multiple labelers label the same items independently, then measure agreement. Cohen's Kappa and Krippendorff's Alpha are the standard metrics — they measure agreement beyond what you'd expect by chance. Kappa > 0.8 is excellent, 0.6–0.8 is good, below 0.6 signals labeling guideline problems. Low IAA doesn't mean your labelers are bad — it usually means your task definition is ambiguous.
Gold standard questions
Insert pre-labeled 'gold' items (where you know the correct answer) into the labeling stream without telling labelers which items are gold. Track each labeler's accuracy on gold items to identify labelers who are guessing, rushing, or systematically misunderstanding the task. Remove labelers below a quality threshold (typically 85–90% accuracy on gold items) and re-label their completed work.
Consensus labeling
Each item gets labeled by 3–5 labelers, and you use majority vote as the final label. For items where labelers disagree, escalate to an expert reviewer. Consensus dramatically reduces random labeling errors but doesn't catch systematic errors (if all labelers misunderstand the same guideline, consensus amplifies the mistake). Cost: 3–5x the base labeling cost.
Adjudication workflows
When labelers disagree, don't just take majority vote — route the disagreement to a senior reviewer who examines the item and the competing labels and makes a final decision. Document the adjudication reasoning: these decisions become the case law that clarifies your labeling guidelines. Over time, adjudication logs build a library of edge case resolutions that new labelers can reference.
The labeling guideline is your most important document
Every labeling quality problem traces back to the labeling guidelines. Good guidelines include: a clear definition of each category with 3–5 examples, explicit instructions for edge cases, a decision tree for ambiguous inputs, and a list of common mistakes with corrections. Treat your labeling guideline as a living document — update it every time an adjudication reveals an ambiguity, and re-train labelers when guidelines change.
Build Data-Driven AI Products in the Masterclass
Data strategy, labeling operations, evaluation design, and the full AI product development lifecycle are covered in the AI PM Masterclass — taught by a Salesforce Sr. Director PM.
Managing Labeling Operations at Scale
Labeling at scale is an operations problem, not a technology problem. The technology is straightforward: annotation tools, task queues, quality dashboards. The hard part is managing labeler productivity, maintaining quality as you scale, handling the logistics of data pipelines, and making the right build-vs-buy decisions.
Labeling platform selection
Build vs. buy depends on volume and specialization. Below 10,000 labels per month, use a managed platform (Label Studio, Labelbox, Scale AI, Prodigy). Above 100,000 labels per month with domain-specific workflows, building custom tooling often pays off. The key features to evaluate: annotation interface customization, quality metrics dashboards, integration with your ML training pipeline, and support for active learning (prioritizing which items to label next).
PM decision: Managed platforms charge $0.05–$2.00 per label plus platform fees. Calculate your monthly labeling volume to determine break-even for building in-house tooling.
Labeler onboarding and calibration
New labelers take 2–4 weeks to reach full quality regardless of experience. Structure onboarding in phases: (1) Read guidelines and study examples. (2) Label a calibration set of 50–100 items and compare against gold labels. (3) Review errors with a reviewer. (4) Label a second calibration set — only move to production if accuracy exceeds threshold. Skipping calibration is the most common operational mistake: it floods your dataset with low-quality labels from untrained labelers.
PM decision: Invest upfront in labeler calibration or pay later in data cleaning. The upfront investment almost always produces better data and costs less total.
Throughput vs. quality management
Labelers who work faster produce more labels but typically at lower quality. Track both metrics per labeler: labels per hour and accuracy on gold items. Set minimum quality thresholds and don't reward speed at the expense of quality. The optimal productivity target varies by task complexity — simple binary classification might allow 200+ labels per hour, while complex multi-label annotation might only sustain 20–30 per hour at acceptable quality.
PM decision: Set quality floors first, then optimize throughput within those constraints. Never set throughput targets without quality guardrails.
Data pipeline integration
Labels are only useful when they flow into your training pipeline. Automate the pipeline: raw data enters the annotation queue, labeled data gets quality-checked, approved labels merge into the training dataset, and the model retrains automatically when enough new labels accumulate. Manual handoffs between labeling and training create bottlenecks and stale data. The goal is a continuous pipeline where labeling improvements translate to model improvements within days, not months.
PM decision: The value of labels depreciates over time if they sit unused. Invest in pipeline automation to close the loop between labeling and model improvement.
When to Invest in More Labels vs. Better Labels
This is the most important strategic question in data labeling: should you label more data or improve the quality of your existing labels? The answer depends on where you are on the learning curve and what's actually bottlenecking your model.
Invest in more labels when...
Your model is underfitting (training accuracy is low). You have high-quality labels but not enough of them. Specific categories have fewer than 100 training examples. Adding 2x more data in a recent test improved accuracy. You're building a new model from scratch. The practical test: label 500 more examples and retrain — if accuracy improves meaningfully, you need more data.
Invest in better labels when...
Your model is overfitting (training accuracy is high, test accuracy is low). Inter-annotator agreement is below 0.7. Your labeling guidelines haven't been updated in months. Error analysis shows the model is confidently wrong on cases where labels are ambiguous. You have enough volume but the model plateaued. The practical test: re-label 500 items with stricter quality controls — if accuracy improves, you have a quality problem.
Active learning: label smarter, not more
Instead of randomly sampling items to label, use the model's uncertainty to prioritize. Items where the model is least confident are the most informative training examples. Active learning can reduce the total labels needed by 50–80% compared to random sampling. The workflow: train model on initial labels, run model on unlabeled data, send lowest-confidence items to labelers, retrain, repeat.
Data augmentation: multiply your labels
For some tasks, you can generate synthetic variations of labeled examples to multiply your effective dataset size. Text augmentation (synonym replacement, back-translation, paraphrasing with LLMs), image augmentation (rotation, cropping, color shifts), and structured data augmentation (noise injection, feature perturbation) can 3–10x your dataset. Always validate that augmented data doesn't degrade model performance — test on a held-out set of original (non-augmented) data.
The data flywheel every AI PM should build
The best AI products create a self-reinforcing labeling loop: user interactions generate raw data, model predictions on that data create candidate labels, confident predictions become training data automatically, uncertain predictions route to human review, human corrections become high-value training examples. This flywheel means your model improves continuously from production usage rather than requiring periodic manual labeling campaigns. Building this loop is one of the highest-leverage things an AI PM can do.