In-Context Learning Explained for Product Managers

Classical machine learning has a clear learning loop: show the model labeled examples, compute a loss, run gradient descent, update the weights. Repeat thousands of times. Learning is baked into the parameters. When inference runs, those parameters are fixed.

In-context learning breaks this assumption. You show the model a handful of input-output pairs inside the prompt, and the model adapts its behavior for your task without any gradient descent and without changing a single weight. The same frozen model produces different outputs depending on what examples appear in its context window.

This emerged as a surprise in the 2020 GPT-3 paper. Smaller models did not show it in any meaningful way. Above roughly 100 billion parameters, the behavior appeared as what researchers called an emergent capability: something that was not explicitly trained for but arose from scale. The working explanation is that very large models internalize so much structure about language and tasks during pre-training that reading a few examples is enough to activate the right behavior. The model recognizes what task is being described and routes accordingly.

Mechanistically, this happens through the attention mechanism during the forward pass. As the model processes each token in your prompt, attention heads are reading the examples you provided and building up an implicit representation of the task. By the time the model reaches the end of your input and starts generating output, it has already done the equivalent of light task inference from pattern recognition across your examples.

ICL covers a range of example counts, each with a different cost and benefit profile. Understanding the spectrum lets you pick the right point for your product without over-engineering.

The conventional advice was that more examples always helps up to a point. That is still roughly true, but with two important caveats. First, quality dominates quantity. Five well-chosen, consistently formatted examples outperform twenty messy ones. Second, many-shot ICL is now a real option. With context windows reaching 1 million tokens on models like Gemini 1.5 Pro and Claude 3, you can fit hundreds of labeled examples into a single prompt. Research published in 2024 showed many-shot ICL closing much of the gap with fine-tuning on certain classification tasks, without any training infrastructure.

These three approaches are often presented as competing alternatives, but they address different problems. The decision is not which is best in general. The decision is which is right for your specific constraint set today.

The most common mistake product teams make is jumping to fine-tuning before exhausting ICL. Fine-tuning feels like a real investment and therefore like a signal of seriousness. But it introduces training infrastructure, model snapshots that go stale, and regression risk on every base model upgrade. ICL with thoughtful examples solves the problem faster and keeps you flexible while the task definition is still evolving.

The second most common mistake is conflating factual grounding with task adaptation. RAG is for knowledge. ICL is for behavior. If you need the model to know something specific (a policy document, a product catalog, last week's news), RAG is the right tool. If you need the model to respond in a specific way to a class of inputs, ICL is the right tool. Many products need both.

Researchers have proposed several explanations for what ICL is actually doing mechanistically. None is complete, but they give useful intuitions for product decisions.

The most supported view is that ICL does at least three things simultaneously. First, it establishes the label space and format: by seeing your output examples, the model knows what kinds of answers are valid and how to structure them. Second, it resolves task ambiguity: a task description in natural language can mean many things, and examples pin down the intended interpretation. Third, it activates latent task knowledge: if the task is something the model encountered during pre-training in some form, examples trigger that knowledge to surface.

This explains both why ICL works well and where it breaks down. It works well when the task resembles something in the pre-training distribution and when your examples consistently demonstrate the right format. It breaks down in the following specific ways:

A useful mental model: ICL is pattern matching against your examples, not reasoning about your intent. Every failure mode follows from taking that model seriously. If you show inconsistent patterns, the match is unreliable. If production inputs do not match your example patterns, the match fails. If your examples do not cover ambiguous cases, the model matches on incomplete signal.

ICL is not just a prompting technique. It reframes several product decisions that PMs are typically not trained to think about. Here are the six most important ones.