Token Efficiency for AI Products: Cut LLM Costs Without Cutting Quality

Why Token Count Is Your Hidden Product Variable

Every LLM API call has a price, and that price is denominated in tokens. GPT-5.6 Sol charges $5 per million input tokens and $30 per million output tokens. Claude Sonnet 5 charges $3 and $15. At small scale this feels trivial. At production scale, with thousands of daily active users sending multi-turn conversations or processing long documents, token spend becomes a major line item that directly determines whether your AI feature is profitable.

Most product teams discover the token problem after launch, when the cloud bill arrives. The better approach is to treat token count as a first-class design constraint, alongside latency and accuracy. A feature that produces identical output quality at half the token count is not just cheaper — it is a faster, more scalable product.

Input tokens

Everything you send to the model: the system prompt, conversation history, retrieved documents, tool definitions, and the user's message. Input tokens are typically priced 5-10x cheaper than output tokens.

Output tokens

The model's response. Output is where cost concentrates — the model generates tokens one at a time, which is both slow and expensive. Reducing unnecessary verbosity in responses is high-leverage.

Cached input tokens

Most major providers now offer prompt caching, where repeated prefixes are stored and not re-billed at full price. Cache hits on Claude cost 10% of normal input price, making caching a structural cost reduction.

Context window tax

Longer contexts cost more even when the model doesn't 'need' all of them. A 100K-token system prompt processed on every request is a 100K-token bill on every request, regardless of how much of it was actually relevant.

The 5 Biggest Token Wasters in AI Products

Before optimizing, diagnose. Most AI products bleed tokens from the same five sources. Check each one before writing a single line of optimization code.

1. Bloated system prompts

Symptom: System prompts that grew over months of iteration, now containing redundant instructions, example inputs/outputs that could live in the user turn, and paragraphs written for edge cases that represent 0.1% of traffic.

Fix: Audit your system prompt with a token counter monthly. Cut every instruction that can be moved to a few-shot example or that handles an edge case rarely triggered. Typical savings: 30-50% of system prompt length.

2. Full conversation history in every request

Symptom: Multi-turn chat products that send the entire conversation history on every turn — including messages from 20 turns ago that have zero bearing on the current question.

Fix: Implement conversation summarization: after N turns, compress older turns into a running summary. Summarize with a small model (Haiku-class). Store the summary, discard the raw turns. Typical savings: 60-80% on long conversations.

3. Over-retrieved RAG context

Symptom: Retrieval-augmented generation setups that pull 10-20 chunks 'to be safe,' when the model only needs 2-3 to answer the question. Unused retrieved context is pure cost with zero benefit.

Fix: Add a reranker layer to select only the top 2-3 chunks after retrieval. Or use an LLM-as-judge step to filter irrelevant chunks before the final call. The reranker/filter call costs less than the wasted context tokens.

4. Unnecessary output verbosity

Symptom: System prompts that instruct the model to 'be thorough' or 'explain your reasoning' when the user interface only displays the conclusion — the reasoning tokens are generated, billed, and then discarded by the UI.

Fix: Match your output instructions to what actually gets displayed. For structured outputs, use JSON mode to suppress prose padding. For summaries, specify a target length in sentences, not in 'be comprehensive.'

5. No model routing

Symptom: Using a frontier model (GPT-5.6 Sol, Claude Sonnet 5) for every request, including simple classification tasks, keyword extraction, or yes/no decisions that a 10x cheaper small model handles equally well.

Fix: Map request types to model tiers: intent classification runs on a small model, complex synthesis runs on a frontier model. A simple routing layer pays for itself in days. Typical savings: 40-70% of total API spend.

Prompt Caching: The Fastest Structural Win

Prompt caching is the single highest-leverage token optimization for products with a stable system prompt or shared knowledge base. Anthropic, OpenAI, and Google all now offer it. The economics are striking: on Anthropic's API, cached input tokens cost $0.30 per million (instead of $3.00 — a 90% reduction). On OpenAI, cached tokens cost 50% of the normal rate.

The mechanism is simple: the provider hashes a prefix of your prompt. If the same prefix has been seen recently (within the TTL, typically 5 minutes on Anthropic), the provider returns the cached KV state and charges the cache rate instead of the full input rate. Your system prompt, retrieved knowledge base, or tool definitions are sent once and reused many times.

When caching saves the most:

✓Large system prompts (1,000+ tokens) sent on every request

✓Knowledge base documents injected into every RAG call

✓Tool definitions for agent-based products with many tools

✓Few-shot examples that are static and don't change per user

✓Multi-turn conversations where the early context is stable

To enable caching on Anthropic, add a cache_control: {"type": "ephemeral"} breakpoint at the end of the prefix you want cached. Structure your prompts so the stable content (system prompt, knowledge base) comes first and the dynamic content (user message) comes last. The cache only applies to the prefix before the breakpoint.

PM tip: Model caching in your cost projections

When estimating AI feature costs, use two figures: the no-cache cost (worst case, cache TTL expires between calls) and the full-cache cost (best case, all users hit the cache). Real costs land between them, depending on your request frequency. For products with multiple requests per user per hour, assume 80-90% cache hit rate. For low-frequency use cases, assume 30-50%.

Learn to Build Profitable AI Products

The AI PM Masterclass teaches you how to design AI features with cost, latency, and quality in mind from day one — taught live by a Salesforce Sr. Director PM.

Compression and Sampling Tradeoffs

Beyond caching, several prompt-design decisions directly reduce token count without changing what the model knows. These are engineering choices, but PMs need to understand them to weigh quality vs. cost tradeoffs during feature design.

Zero-shot vs. few-shot

Few-shot examples are expensive — each example can add 50-200 tokens to every request. They are worth it when the task is ambiguous and the model needs anchoring. For well-defined, high-frequency tasks (classifying sentiment, extracting a phone number), zero-shot with a precise task description typically matches few-shot quality at a fraction of the cost. Test both before defaulting to few-shot.

Quality risk: zero-shot may produce more formatting variation. Mitigation: use structured outputs (JSON mode) to enforce consistency.

Summarization before injection

When injecting user-provided content (uploaded documents, email threads, web pages), summarize the content with a cheap small model first, then inject the summary into the frontier model's context. The summary typically represents 5-15% of the original token count while preserving the meaning needed for most queries.

Quality risk: summaries lose detail. Mitigation: keep the original available for retrieval on specific factual lookups.

Output format constraints

Asking the model to 'respond in JSON' or 'answer in one sentence' reduces output tokens dramatically. A model instructed to produce structured JSON for a simple extraction task uses 2-3x fewer tokens than the same model asked to 'explain and then provide the answer.' Constrain the output format to the minimum the downstream system actually needs.

Quality risk: over-constrained outputs may truncate nuance. Mitigation: tune constraints per task type, not globally.

Streaming and early stopping

With streaming enabled, your application receives tokens as they are generated. For use cases where the model reliably hits a stopping condition early (producing a JSON object, completing a fixed-length form), you can stop receiving tokens once the condition is met and cancel the generation. You pay only for tokens generated up to the cancel point.

Engineering complexity: requires managing streaming connections and cancellation logic. Worth it for high-volume structured output use cases.

Building Token Budgets Into Your Feature Design

Token efficiency is not a post-launch optimization — it is a design input. The best AI PMs treat token budget the same way infrastructure engineers treat compute budget: as a constraint that shapes design decisions from the start. Here is how to operationalize it.

Step 1: Define a per-request token target

Before writing the first version of your system prompt, set a target: how many input tokens should this request consume at steady state? Anchor on your model cost, desired margin, and expected request volume. A feature expected to serve 50,000 requests/day at $0.01 per request has a total token budget of roughly 3,300 input tokens per request (at GPT-5.6 Sol input pricing).

Step 2: Instrument your prompts

Add token-counting to your logging pipeline from day one. Log input tokens, output tokens, cached tokens, and cache hit rate on every request. Without this data, optimization is guesswork. Most providers return token counts in the API response — capture them.

Step 3: Set a token budget alert threshold

Alert when a feature's average token count exceeds your target by 20%. This catches prompt drift (engineers adding instructions without removing old ones), context bloat (conversations growing longer than modeled), and retrieval misconfiguration (RAG pulling too many chunks).

Step 4: Include token cost in your feature's success metrics

Track cost per output alongside quality metrics (task completion rate, user satisfaction). A feature that achieves 95% task completion at $0.008/request is better than one achieving 96% at $0.022/request if the quality delta does not justify the cost. Make this tradeoff explicit in your PRD and in post-launch reviews.

Step 5: Run periodic prompt audits

Every 60 days, re-read your system prompt with fresh eyes and ask: does every sentence earn its token count? Are there instructions that were added for edge cases that turned out to be rare? Have any instructions become redundant because the model has improved? Prompt bloat accumulates silently over time.

Real-world benchmark

Teams that instrument token usage from launch and run quarterly prompt audits typically see 40-60% cost reduction in the first 6 months without any quality degradation — because they are not optimizing a prompt that was already tight, they are removing the inevitable accretion of instructions that no single engineer thought to remove. The savings compound: lower token count also means lower latency, which improves user experience and enables tighter SLA commitments.