Advanced Retrieval Strategies: Beyond Basic RAG for AI Products

Embedding similarity measures semantic closeness, but users often describe problems using different vocabulary than the source documents use. A user asking 'why is my app crashing?' won't match a document titled 'Memory allocation errors in containerized deployments' even though the answer is there. Dense embeddings capture semantic meaning but miss lexical overlap that keyword search would catch.

Trade-off: This is the fundamental limitation of vector-only retrieval. Solutions include hybrid search (combining dense and sparse retrieval) and query expansion. Fixing this alone typically improves recall by 15-25%.

Bi-encoder embeddings (used by most embedding models) compute query and document vectors independently, then compare them with cosine similarity. This is fast but crude. A query and document might be 'similar' in embedding space without the document actually answering the query. The document might discuss the same topic but from an irrelevant angle, contain outdated information, or address a different aspect entirely.

Trade-off: Cross-encoder rerankers solve this by jointly encoding the query-document pair, producing much more accurate relevance scores. The cost is latency: reranking adds 50-200ms per query depending on the number of candidates.

Many real questions require information from multiple documents that individually don't contain the answer. 'What's our churn rate for enterprise customers who onboarded after the pricing change?' requires joining information across pricing change dates, customer segments, and churn metrics. Basic top-k retrieval finds documents similar to the query, not documents that together compose an answer.

Trade-off: Query decomposition and agentic retrieval address this by breaking complex queries into sub-queries or letting the LLM iteratively search for the information it needs. These approaches are slower but dramatically better for analytical questions.

Retrieving more chunks increases the chance of including the right information, but it also increases noise. Long contexts with irrelevant information degrade LLM answer quality — models get distracted by plausible-but-wrong context, especially content in the middle of the prompt (the 'lost in the middle' effect). Basic RAG has no mechanism to filter out low-value retrieved content.

Trade-off: Contextual compression solves this by summarizing or filtering retrieved chunks before injection, keeping only the information relevant to the specific query. This reduces prompt size and improves answer accuracy simultaneously.

Combine vector similarity search (dense retrieval) with traditional keyword search (sparse retrieval, typically BM25). Dense retrieval excels at semantic matching — understanding that 'revenue growth' and 'top-line expansion' mean the same thing. Sparse retrieval excels at exact term matching — finding documents that contain specific product names, error codes, or technical terms. Run both searches in parallel, then merge results using Reciprocal Rank Fusion (RRF) or a learned linear combination. Most production RAG systems use a 0.7 dense / 0.3 sparse weighting as a starting point, then tune based on query logs.

Trade-off: Adds BM25 index maintenance and a fusion layer. Infrastructure cost increases modestly, but recall improvements of 15-30% are typical. Nearly every production RAG system should use hybrid search — the cost-benefit ratio is overwhelmingly positive.

After initial retrieval returns the top 20-50 candidates, pass each query-document pair through a cross-encoder model (like Cohere Rerank, BGE-reranker, or a fine-tuned model) that jointly attends to both query and document. Cross-encoders produce far more accurate relevance scores than bi-encoder cosine similarity because they can model fine-grained interactions between query terms and document content. Rerank the candidates, then take the top 3-5 for the final prompt. This two-stage pipeline (fast retrieval, then accurate reranking) is the standard architecture for high-quality RAG.

Trade-off: Adds 50-200ms of latency per query (depending on the number of candidates and reranker model size). For user-facing products where answer quality matters more than sub-second latency, this is almost always worth it. For high-throughput, latency-sensitive applications, consider caching reranked results for common queries.

Use the LLM to break a complex query into simpler sub-queries before retrieval. 'Compare our Q1 and Q2 customer acquisition costs across enterprise and SMB segments' becomes four separate retrieval queries: Q1 enterprise CAC, Q1 SMB CAC, Q2 enterprise CAC, Q2 SMB CAC. Each sub-query retrieves more targeted chunks than the original compound query would. The sub-results are then combined and passed to the LLM for synthesis. This is critical for analytical, comparative, or multi-entity queries that a single embedding can't represent well.

Trade-off: Multiplies the number of retrieval calls (and LLM calls for decomposition). Latency increases proportionally. Best applied selectively — use a query classifier to route simple queries directly and only decompose complex ones. The decomposition step itself can fail if the LLM misunderstands the query structure.

After retrieving chunks, use a smaller LLM or a trained extractor to compress each chunk, keeping only the sentences relevant to the specific query. A 500-token chunk about employee benefits might contain one paragraph about parental leave policy — if the query is about parental leave, contextual compression extracts just that paragraph. This reduces the total context size passed to the generator LLM, lowers cost, and eliminates the 'lost in the middle' effect by removing distracting content.

Trade-off: Adds an LLM call per retrieved chunk (or a batch call for all chunks). The compression model can occasionally remove relevant information, so evaluation is essential. For cost optimization, use a small, fast model for compression — this is a low-complexity task that doesn't need a frontier model.

Instead of a fixed retrieve-then-generate pipeline, give the LLM retrieval tools and let it decide what to search for, evaluate the results, and iteratively refine its searches. The LLM might search for 'parental leave policy,' find a reference to 'Policy Document 2024-HR-12,' then search for that specific document, extract the relevant section, and then answer. This is the most powerful retrieval strategy because the LLM applies reasoning to the retrieval process itself — but it's also the most expensive and hardest to control.

Trade-off: Latency is unpredictable (1-10+ seconds depending on how many retrieval iterations the agent performs). Cost scales with iterations. Requires careful guardrails to prevent infinite loops or irrelevant searches. Best for complex, high-value queries where correctness matters more than speed — internal knowledge bases, legal research, technical support escalation.