arrow_backBack to Dispatch
7 min readAI Engineering

LLM Retrieval: How RAG Works in 2026

LLM retrieval connects language models to external knowledge. How RAG works, strategies, embeddings, vector DBs, and production.

LLM retrieval is the process of fetching relevant external information and feeding it to a large language model before it generates a response. Instead of relying solely on the model's training data (which has a cutoff date and cannot include your private documents), retrieval gives the LLM access to real-time, domain-specific, or proprietary knowledge. The most common implementation is Retrieval-Augmented Generation (RAG), which combines a retrieval system with a language model to produce grounded, accurate, and verifiable outputs.

For engineering teams building AI-powered products, understanding LLM retrieval is not optional. It is the foundation for building AI systems that actually work in production rather than impressive demos that fail when users ask about real data.

Why LLM Retrieval Matters

Large language models have three fundamental limitations that retrieval solves:

Knowledge Cutoff

Every LLM has a training data cutoff. GPT-4's training data stops at a specific date. Claude has its own cutoff. This means the model literally does not know about events, products, or data published after that date. Retrieval gives the model access to current information.

Hallucination

When an LLM does not have enough information to answer a question accurately, it fills in gaps with plausible-sounding but fabricated content. This is not a bug. It is how the model works. Retrieval reduces hallucination by providing the model with actual source material to reference.

Private Data Access

LLMs are trained on public data. They do not know about your company's internal documentation, your customer data, your product specifications, or your proprietary processes. Retrieval connects the model to your private knowledge base.

How LLM Retrieval Works

The retrieval process follows a pipeline with distinct stages.

1. Document Ingestion

Before retrieval can happen, you need to prepare your documents:

Chunking: Split documents into smaller pieces. The optimal chunk size depends on the embedding model and the use case, but 200-500 tokens per chunk is a common starting point. Overlapping chunks (50-100 token overlap) help maintain context across chunk boundaries.

Embedding: Convert each chunk into a vector representation using an embedding model. The embedding captures the semantic meaning of the text. Common embedding models include OpenAI text-embedding-3-small, Cohere embed-v3, and open-source models like BGE-large.

Indexing: Store the vectors in a vector database for fast similarity search. Popular vector databases include Pinecone, Weaviate, Qdrant, pgvector (PostgreSQL extension), and ChromaDB.

2. Query Processing

When a user asks a question:

Query embedding: Convert the user's question into a vector using the same embedding model used for documents.

Similarity search: Find the most similar document chunks by comparing the query vector to stored vectors. The similarity metric is typically cosine similarity or dot product. Return the top-k most similar chunks (usually 5-20 chunks).

Re-ranking (optional): Use a cross-encoder model to re-rank the retrieved chunks by relevance. This adds latency but improves retrieval quality significantly.

3. Context Assembly

Combine the retrieved chunks with the user's question into a prompt:

You are a helpful assistant. Answer the question based on the following context.

Context:
[Retrieved chunk 1]
[Retrieved chunk 2]
[Retrieved chunk 3]

Question: [User's question]

The context window of the LLM limits how many chunks you can include. GPT-4 supports 128K tokens, Claude supports 200K tokens. In practice, 3-10 chunks (2,000-5,000 tokens) are usually sufficient.

4. Generation

The LLM generates a response using both the retrieved context and its training knowledge. The model can now:

  • Answer questions about your specific data
  • Cite sources from the retrieved chunks
  • Provide accurate, up-to-date information
  • Reduce hallucination by grounding responses in retrieved facts

Retrieval Strategies

Dense Retrieval

Dense retrieval uses vector embeddings to find semantically similar documents. This is the most common approach and works well for most use cases.

Pros:

  • Captures semantic meaning (finds related content even with different wording)
  • Works across languages
  • Improves with better embedding models

Cons:

  • May miss exact keyword matches
  • Requires embedding computation for all documents
  • Sensitive to embedding model quality

Sparse Retrieval

Sparse retrieval uses traditional keyword matching algorithms like BM25. This approach finds documents that contain the exact query terms.

Pros:

  • Fast and simple to implement
  • Exact keyword matching
  • No embedding computation required

Cons:

  • Misses semantic similarity
  • Requires careful tokenization
  • Does not handle synonyms well

Hybrid Retrieval

Hybrid retrieval combines dense and sparse approaches. You run both retrieval methods in parallel and merge the results.

Pros:

  • Best of both worlds
  • Higher recall than either method alone
  • More robust to different query types

Cons:

  • More complex to implement
  • Requires maintaining two retrieval systems
  • Higher latency

Query Transformation

Before retrieval, transform the user's query to improve retrieval quality:

Query expansion: Generate multiple variations of the query to increase recall. HyDE (Hypothetical Document Embeddings): Generate a hypothetical answer to the query, then use the answer's embedding for retrieval. Step-back prompting: Generate a more general version of the query to retrieve broader context.

Embedding Models

The embedding model determines the quality of your retrieval system.

Commercial Options

OpenAI text-embedding-3-small: Good balance of quality and cost. $0.02 per 1M tokens. OpenAI text-embedding-3-large: Higher quality, higher cost. $0.13 per 1M tokens. Cohere embed-v3: Strong multilingual support. Good for non-English content.

Open-Source Options

BGE-large (BAAI): High quality, 335M parameters. Requires GPU for inference. E5-large-v2: Strong performance on retrieval benchmarks. Nomic-embed-text: Lightweight, good for resource-constrained environments.

Selection Criteria

Choose an embedding model based on:

  • Quality: How well does it retrieve relevant documents?
  • Speed: How fast can it compute embeddings?
  • Cost: What is the per-token cost?
  • Dimension: How many dimensions does the vector have? (Higher dimensions = more storage, potentially better quality)
  • Language support: Does it support your document languages?

Vector Databases

The vector database stores embeddings and enables fast similarity search.

Managed Options

Pinecone: Fully managed, easy to use, good for prototyping and production. Weaviate: Open-source with managed cloud option, supports hybrid search. Qdrant: Open-source with managed cloud, good performance at scale.

Self-Hosted Options

pgvector: PostgreSQL extension, good if you already use PostgreSQL. ChromaDB: Lightweight, good for development and small datasets. Milvus: High performance, good for large-scale deployments.

Selection Criteria

Choose a vector database based on:

  • Scale: How many documents do you need to index?
  • Latency: What is your required query response time?
  • Filtering: Do you need metadata filtering alongside vector search?
  • Cost: What is your budget for infrastructure?
  • Operations: Do you want managed or self-hosted?

Production Implementation

Architecture Patterns

Naive RAG: Simple retrieval + generation. Good for basic use cases. Advanced RAG: Adds query transformation, re-ranking, and chunk optimization. Better quality. Modular RAG: Separate retrieval, transformation, and generation modules. Most flexible.

Performance Optimization

Caching: Cache common queries and their results. Reduces latency and cost. Async processing: Run retrieval and generation in parallel when possible. Batch embedding: Compute embeddings for multiple documents at once. Index optimization: Use HNSW or IVF indexes for faster similarity search.

Quality Evaluation

Measure retrieval quality with these metrics:

Recall@k: What fraction of relevant documents are in the top-k results? Precision@k: What fraction of top-k results are relevant? MRR (Mean Reciprocal Rank): How high is the first relevant result? End-to-end accuracy: Does the RAG system produce correct answers?

Common Failure Modes

Retrieval miss: The relevant document is not in the top-k results. Fix: improve chunking, try different embedding models, or increase k. Context overflow: Too many chunks exceed the LLM's context window. Fix: reduce chunk count or use shorter chunks. Noisy context: Retrieved chunks are not relevant to the query. Fix: improve chunking strategy, add metadata filtering, or use re-ranking. Answer hallucination: The LLM ignores retrieved context and generates from training data. Fix: improve prompt engineering, add source citation requirements.

Frequently Asked Questions

What is the difference between RAG and fine-tuning?

RAG retrieves external information at query time and includes it in the prompt. Fine-tuning modifies the model's weights to encode knowledge permanently. RAG is better for dynamic, frequently changing data. Fine-tuning is better for domain-specific language patterns or style. Most production systems use RAG for knowledge and fine-tuning for behavior.

How many documents do I need for effective RAG retrieval?

Effective RAG retrieval works with any number of documents, from 10 to millions. The key is document quality and relevance, not quantity. A well-curated knowledge base of 100 high-quality documents often performs better than 10,000 low-quality ones. Start with your most important documents and expand based on user feedback.

What chunk size should I use for document retrieval?

The optimal chunk size depends on your embedding model and use case. A common starting point is 200-500 tokens per chunk with 50-100 token overlap. Smaller chunks provide more precise retrieval but may lose context. Larger chunks provide more context but may dilute relevance. Test different sizes with your actual data.

How do I handle queries that require information from multiple documents?

For queries requiring information from multiple documents, retrieve more chunks (10-20) and let the LLM synthesize across them. Alternatively, break complex queries into sub-queries, retrieve for each sub-query, and combine the results. Advanced techniques like multi-hop RAG chain retrieval across multiple documents.

What is the cost of running a RAG system in production?

Costs include: embedding computation ($0.02-0.13 per 1M tokens), vector database storage ($0.05-0.50 per GB per month), vector database queries ($0.0001-0.001 per query), and LLM inference for generation ($0.01-0.10 per query). Total cost per query typically ranges from $0.01 to $0.15 depending on the components used.

Related Resources