arrow_backBack to Dispatch
6 min read

Untitled

id: 77 slug: "what-is-kv-cache-explained" title: "What Is KV Cache? Key-Value Cache Explained for LLMs" date: "2026-07-14" excerpt: "KV cache in LLMs stores key-value pairs to avoid redundant computation. How it works, why latency matters, and optimization." category: "AI Development" author: "4M Labs Engineering"

What Is KV Cache? Key-Value Cache Explained for LLMs

KV cache (key-value cache) is a memory optimization technique used in large language models (LLMs) that stores previously computed key and value tensors from attention layers so they do not need to be recomputed for every new token generation. When an LLM generates text token by token, KV cache avoids redundant computation by reusing the attention states from earlier positions, reducing inference latency from quadratic to linear complexity relative to sequence length.

This guide explains what KV cache is, how it works, why it matters for LLM performance, and how to optimize it for production deployments.

How Attention Works Without KV Cache

To understand KV cache, you first need to understand how self-attention works in transformer models. In the attention mechanism, each token in a sequence produces three vectors:

  • Query (Q): What the token is looking for
  • Key (K): What the token offers to other tokens
  • Value (V): The actual information the token carries

When generating a new token, the model computes attention scores between the new token's query and all previous tokens' keys, then uses those scores to weight the values. Without KV cache, the model recomputes keys and values for every token in the sequence every time it generates a new token.

For a sequence of length N, this means:

  • Token 1: Compute K, V for 1 token
  • Token 2: Recompute K, V for 2 tokens
  • Token 3: Recompute K, V for 3 tokens
  • Token N: Recompute K, V for N tokens

Total computation: O(N^2) in the attention layer, which becomes prohibitively expensive for long sequences.

How KV Cache Solves This Problem

KV cache stores the key and value tensors computed at each step so they do not need to be recomputed. When generating token N+1, the model:

  1. Computes Q, K, V only for the new token (token N+1)
  2. Retrieves cached K and V tensors for tokens 1 through N
  3. Concatenates the new K and V with the cached values
  4. Computes attention using the full set of keys and values
  5. Stores the new K and V in the cache for the next step

This reduces the computation from O(N^2) to O(N) for the attention layer, making long-context LLM inference practical.

Concrete Example

Consider a model generating a 1,000-token response:

Without KV cache:

  • Token 1: Compute K, V for 1 token
  • Token 2: Compute K, V for 2 tokens
  • ...
  • Token 1000: Compute K, V for 1000 tokens
  • Total: 1 + 2 + 3 + ... + 1000 = 500,500 key-value computations

With KV cache:

  • Token 1: Compute K, V for 1 token, store in cache
  • Token 2: Compute K, V for 1 new token, retrieve cached values
  • ...
  • Token 1000: Compute K, V for 1 new token, retrieve cached values
  • Total: 1,000 key-value computations (one per token)

The reduction is dramatic: from 500,500 to 1,000 computations in the attention layer.

Why KV Cache Matters for Production LLMs

Latency Reduction

KV cache directly reduces time-to-first-token (TTFT) and inter-token latency. Without it, each token generation would take progressively longer as the context grows. With KV cache, token generation time remains roughly constant regardless of context length.

For a production chatbot handling 1,000-token conversations, KV cache can reduce inference time by 10-100x depending on model size and hardware.

Throughput Improvement

By reducing per-token computation, KV cache allows serving more concurrent users on the same hardware. This directly translates to lower infrastructure costs for production deployments.

Context Window Enablement

Modern LLMs support context windows of 128K, 200K, or even 1M tokens. Without KV cache, generating even 1,000 tokens in such a context would be computationally infeasible. KV cache makes long-context inference practical.

KV Cache Memory Requirements

The tradeoff for KV cache is memory usage. Storing key and value tensors for every token in every attention layer consumes significant GPU memory.

Memory Calculation

For a model with:

  • L layers
  • H attention heads
  • D head dimension
  • S sequence length
  • P precision (bytes per element, typically 2 for float16)

Memory for KV cache = 2 (K and V) x L x H x D x S x P

Example for Llama 3.1 8B:

  • L = 32 layers
  • H = 32 attention heads
  • D = 128 head dimension
  • S = 8,192 sequence length
  • P = 2 bytes (float16)

KV cache memory = 2 x 32 x 32 x 128 x 8,192 x 2 = 4.3 GB

For a 128K context window, this grows to approximately 68 GB, which exceeds the memory of a single GPU.

Scaling Strategies

To manage KV cache memory at scale:

  1. Quantization: Store KV cache in INT8 or INT4 precision, reducing memory by 2-4x with minimal quality loss
  2. Paged attention: Allocate KV cache memory dynamically in pages (used by vLLM), reducing fragmentation
  3. Group-query attention (GQA): Share key-value heads across query heads, reducing cache size by 4-8x
  4. Sliding window attention: Limit attention to a fixed window, capping cache growth
  5. Offloading: Move older KV cache entries to CPU memory when GPU memory is constrained

KV Cache Optimization for Production

Batching Strategy

Efficient batching maximizes KV cache utilization:

  • Continuous batching: Add new requests to existing batches as old ones complete, keeping GPU utilization high
  • Prefix sharing: Share KV cache for common prefixes (system prompts) across multiple requests
  • Prompt caching: Cache KV cache for frequently used prompts to avoid recomputation

Memory Management

  • Eviction policies: When memory is full, evict oldest or least-reused cache entries
  • Compression: Quantize or prune KV cache to reduce memory footprint
  • Memory pools: Pre-allocate KV cache memory to avoid allocation overhead

Hardware Selection

KV cache memory requirements influence hardware choice:

  • A100 80GB: Suitable for models up to 70B parameters with moderate context lengths
  • H100 80GB: Better throughput for large models with long contexts
  • Multi-GPU: Required for models with very long context windows (128K+)
  • CPU offload: Viable for non-latency-critical workloads

Common KV Cache Issues and Solutions

Out-of-Memory Errors

Symptom: CUDA out of memory during inference Solution: Reduce batch size, use quantized KV cache, enable paged attention, or offload to CPU

Stale Cache Entries

Symptom: Incorrect or outdated responses when reusing cached prompts Solution: Invalidate cache when prompts change, implement cache versioning

Memory Fragmentation

Symptom: Available memory is insufficient despite low utilization Solution: Use paged attention (vLLM) or pre-allocated memory pools

Context Length Limitations

Symptom: Model fails to process long documents Solution: Implement sliding window attention, use chunked processing, or enable KV cache compression

Frequently Asked Questions

What is the difference between KV cache and prompt caching?

KV cache stores key-value tensors from attention layers for individual inference requests. Prompt caching stores KV cache for entire prompts that are reused across multiple requests. Prompt caching is a higher-level optimization that uses KV cache as its underlying mechanism. For example, if 100 users send the same system prompt, prompt caching computes the KV cache once and shares it across all 100 requests.

How much memory does KV cache use?

KV cache memory depends on model size and sequence length. For a 7B parameter model with 8K context, KV cache uses approximately 1-2 GB. For a 70B model with 128K context, it can exceed 100 GB. Quantization and GQA can reduce this by 2-8x.

Can I disable KV cache?

You can disable KV cache, but it is not recommended for production use. Without KV cache, inference becomes dramatically slower as context length increases. The only practical use case for disabling KV cache is during training or when memory is extremely constrained and sequences are very short.

What is paged attention?

Paged attention (used by vLLM and TensorRT-LLM) manages KV cache memory in fixed-size pages rather than contiguous blocks. This eliminates memory fragmentation, enables dynamic memory allocation, and improves GPU utilization by 2-4x compared to traditional KV cache management.

How does KV cache affect cost?

KV cache memory requirements directly influence GPU selection and batch size, which determine serving cost. Larger KV cache means fewer concurrent requests per GPU, which increases cost per request. Optimizing KV cache through quantization, GQA, and efficient batching can reduce serving costs by 30-60%.

Related Resources