The KV cache landscape is evolving with quantization approaches like KIVI, KVQuant, and TurboQuant offering significant memory reductions—KIVI uses 2-bit per-channel keys with per-token values without fine-tuning for up to 2.6× memory and 4× larger batches; KVQuant uses calibrated mixed-precision with non-uniform quantization and context-decomposition; TurboQuant achieves roughly 6× memory reduction at 3-bit precision through a two-stage process with PolarQuant rotation and QJL correction, all without calibration, prioritizing hardware-friendly efficiency with minimal to no extra training.

StreamingLLM keeps initial tokens as attention sinks to stabilize long-context processing, trading some semantic fidelity in mid-context for faster, simpler processing.

SnapKV focuses on the prefill stage by using an observation window and head-specific pooling to identify important KV positions, often outperforming H2O on benchmarks such as LongBench.

By 2026, research is trending toward latent-space compaction and reasoning-aware compression (examples include Attention Matching and TriAttention), signaling a move beyond traditional KV strategies toward substantial memory reductions.

Token eviction methods like H2O, StreamingLLM, and SnapKV reduce memory by deprioritizing or discarding tokens, with H2O achieving notable throughput gains up to 29× under certain conditions.

Low-rank and latent-space approaches (e.g., MLA/DeepSeek) apply projection-based compression to KV representations, achieving dramatic reductions—up to roughly 93% KV cache reduction in DeepSeek-V2—and can outperform standard attention within the same budget.

Layer-aware allocation in PyramidKV and PyramidInfer assigns memory budgets per Transformer layer to reflect information density decay, boosting memory efficiency and throughput.

Architectural and training-time choices like Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce KV cache needs, though GQA requires retraining and has become common in recent open-weight LLMs.

KV cache size grows with sequence length and batch size, creating memory bottlenecks in production inference, with concrete comparisons between 30B and 7B models illustrating the challenge.

Low-rank KV methods such as Palu and LoRC target the hidden dimension rather than sequence length or bit width, offering a balance between accuracy and reconstruction overhead and complementing quantization and eviction strategies.

The practical takeaway is that KV cache bottlenecks can be alleviated through a mix of eviction, quantization, low-rank, and architectural strategies, with deployment choices driven by training requirements, hardware, and the desired balance of memory, throughput, and accuracy.