Overview
Direct Answer
A memory optimisation technique in transformer-based models that caches previously computed key and value tensors during autoregressive generation, eliminating redundant recalculation as each new token is produced. This mechanism significantly reduces computational overhead during inference without altering model outputs.
How It Works
During token generation, the transformer computes queries for the current token whilst reusing cached key-value pairs from prior positions rather than recomputing them. The cache is sequentially extended as each new token is generated, allowing attention operations to access historical representations in constant rather than quadratic time relative to sequence length. Modern implementations store these tensors in GPU memory or system RAM, depending on batch size and model dimensions.
Why It Matters
Key-value caching reduces inference latency by 2–3× on typical sequence lengths, directly lowering operational costs for production language models and enabling real-time interactive applications. For resource-constrained environments and large-scale deployments, this optimisation determines practical feasibility of transformer inference at scale.
Common Applications
Used extensively in conversational AI systems, real-time code generation tools, and streaming text summarisation services. Dialogue systems relying on multi-turn interactions particularly benefit from avoiding reprocessing of prior conversation history.
Key Considerations
Cache memory consumption scales linearly with batch size and sequence length, creating practical limits on concurrency and maximum context window. Careful management is required to prevent memory exhaustion, and cache invalidation strategies vary across frameworks and hardware configurations.
Cross-References(2)
More in Deep Learning
Vanishing Gradient
ArchitecturesA problem in deep networks where gradients become extremely small during backpropagation, preventing earlier layers from learning.
Pooling Layer
ArchitecturesA neural network layer that reduces spatial dimensions by aggregating values, commonly using max or average operations.
Self-Attention
Training & OptimisationAn attention mechanism where each element in a sequence attends to all other elements to compute its representation.
Pre-Training
Language ModelsThe initial phase of training a deep learning model on a large unlabelled corpus using self-supervised objectives, establishing general-purpose representations for downstream adaptation.
Knowledge Distillation
ArchitecturesA model compression technique where a smaller student model learns to mimic the behaviour of a larger teacher model.
Mixed Precision Training
Training & OptimisationTraining neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.
Fine-Tuning
ArchitecturesThe process of taking a pretrained model and further training it on a smaller, task-specific dataset.
Fully Connected Layer
ArchitecturesA neural network layer where every neuron is connected to every neuron in the adjacent layers.