Overview
Direct Answer
Data parallelism is a distributed training approach in which an identical model is replicated across multiple devices, each processing different subsets of training data in parallel, with gradient updates synchronised across all replicas after each iteration. This strategy enables significant acceleration of training for large datasets without modifying the model architecture.
How It Works
Each device holds a complete copy of the model and processes a distinct batch of training examples independently. After the forward pass and backpropagation, gradients computed on each device are aggregated (typically via averaging) through a synchronisation mechanism such as all-reduce. The synchronised gradients are then applied uniformly to update model weights across all replicas before the next iteration begins.
Why It Matters
Organisations training large-scale models benefit from reduced time-to-convergence, enabling faster experimentation cycles and reduced computational cost per training run. This approach scales nearly linearly with device count for large batch sizes, making it economically viable to train models on datasets that would be prohibitively slow on single-device setups.
Common Applications
Computer vision model training on image classification datasets, natural language processing tasks such as large transformer model pretraining, and recommendation system training on e-commerce platforms routinely employ this strategy to reduce wall-clock training time from weeks to days.
Key Considerations
Communication overhead between devices can become a bottleneck at scale, particularly with slower interconnects or very frequent synchronisation. Effective batch size increases with the number of devices, which may require adjusted learning rates and can affect model convergence behaviour and final accuracy if not compensated appropriately.
Cross-References(1)
More in Deep Learning
Diffusion Model
Generative ModelsA generative model that learns to reverse a gradual noising process, generating high-quality samples from random noise.
Key-Value Cache
ArchitecturesAn optimisation in autoregressive transformer inference that stores previously computed key and value tensors to avoid redundant computation during sequential token generation.
Self-Attention
Training & OptimisationAn attention mechanism where each element in a sequence attends to all other elements to compute its representation.
Mixture of Experts
ArchitecturesAn architecture where different specialised sub-networks (experts) are selectively activated based on the input.
Flash Attention
ArchitecturesAn IO-aware attention algorithm that reduces memory reads and writes by tiling the attention computation, enabling faster training of long-context transformer models.
Model Parallelism
ArchitecturesA distributed training approach that partitions a model across multiple devices, enabling training of models too large to fit in a single accelerator's memory.
Contrastive Learning
ArchitecturesA self-supervised learning approach that trains models by comparing similar and dissimilar pairs of data representations.
Weight Decay
ArchitecturesA regularisation technique that penalises large model weights during training by adding a fraction of the weight magnitude to the loss function, preventing overfitting.