Overview
Direct Answer
Tensor parallelism is a distributed training strategy that partitions individual weight matrices and activation tensors across multiple devices along specific dimensions, enabling computation of a single model layer to occur in parallel. Unlike data parallelism, which replicates the entire model, this approach reduces memory footprint per device by distributing the mathematical operations of matrix multiplications themselves.
How It Works
During forward and backward propagation, weight matrices are split column-wise or row-wise across devices. Each device computes a partial result on its assigned tensor slice, then results are aggregated through collective operations (e.g. all-reduce). Communication overlaps with computation where feasible, minimising synchronisation overhead. The granularity and axis of partitioning depend on the layer type and target batch size.
Why It Matters
This approach enables training of exceptionally large models that would exceed single-device memory constraints, directly impacting capability and cost-efficiency in large language model and vision transformer development. Organisations prioritise it when model scale exceeds practical limits of other parallelism strategies, particularly when batch sizes cannot be increased freely.
Common Applications
Tensor parallelism is widely deployed in training large transformer-based language models and multimodal systems where model dimension is the primary scaling factor. It is frequently combined with pipeline and data parallelism in systems handling billions of parameters.
Key Considerations
Communication bandwidth between devices becomes a critical bottleneck; synchronous all-reduce operations can introduce substantial latency on slower interconnects. The strategy is most effective on high-bandwidth clusters and less suitable for models with small embedding or hidden dimensions relative to device count.
Cross-References(1)
More in Deep Learning
LoRA
Language ModelsLow-Rank Adaptation — a parameter-efficient fine-tuning technique that adds trainable low-rank matrices to frozen pretrained weights.
Mixed Precision Training
Training & OptimisationTraining neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.
Mixture of Experts
ArchitecturesAn architecture where different specialised sub-networks (experts) are selectively activated based on the input.
State Space Model
ArchitecturesA sequence modelling architecture based on continuous-time dynamical systems that processes long sequences with linear complexity, offering an alternative to attention-based transformers.
Pretraining
ArchitecturesTraining a model on a large general dataset before fine-tuning it on a specific downstream task.
Residual Connection
Training & OptimisationA skip connection that adds a layer's input directly to its output, enabling gradient flow through deep networks and allowing training of architectures with hundreds of layers.
Embedding
ArchitecturesA learned dense vector representation of discrete data (like words or categories) in a continuous vector space.
Weight Decay
ArchitecturesA regularisation technique that penalises large model weights during training by adding a fraction of the weight magnitude to the loss function, preventing overfitting.