Overview
Direct Answer
Bagging (Bootstrap Aggregating) is an ensemble method that reduces variance by training multiple models independently on random subsets of the training data drawn with replacement, then combining their predictions through averaging or voting. This approach is particularly effective for high-variance algorithms such as decision trees.
How It Works
The method generates B bootstrap samples by randomly sampling the original dataset with replacement, each typically the same size as the original. A base learner trains independently on each sample, producing B distinct models. For regression tasks, predictions are averaged across all models; for classification, a majority vote determines the final output. This independence between training iterations enables parallel computation.
Why It Matters
Bagging significantly improves model stability and generalisation performance, reducing overfitting without requiring algorithmic modification. Teams benefit from lower prediction variance and more reliable confidence intervals, which is critical for high-stakes decisions in finance, healthcare, and risk management where model robustness directly impacts business outcomes.
Common Applications
Random forests, a bagged ensemble of decision trees, are widely deployed in credit risk assessment, medical diagnosis support systems, and feature importance analysis. The technique also improves neural network robustness and is applied in manufacturing defect detection and customer churn prediction.
Key Considerations
Bagging reduces variance but provides minimal bias reduction; it works best with unstable learners prone to overfitting. Computational cost scales linearly with the number of models trained, and gains diminish as base learner stability increases, requiring practitioners to balance accuracy improvements against training overhead.
Referenced By1 term mentions Bagging
Other entries in the wiki whose definition references Bagging — useful for understanding how this concept connects across Machine Learning and adjacent domains.
More in Machine Learning
Matrix Factorisation
Unsupervised LearningA technique that decomposes a matrix into constituent matrices, widely used in recommendation systems and dimensionality reduction.
Elastic Net
Training TechniquesA regularisation technique combining L1 and L2 penalties, balancing feature selection and coefficient shrinkage.
Batch Learning
MLOps & ProductionTraining a machine learning model on the entire dataset at once before deployment, as opposed to incremental updates.
Online Learning
MLOps & ProductionA machine learning method where models are incrementally updated as new data arrives, rather than being trained in batch.
Experiment Tracking
MLOps & ProductionThe systematic recording of machine learning experiment parameters, metrics, artifacts, and code versions to enable reproducibility and comparison across training runs.
Clustering
Unsupervised LearningUnsupervised learning technique that groups similar data points together based on inherent patterns without predefined labels.
Hierarchical Clustering
Unsupervised LearningA clustering method that builds a tree-like hierarchy of clusters through successive merging or splitting of groups.
A/B Testing
Training TechniquesA controlled experiment comparing two variants to determine which performs better against a defined metric.