Overview
Direct Answer
Gradient boosting is an ensemble machine learning method that constructs models sequentially, with each successive model trained to predict and correct the residual errors left by the combined predictions of all previous models. The technique uses gradient descent optimisation to minimise a loss function across iterations.
How It Works
The algorithm initialises with a base learner, then iteratively fits new weak learners (typically decision trees) to the negative gradient of the loss function computed on training data. Each new model is weighted and added to the ensemble, with subsequent models focusing on instances or residuals where the existing ensemble performs poorly. Learning rates control the contribution of each addition, balancing model complexity against convergence speed.
Why It Matters
Gradient boosting achieves state-of-the-art predictive accuracy across classification and regression tasks, often outperforming alternative ensemble methods on tabular data. Organisations deploy it for high-stakes applications requiring robust generalisation, including credit risk assessment, fraud detection, and customer churn prediction, where incremental accuracy improvements directly translate to measurable business value.
Common Applications
Applications span financial services for loan default prediction, e-commerce for demand forecasting, healthcare for patient outcome modelling, and insurance for claims assessment. XGBoost, LightGBM, and CatBoost represent widely adopted open-source implementations used across industries for competition benchmarks and production systems.
Key Considerations
The sequential training process is computationally expensive and harder to parallelise than batch ensemble methods. Practitioners must carefully tune hyperparameters including learning rate, tree depth, and iteration count to avoid overfitting whilst maintaining interpretability.
Referenced By2 terms mention Gradient Boosting
Other entries in the wiki whose definition references Gradient Boosting — useful for understanding how this concept connects across Machine Learning and adjacent domains.
More in Machine Learning
Bagging
Advanced MethodsBootstrap Aggregating — an ensemble method that trains multiple models on random subsets of data and averages their predictions.
Markov Decision Process
Reinforcement LearningA mathematical framework for modelling sequential decision-making where outcomes are partly random and partly controlled.
Regularisation
Training TechniquesTechniques that add constraints or penalties to a model to prevent overfitting and improve generalisation to new data.
Reinforcement Learning
MLOps & ProductionA machine learning paradigm where agents learn optimal behaviour through trial and error, receiving rewards or penalties.
Batch Learning
MLOps & ProductionTraining a machine learning model on the entire dataset at once before deployment, as opposed to incremental updates.
Online Learning
MLOps & ProductionA machine learning method where models are incrementally updated as new data arrives, rather than being trained in batch.
UMAP
Unsupervised LearningUniform Manifold Approximation and Projection — a dimensionality reduction technique for visualisation and general non-linear reduction.
Transfer Learning
Advanced MethodsA technique where knowledge gained from training on one task is applied to a different but related task.