Stochastic Gradient Descent — Technology Wiki

Overview

Direct Answer

Stochastic Gradient Descent (SGD) is an optimisation algorithm that updates model parameters using the gradient computed from a single training example or small batch, rather than the entire dataset. This probabilistic approach to parameter adjustment trades some convergence certainty for computational efficiency and faster iteration cycles.

How It Works

At each iteration, SGD samples a single instance or mini-batch randomly from the training set, computes the loss gradient with respect to that sample, and adjusts parameters in the direction opposite to the gradient by a step size called the learning rate. The stochastic nature—randomness in sample selection—introduces noise into the parameter trajectory, which can help escape local minima and reduce memory requirements compared to full-batch methods.

Why It Matters

SGD enables training on datasets too large to fit in memory and reduces wall-clock time per iteration significantly, making it essential for modern deep learning at scale. The noise-induced exploration properties often lead to better generalisation on unseen data, whilst the reduced computational footprint per step allows practitioners to iterate on model design rapidly.

Common Applications

SGD is the foundation for training neural networks across computer vision, natural language processing, and recommendation systems. It underpins backpropagation in deep learning frameworks and remains standard in federated learning environments where data partitioning across devices necessitates sample-wise or batch-wise updates.

Key Considerations

The learning rate becomes critical since constant steps with noisy gradients risk divergence; adaptive variants like Adam and RMSprop address this by adjusting step sizes per parameter. Convergence guarantees weaken compared to batch gradient descent, and practitioners must balance batch size, learning rate scheduling, and epoch count empirically.

Cross-References(1)

Machine Learning

Gradient Descent

Referenced By1 term mentions Stochastic Gradient Descent

Other entries in the wiki whose definition references Stochastic Gradient Descent — useful for understanding how this concept connects across Machine Learning and adjacent domains.

Mini-Batch·Machine Learning

Related in Training Techniques

Ridge Regression

A regularised regression technique that adds an L2 penalty term to prevent overfitting by constraining coefficient magnitudes.

Elastic Net

A regularisation technique combining L1 and L2 penalties, balancing feature selection and coefficient shrinkage.

Cross-Validation

A resampling technique that partitions data into subsets, training on some and validating on others to assess model generalisation.

Overfitting

When a model learns the training data too well, including noise, resulting in poor performance on unseen data.

Underfitting

When a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and test data.

Bias-Variance Tradeoff

The balance between a model's ability to minimise bias (error from assumptions) and variance (sensitivity to training data fluctuations).

Regularisation

Techniques that add constraints or penalties to a model to prevent overfitting and improve generalisation to new data.

Gradient Descent

An optimisation algorithm that iteratively adjusts parameters in the direction of steepest descent of the loss function.

Adam Optimiser

An adaptive learning rate optimisation algorithm combining momentum and RMSProp for efficient deep learning training.

Learning Rate

A hyperparameter that controls how much model parameters are adjusted with respect to the loss gradient during training.

Loss Function

A mathematical function that measures the difference between predicted outputs and actual target values during model training.

Backpropagation

The algorithm for computing gradients of the loss function with respect to network weights, enabling neural network training.

More in Machine Learning

Ensemble Methods

MLOps & Production

Machine learning techniques that combine multiple models to produce better predictive performance than any single model, including bagging, boosting, and stacking approaches.

Collaborative Filtering

Unsupervised Learning

A recommendation technique that makes predictions based on the collective preferences and behaviour of many users.

Model Serving

MLOps & Production

The infrastructure and processes for deploying trained machine learning models to production environments for real-time predictions.

Markov Decision Process

Reinforcement Learning

A mathematical framework for modelling sequential decision-making where outcomes are partly random and partly controlled.

SHAP Values

MLOps & Production

A game-theoretic approach to explaining individual model predictions by computing each feature's marginal contribution, based on Shapley values from cooperative game theory.

Model Serialisation

MLOps & Production

The process of converting a trained model into a format that can be stored, transferred, and later reconstructed for inference.

Unsupervised Learning

MLOps & Production

A machine learning approach where models discover patterns and structures in data without labelled examples.

DBSCAN

Unsupervised Learning

Density-Based Spatial Clustering of Applications with Noise — a clustering algorithm that finds arbitrarily shaped clusters based on density.