Loss Function — Technology Wiki

Overview

Direct Answer

A loss function is a mathematical formula that quantifies the disparity between a model's predicted values and ground-truth target values, serving as the objective that optimisation algorithms minimise during training. It transforms prediction errors into a scalar cost metric that guides iterative parameter adjustment.

How It Works

During each training iteration, the function computes error magnitude across a batch of samples, aggregating individual prediction discrepancies into a single scalar value. Optimisation algorithms (such as gradient descent) calculate the gradient of this scalar with respect to model parameters, then adjust weights in directions that reduce the loss value. The choice of formula—whether mean squared error, cross-entropy, or other variants—directly influences which types of errors the model penalises most heavily.

Why It Matters

Selecting an appropriate loss function fundamentally determines model behaviour, convergence speed, and final accuracy. Misaligned choices lead to suboptimal training, poor generalisation, or failure to capture business objectives (e.g., prioritising precision over recall in fraud detection). In regulated industries, the loss function can encode compliance requirements directly into the training objective.

Common Applications

Regression tasks employ mean squared error to penalise prediction magnitude errors. Classification systems use cross-entropy to discourage incorrect probability assignments. Imbalanced datasets benefit from weighted variants that increase penalty for minority classes. Recommendation systems and natural language processing models rely on task-specific formulations to optimise ranking or sequence generation quality.

Key Considerations

The loss function must remain differentiable for gradient-based optimisation, and its scale relative to data distributions significantly affects learning dynamics. No single formula universally suits all problems; practitioners must align the mathematical formulation with downstream business metrics and model behaviour requirements.

Referenced By3 terms mention Loss Function

Other entries in the wiki whose definition references Loss Function — useful for understanding how this concept connects across Machine Learning and adjacent domains.

Backpropagation·Machine Learning Gradient Descent·Machine Learning Weight Decay·Deep Learning

Related in Training Techniques

Ridge Regression

A regularised regression technique that adds an L2 penalty term to prevent overfitting by constraining coefficient magnitudes.

Elastic Net

A regularisation technique combining L1 and L2 penalties, balancing feature selection and coefficient shrinkage.

Cross-Validation

A resampling technique that partitions data into subsets, training on some and validating on others to assess model generalisation.

Overfitting

When a model learns the training data too well, including noise, resulting in poor performance on unseen data.

Underfitting

When a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and test data.

Bias-Variance Tradeoff

The balance between a model's ability to minimise bias (error from assumptions) and variance (sensitivity to training data fluctuations).

Regularisation

Techniques that add constraints or penalties to a model to prevent overfitting and improve generalisation to new data.

Gradient Descent

An optimisation algorithm that iteratively adjusts parameters in the direction of steepest descent of the loss function.

Stochastic Gradient Descent

A variant of gradient descent that updates parameters using a randomly selected subset of training data each iteration.

Adam Optimiser

An adaptive learning rate optimisation algorithm combining momentum and RMSProp for efficient deep learning training.

Learning Rate

A hyperparameter that controls how much model parameters are adjusted with respect to the loss gradient during training.

Backpropagation

The algorithm for computing gradients of the loss function with respect to network weights, enabling neural network training.

More in Machine Learning

SMOTE

Feature Engineering & Selection

Synthetic Minority Over-sampling Technique — a method for addressing class imbalance by generating synthetic examples of the minority class.

Feature Engineering

Feature Engineering & Selection

The process of using domain knowledge to create, select, and transform input variables to improve model performance.

Polynomial Regression

Supervised Learning

A form of regression analysis where the relationship between variables is modelled as an nth degree polynomial.

Ensemble Learning

MLOps & Production

Combining multiple machine learning models to produce better predictive performance than any single model.

Batch Learning

MLOps & Production

Training a machine learning model on the entire dataset at once before deployment, as opposed to incremental updates.

Boosting

Supervised Learning

An ensemble technique that sequentially trains models, each focusing on correcting the errors of previous models.

Multi-Task Learning

MLOps & Production

A machine learning approach where a model is simultaneously trained on multiple related tasks to improve generalisation.

Deep Reinforcement Learning

Reinforcement Learning

Combining deep neural networks with reinforcement learning to enable agents to learn complex decision-making from raw sensory input.