Underfitting — Technology Wiki

Overview

Direct Answer

Underfitting occurs when a machine learning model lacks sufficient complexity to learn the underlying patterns and relationships in the training data, resulting in consistently poor predictive performance on both training and test datasets. This typically indicates the model architecture or feature set is inadequate rather than a problem with data quality.

How It Works

An overly simplistic model—such as linear regression applied to non-linear data or a shallow neural network for complex classification tasks—cannot represent the decision boundaries or functional relationships present in the data. The model remains biased away from the true target function, causing high training loss and high test loss simultaneously, with no opportunity to improve through additional training iterations.

Why It Matters

Organisations investing in machine learning initiatives require models that generalise effectively to new data. Underfitting wastes computational resources and delays deployment timelines, whilst producing unreliable predictions that undermine business decisions in domains such as credit risk assessment, demand forecasting, and medical diagnostics.

Common Applications

Underfitting is frequently observed when applying simple baseline models to complex datasets in finance, healthcare analytics, and natural language processing. Examples include using a linear model for image classification, applying polynomial degree-1 regression to non-linear physical phenomena, or deploying shallow decision trees for high-dimensional fraud detection.

Key Considerations

Distinguishing between underfitting and other performance issues requires comparative analysis across model complexity levels and cross-validation strategies. Practitioners must balance model sophistication against interpretability requirements and computational constraints, avoiding the assumption that increased complexity always resolves performance deficits.

Related in Training Techniques

Ridge Regression

A regularised regression technique that adds an L2 penalty term to prevent overfitting by constraining coefficient magnitudes.

Elastic Net

A regularisation technique combining L1 and L2 penalties, balancing feature selection and coefficient shrinkage.

Cross-Validation

A resampling technique that partitions data into subsets, training on some and validating on others to assess model generalisation.

Overfitting

When a model learns the training data too well, including noise, resulting in poor performance on unseen data.

Bias-Variance Tradeoff

The balance between a model's ability to minimise bias (error from assumptions) and variance (sensitivity to training data fluctuations).

Regularisation

Techniques that add constraints or penalties to a model to prevent overfitting and improve generalisation to new data.

Gradient Descent

An optimisation algorithm that iteratively adjusts parameters in the direction of steepest descent of the loss function.

Stochastic Gradient Descent

A variant of gradient descent that updates parameters using a randomly selected subset of training data each iteration.

Adam Optimiser

An adaptive learning rate optimisation algorithm combining momentum and RMSProp for efficient deep learning training.

Learning Rate

A hyperparameter that controls how much model parameters are adjusted with respect to the loss gradient during training.

Loss Function

A mathematical function that measures the difference between predicted outputs and actual target values during model training.

Backpropagation

The algorithm for computing gradients of the loss function with respect to network weights, enabling neural network training.

More in Machine Learning

Meta-Learning

Advanced Methods

Learning to learn — algorithms that improve their learning process by leveraging experience from multiple learning episodes.

UMAP

Unsupervised Learning

Uniform Manifold Approximation and Projection — a dimensionality reduction technique for visualisation and general non-linear reduction.

Self-Supervised Learning

Advanced Methods

A learning paradigm where models generate their own supervisory signals from unlabelled data through pretext tasks.

Naive Bayes

Supervised Learning

A probabilistic classifier based on applying Bayes' theorem with the assumption of independence between features.

Lasso Regression

Feature Engineering & Selection

A regularised regression technique that adds an L1 penalty, enabling feature selection by driving some coefficients to zero.

Collaborative Filtering

Unsupervised Learning

A recommendation technique that makes predictions based on the collective preferences and behaviour of many users.

Association Rule Learning

Unsupervised Learning

A method for discovering interesting relationships and patterns between variables in large datasets.

Tabular Deep Learning

Supervised Learning

The application of deep neural networks to structured tabular datasets, competing with traditional methods like gradient boosting through specialised architectures and regularisation.