Transfer Learning — Technology Wiki

Overview

Direct Answer

Transfer learning is a machine learning methodology in which a model trained on a source domain or task is repurposed and fine-tuned for application to a target domain or task. This approach leverages learned representations—weights, feature hierarchies, and patterns—rather than training from random initialisation.

How It Works

A pre-trained model, typically trained on large-scale datasets, serves as a feature extractor. During the adaptation phase, either the final layers are retrained on new data whilst keeping earlier layers frozen, or all parameters undergo fine-tuning with a lower learning rate. This process preserves foundational knowledge whilst specialising the model to new target characteristics.

Why It Matters

Organisations deploy this technique to reduce computational cost, accelerate model development cycles, and achieve higher accuracy with limited labelled data. In resource-constrained settings, leveraging pre-trained models substantially lowers training time and infrastructure requirements whilst improving convergence speed and generalisation performance.

Common Applications

Computer vision applications utilise ImageNet-pretrained models for medical image analysis, object detection, and satellite imagery classification. Natural language processing commonly applies models trained on large corpora to domain-specific tasks including sentiment analysis, named entity recognition, and document classification.

Key Considerations

Domain mismatch between source and target can degrade performance if the learned representations are insufficiently similar. Practitioners must balance the trade-off between preserving pre-trained weights and adapting to target-specific characteristics, requiring careful selection of which layers to freeze and adjustment of hyperparameters.

Related in Advanced Methods

Semi-Supervised Learning

A learning approach that combines a small amount of labelled data with a large amount of unlabelled data during training.

Self-Supervised Learning

A learning paradigm where models generate their own supervisory signals from unlabelled data through pretext tasks.

Meta-Learning

Learning to learn — algorithms that improve their learning process by leveraging experience from multiple learning episodes.

Curriculum Learning

A training strategy that presents examples to a model in a meaningful order, typically from easy to hard.

Bagging

Bootstrap Aggregating — an ensemble method that trains multiple models on random subsets of data and averages their predictions.

Bandit Algorithm

An online learning algorithm that balances exploration of new options with exploitation of known good options to maximise reward.

More in Machine Learning

Naive Bayes

Supervised Learning

A probabilistic classifier based on applying Bayes' theorem with the assumption of independence between features.

Batch Learning

MLOps & Production

Training a machine learning model on the entire dataset at once before deployment, as opposed to incremental updates.

SMOTE

Feature Engineering & Selection

Synthetic Minority Over-sampling Technique — a method for addressing class imbalance by generating synthetic examples of the minority class.

Support Vector Machine

Supervised Learning

A supervised learning algorithm that finds the optimal hyperplane to separate different classes in high-dimensional space.

Clustering

Unsupervised Learning

Unsupervised learning technique that groups similar data points together based on inherent patterns without predefined labels.

Feature Selection

MLOps & Production

The process of identifying and selecting the most relevant input variables for a machine learning model.

Markov Decision Process

Reinforcement Learning

A mathematical framework for modelling sequential decision-making where outcomes are partly random and partly controlled.

Model Calibration

MLOps & Production

The process of adjusting a model's predicted probabilities so they accurately reflect the true likelihood of outcomes, essential for risk-sensitive decision-making.