Overview
Direct Answer
Model distillation is a compression technique in which a smaller, more efficient model—called the student—learns to approximate the predictions and internal representations of a larger, pre-trained model—called the teacher. The student model achieves comparable performance whilst requiring substantially fewer parameters and computational resources.
How It Works
The teacher model generates soft probability distributions (logits) over training data, which encode richer decision boundaries than hard labels alone. The student is trained on a combined loss function that minimises divergence from the teacher's soft outputs whilst maintaining accuracy on the original task. Temperature scaling adjusts the softness of these distributions, controlling the level of knowledge transfer and enabling the student to learn generalised patterns rather than memorising specific examples.
Why It Matters
Distillation enables deployment of high-performance models on resource-constrained devices, reducing latency, energy consumption, and infrastructure costs in production environments. This is critical for real-time applications such as mobile inference, edge computing, and large-scale serving where computational efficiency directly impacts operational expenses and user experience.
Common Applications
Applications include compressing language models for on-device natural language processing, accelerating computer vision models in autonomous systems, and optimising recommendation engines in e-commerce platforms. Financial institutions use distillation to deploy fraud detection models with lower latency, whilst healthcare organisations compress diagnostic models for integration into clinical decision-support systems.
Key Considerations
Knowledge transfer is not guaranteed; student models may fail to capture all nuances of teacher behaviour, particularly on out-of-distribution data. Determining optimal student architecture, temperature hyperparameters, and loss weighting between task accuracy and teacher mimicry requires substantial experimentation and validation.
More in Artificial Intelligence
Backward Chaining
Reasoning & PlanningAn inference strategy that starts with a goal and works backward through rules to determine what facts must be true.
AUC Score
Evaluation & MetricsArea Under the ROC Curve, a single metric summarising a classifier's ability to distinguish between classes.
Retrieval-Augmented Generation
Infrastructure & OperationsA technique combining information retrieval with text generation, allowing AI to access external knowledge before generating responses.
In-Context Learning
Prompting & InteractionThe ability of large language models to learn new tasks from examples provided within the input prompt without parameter updates.
Knowledge Representation
Foundations & TheoryThe field of AI dedicated to representing information about the world in a form that computer systems can use for reasoning.
Forward Chaining
Reasoning & PlanningAn inference strategy that starts with known facts and applies rules to derive new conclusions until a goal is reached.
AI Inference
Training & InferenceThe process of using a trained AI model to make predictions or decisions on new, unseen data.
AI Fairness
Safety & GovernanceThe principle of ensuring AI systems make equitable decisions without discriminating against any group based on protected attributes.