Chapter 09 · Sustainability & AISustainability & AI

Model distillation

Meaning statusEstablishedSource recordDirect document linkedWhy these are different

Definition

Training a compact 'student' model to reproduce the behaviour of a larger 'teacher' model, preserving much of its performance with far fewer parameters.

References

arXivHinton et al. — Distilling the Knowledge in a Neural Network (2015)

This source provides part of the technical or institutional basis for the definition.

Communications of the ACM 2020Schwartz et al. — Green AI

This source supports the explanation of how the term is applied, measured or governed in practice.

Overview

What it means

Named by Hinton and colleagues in 2015, distillation transfers the soft, probabilistic outputs of a large model into a small one. It is now a standard efficiency technique, alongside quantisation and pruning, for deploying capable models on limited hardware.

How it is used

Distilled models run on modest infrastructure or at the edge — enabling on-device environmental sensors, cost-effective document processing, and lower-energy inference at scale.

Why it matters

Distillation is a practical lever of frugal AI: it decouples the capability of large models from their operating footprint, and its existence undercuts the assumption that useful AI must be maximal AI.