Chapter 09 · Sustainability & AISustainability & AI
Transformer model
Definition
A neural-network architecture built on attention mechanisms that weigh the relevance of different parts of the input, enabling effective learning from large sequential datasets.
References
This source provides part of the technical or institutional basis for the definition.
This source supports the explanation of how the term is applied, measured or governed in practice.
This source supports the wider sustainability significance and context described in the entry.
Overview
What it means
Introduced in the 2017 paper 'Attention Is All You Need', the transformer displaced recurrent architectures and became the basis of modern language models, and increasingly of vision and scientific models. Its scalability drives both capability growth and training energy demand.
How it is used
Transformers power the generative AI tools now used in sustainability analysis and reporting, and also scientific models such as weather forecasters — ECMWF's AIFS and Google's GraphCast both build on transformer-style designs.
Why it matters
One architecture now underlies most frontier AI, from chatbots to climate models. Understanding its attention mechanism is less important than grasping the consequence: capability concentrated in very large models trained by very few organisations.