Transformer
The transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need". Nearly every modern language model is built on it, and its central idea is the attention mechanism.
How it works
Attention lets the model look at every other word at once while processing each one, and decide which of them matter. In "the bank refused the loan because it was unprofitable", attention is what ties "it" to the right word.
Before transformers, text was processed sequentially, word by word, and links between distant parts of a sentence were lost. Attention removed that limit and made training parallel at the same time, which is what allowed models to be trained on genuinely large corpora.
The price is that computation grows quadratically with length. Double the context and you quadruple the work, which is why long context is expensive and advances more slowly than one would like.
Why this matters
- It explains the cost of long contextA 100,000-token request does not cost twice a 50,000-token one; it costs noticeably more.
- It explains "lost in the middle"Models hold the beginning and end of a long text noticeably better than the middle. Put what matters at the edges.
- One architecture for everythingText, images, audio and code all run through the same mechanism, which is where multimodal models come from.