Attention Is All You Need (If You Can Afford It)

🌍 English  ·  Leggi in italiano →

It all starts with a somewhat stubborn idea: getting a small language model — one that generates text, like its giant siblings — to run on tiny hardware. Little memory. No GPU, no cloud, no racks.

When memory is counted in megabytes (yes, megabytes), you can't afford to use things without understanding them. Every architectural choice becomes a concrete question: what does this cost me, in bytes? And in answering those questions, concepts that seem abstract — attention, context window, parameters, Mixture of Experts — suddenly become physical objects, with a weight and a place to live.

About the project itself I won't say much here: I'm still wrestling with it, and I'll report back when there's something worth telling. What follows is the step before that — the analogies I had to build in order to face it, useful for pinning down concepts I already had in my head, dismantling wrong ones I'd built for myself, and picking back up a field I had mostly abandoned after a few experiments between 2015 and 2020.

No mathematical formulas (well, almost): let's start with a scribe, some notebooks, and a library.

The Building and the Scribe

A language model that generates text is a scribe who writes one word at a time: he looks at everything written so far and proposes the next word. Then he starts again.

He is a scribe at work: his skills are already formed — he learned them at school, during training — and while writing he learns nothing new. Keep that in mind, because the distinction will matter later.

Inside, the model (the scribe) is built like a building of floors (the layers): text enters at the ground floor and climbs, getting reworked a little at every floor. Every floor has to do the same fundamental thing: look at the words already written to grasp the context. The whole difference between architectures lies in how each floor does it. There are two main methods, and we'll take them one at a time.

The Notebook

The first method is attention, the mechanism that made Transformers famous. In the analogy: a notebook.

The floor takes notes on every word that passes and keeps them. When a new word arrives, it goes back and rereads the pages, deciding which ones matter most right now: "what was the robot's name again? Ah, there it is, three pages back". It's extremely precise at recalling details, because every word has its own page of notes, never rewritten and never compressed.

But it has a structural cost: the more words you keep, the more pages you need. The written pages, in technical jargon, are called the KV cache — and their sheer bulk is the true price of attention. The notebook grows with every word, forever.

The Running Summary

The second method is recurrence: listening while keeping only a summary in mind, continuously updated. With every new word, you blend what you knew with what just arrived — some of it enters the summary, some of it fades out — and you move on. You don't remember each exact word: you remember the accumulated meaning.

The upside is enormous: the summary takes up the same space in your head, whether the speech lasts a minute or an hour. For hardware with memory in short supply, that's gold. The downside is the mirror image: on precise details ("copy exactly that name that appeared eighty words ago") a summary is blurrier than a written page.

There's an objection that comes naturally to anyone who knows the history: the summary, described like this, is the old RNN mechanism — abandoned years ago precisely because it doesn't parallelize. The flaw is in the mechanism, not in one of its phases: every word has to wait for the previous one to finish, so any text becomes a single-file line of steps, and GPUs — machines built to do a thousand things at once — just stand there watching.

You pay that bill twice. When the scribe has to digest text he already has in front of him — your request, a document — a thousand words are a thousand steps, and you wait. But the killing blow lands at school, that is, in training: the months in which the model learns by reading mountains of text before ever writing a single word for you. There, the words aren't a thousand: they're billions. In single file it isn't slow — it's unworkable.

What makes the summary's comeback possible is one detail: who directs traffic at the entrance. Picture two gatekeepers posted at the door of the summary-warehouse. The first looks at the incoming word and decides how much of it to let in: "proper noun — all of it"; "comma — barely anything". The second, looking at the same word, decides how much to fade what's already inside. These aren't hand-written rules: the two gatekeepers are parameters, learned at school like the rest of the scribe's brain, and frozen from then on.

The whole difference from the old RNN is where the gatekeeper stands. He used to stand inside the warehouse: to decide how much to let in, he first had to look at what was on the shelves. A tank whose valve is controlled by the water it holds — and then the line at the entrance can't flow; you go one at a time, no way around it.

Now the gatekeepers stand at the door and never look inside. They only look at who's arriving. And so, when you already have the whole text in front of you, you can put a thousand gatekeepers on it in parallel: each reads its own word and jots down two numbers — how much gets in, and how much fades. No one waits for anyone.

Once that's done, the summaries no longer need to be replayed: they can be computed. Because there isn't just one summary — there's one for every point in the text, since at every point the scribe must be able to propose the next word. And a word's contribution to any given point is simply how much of it got in, multiplied by all the fadings that rolled over it in the meantime. Products and sums, and products can be grouped however you like: do half the text and half the text, then combine; then quarters, then eighths. A thousand summaries close in about ten tree-shaped steps instead of a thousand in single file — with the exact same result you'd have gotten going in order.

Don't take my word for it: let's check with numbers, in miniature. Four words; for each one the gatekeepers have already jotted down the two numbers — how much gets in (say 6, 4, 2, 8) and how much what was already there fades (½ for all of them, to keep life simple). In single file: 6; then 6·½ + 4 = 7; then 7·½ + 2 = 5.5; then 5.5·½ + 8 = 10.75. Three steps, each one standing around waiting for the previous.

As a tree, instead: two crews start at the same instant. The first works words 1–2 and hands over two numbers: "my chunk is worth 7, and anything that came before me I'd fade by ½·½ = ¼". The second, in parallel, works words 3–4 as if the text began there, and hands over: "worth 2·½ + 8 = 9, fade ¼". One stitching step remains: 7·¼ + 9 = 10.75. Not similar: identical, down to the decimal — because products and sums give the same result however you group them. On four words you've saved one step, big deal; but on a thousand, a line of a thousand steps closes in about ten levels.

And here's the part that surprises: this isn't just a school matter. The real dividing line isn't between training and use, but between the text the scribe already has in front of him and the text he's still producing. When you hand him your request, it's already all there: it enters the warehouse in parallel, in a flash, a thousand gatekeepers at once. Then the scribe starts writing, and every word he writes re-enters the warehouse in exactly the same way — same two gatekeepers, same rules — but now one at a time, because the next word doesn't exist yet.

That's why an enormous context gets digested almost instantly while the reply keeps you waiting: not two speeds — two regimes. And it holds for the notebook exactly as for the summary: Transformers too digest the text in front of them all in parallel, then write one word at a time. And the second regime, the word-by-word one, is precisely the one a small device needs: one update at a time, memory always the same.

One last thing, which will come in handy: every word enters at full weight and starts fading from there. The summary always knows very well what happened a moment ago, and ever more vaguely what happened at the beginning.

The Recipe for the Floors

So why choose between Summaries and Notebooks? The answer that works is a mix, and it can be written like a recipe: one letter per floor. For example, in an eight-floor building:

S S A S S A S S

where S is a summary floor and A a notebook (attention) floor. Six floors out of eight run on summaries — nearly free, memory-wise — and two keep the notebook, placed some distance apart, so that when a precise detail needs fishing out, someone has it written down.

graph BT
  IN(["Incoming text"]) --> P1["Floor 1 · S"]
  P1 --> P2["Floor 2 · S"]
  P2 --> P3["Floor 3 · A"]
  P3 --> P4["Floor 4 · S"]
  P4 --> P5["Floor 5 · S"]
  P5 --> P6["Floor 6 · A"]
  P6 --> P7["Floor 7 · S"]
  P7 --> P8["Floor 8 · S"]
  P8 --> OUT(["Next word"])
  classDef acc fill:#ffe8cc,stroke:#e8892b,color:#111;
  classDef rec fill:#e7f0ff,stroke:#3b6fb0,color:#111;
  class P3,P6 acc
  class P1,P2,P4,P5,P7,P8 rec

Text enters at the ground floor and climbs. In orange, the two notebook floors (A); in blue, the six summary floors (S): it's the Griffin pattern — "two recurrent, one attention" — repeated.

This is no hobbyist's notion: DeepMind showed with Griffin (and the open models RecurrentGemma) that this alternation, in almost identical proportions, holds its own against pure Transformers while using less memory and generating faster. The same lever works at billions of parameters on TPUs and at millions of parameters on a chip that costs a few euros: the only thing that changes is who pays the memory bill.

Notebooks on a Diet: Fewer Pages, One Single Notebook

The two surviving notebooks (if needed) can be put on a diet too, with two tricks.

The sliding window: the notebook has a fixed number of pages — say 256. When you write page 257, you tear out the oldest. You don't need to remember the whole book: for syntax and the thread of the discourse, "recent" is enough.

One notebook for all the readers (technically multi-query attention): so far I've described a floor's notebook as if a single hand held it. In reality every floor has several readers — say eight — and each one rereads the same pages with a different question in mind ("who is the subject?", "what tense are we in?"). By default each keeps his own notebook: eight different notebooks, because each reader writes down, about the same words, only what serves his own question. The trick is simple and immensely powerful — one single, shared notebook: everyone writes and rereads the same pages, each with his own question in mind. You lose a bit of finesse — eight points of view forced to share the same notes — but in practice it's a price gladly paid. The saving doesn't come from tighter handwriting, but from the number of notebooks: from eight down to one.

And it's a spectrum, not a switch: halfway along there are those who keep a few notebooks, each shared by a group of readers (that's GQA), and those who genuinely compress the pages into a condensed form (DeepSeek's MLA). But that's a different diet.

A notebook with a fixed page count, and a single one for all readers: from the main line item down to pocket change.

The Ring-Bound Notebook

"You tear out the oldest page" is a nice image, but think about it like a librarian: tearing out the first page and shifting all the others down a slot is real drudgery — and in a computer, "shifting" means physically moving memory at every word. A waste.

The trick actually used to implement it is called a circular buffer, and in our analogy it's a notebook bound in a ring: 256 fixed pages and a bookmark. Nothing ever gets torn out. When the bookmark reaches the last page, it loops back to the first and writes over the oldest note — which was exactly the one due to be forgotten anyway. The pages never move; only the bookmark does. Word number 300 always lands on page 300 divided by 256, remainder 44: one operation, always the same.

There's a small price to pay: the pages are no longer in writing order, so whoever rereads them has to know that the most recent one is the page with the bookmark, and the oldest is the one right after it. In exchange, the notebook's memory is fixed forever: you set it up once, at power-on, and it never grows by a single byte — which is exactly what you need when every byte counts.

The Context Window Is the Scribe's Desk

Let's keep adding pieces to our analogy: when we read "a one-million-token context window", we're talking about the size of the scribe's desk — how much text the model can keep in front of it and reread when needed.

And you can see why it costs so much. In the big Transformers the notebook isn't just one: every floor has its own, and no one ever tears out a page. A million words, across dozens of floors, with notes on every word: hundreds of gigabytes of memory just for the notes, before any thinking even happens. That's why long contexts live in data centers: the problem isn't just reasoning — it's keeping the notebook open.

The Notebook Relay

A fair objection: if each notebook (in our minimal model) only rereads the last 256 words, is the model blind beyond that distance? No, for two reasons.

First: the summary floors have no limit — the summary carries everything from the beginning (faded, perhaps), by construction.

The second is subtler: the notebooks pass the baton. The floors are stacked, so each floor doesn't read the original words but the work already processed by the floors below. When the sixth floor's notebook rereads a word, that word already contains what the third floor's notebook had fished out for it — which in turn was looking further back. Same positions on the desk, ever more refined notes as you climb: lined up, the notebooks carry information far beyond a single window.

The Parameters Are the Scribe's Brain

So far we've only talked about working memory. But when people say a model "has 70 billion parameters", they mean something else: the scribe's brain.

The parameters are the numbers that decide how the scribe does everything: what deserves to enter the summary and what to forget, which notebook pages to reread, which word to propose next. They're learned once, during training — the scribe's school, millions of stories and documents read — and then frozen. The notebook and the summary change with every text and get thrown away at the end; the brain stays.

Three memories, never to be confused:

"70 billion parameters" and "a 1-million context window" are orthogonal axes: how learned the scribe is, and how big his desk is. You can have a monster of erudition at a tiny side table, or a billiard-table desk in front of a barely read scribe.

The Scribe's One Move: the Matvec

If the parameters are the scribe's brain, the matvec is the move he makes to use them — the single gesture he repeats, thousands of times, for every single word.

Matvec stands for matrix times vector. At any instant, the scribe's current thought is a vector: a few hundred numbers representing the word being worked on, in its context. The parameters, on the other hand, are organized into matrices: grids of numbers where every row is a question learned at school — "how much proper noun is there, here? how much goes into the summary? how much should be forgotten?". The matvec puts every question to the thought: it takes a row, compares it number by number with the vector, adds it all up, and gets an answer. Hundreds of rows, hundreds of answers — which become the thought handed to the floor above.

Everything we've met so far — rereading the notebook, updating the summary, picking the manual, proposing the word — under the hood is this move, repeated. In language models, huge or tiny, the work is the matvec; the rest is garnish.

And there's a subtlety that becomes decisive on small hardware. At school (training), thousands of words are processed together: matrix times matrix, the regime GPUs adore, where every weight loaded gets reused a thousand times. But the scribe who's generating works one word at a time: matrix times vector. Each weight is fetched from memory, used once, thrown away. The bottleneck stops being "doing the math" and becomes getting the weights to the math. Even under the microscope, the question is always the same: how much does it cost to move memories?

The Library of Manuals

Last piece: the Mixture of Experts (MoE), which in the analogy becomes a library.

The idea: not all of the scribe's culture has to live in his head. A large share of the parameters can live in a library of specialist manuals — the experts. At every word, a lightning-fast librarian (the router) looks at the current thought — not the bare word, but the word already worked over by the floors below — and decides: "this looks like manual 17 material". The scribe consults that manual, that one only, and writes. (To be precise, there's one library per floor, each with its own librarian and its own manuals; the principle is identical, so here we tell the story of just one.)

The result is a mighty accounting trick: the model owns a lot of parameters, but uses few of them for any single word. Enormous culture, tiny effort per word.

How many manuals does the scribe consult per word? It can be just one — the most aggressive choice, the one made by Switch-style models: maximum savings, but the librarian has to get it right on the first try. Or two or four, blended: more margin, more pages to read. One is the minimum; the rest is a trade-off between savings and safety.

There's a snag, though, that shows up at school, and it's the real problem with MoE: the lazy librarian. If nothing forces him otherwise, the router quickly discovers that always recommending manual 17 works passably well for almost everything — and so that manual becomes immensely learned while the others sit empty: culture paid for and never used. It's called router collapse, and the cure is one extra rule added to school: "spread the questions around" — a reward for keeping the manuals balanced (load balancing). Without it, you've bought a thousand manuals only to keep reading one.

The Scribe's Loaded Dice

I know — we said we'd reached the last piece. But the portrait is still missing something: how does the scribe, from the very same beginning, write a slightly different story every time?

The secret is that the scribe never proposes a word: he proposes a ranking of bets. At every step, his work produces a probability for every word in the dictionary: "dragon 18%, princess 12%, forest 9%…" all the way down to absurd words with microscopic probabilities.

If you always took the top-ranked word, the same beginning would always give the same story — a flat and repetitive one at that. Instead, you draw lots: a wheel of fortune with wedges proportional to the bets. "Dragon" comes up often, "forest" now and then, and every spin nudges the story onto a slightly different track — but almost always a sensible one, because the wheel stays rigged in favor of plausible words.

Two knobs govern the wheel. The first (top-k) removes every wedge from the wheel except the best ones — say the top 40 — so the long tail of absurd words can't come up even by bad luck. The second is the boldness knob (temperature): turned down, it exaggerates the gaps and the favorite almost always wins — cautious, predictable stories; turned up, it flattens the wedges — more surprise, more risk of veering off the road.

And one detail for the engineers: a computer's dice are pseudo-random. Fix the "seed" of the randomness and the sequence of rolls repeats identically — the same story, reproducible on demand. Useless for poetry, priceless for testing: you can demand that two different implementations of the same scribe draw exactly the same word.

The "THE END" Stamp

We're at the end and — speaking of which — a question: how does the scribe stop when the story is over? No timer, no hand-written rule: he learned it at school, like everything else.

The trick lies in how the study texts were prepared: every story was closed with a special word — a THE END stamp, an invented word that exists in no human dictionary, added to the vocabulary on purpose. After hundreds of thousands of stories ending with "…and they lived happily ever after. ⟨THE END⟩", the scribe has internalized the most natural bet in the world: after the closing line of a fairy tale, the most probable word is the stamp.

At generation time it happens on its own: when the wheel of fortune draws the stamp, the scribe puts down the pen, and the stamp is removed from the text before it's handed to you. Notice the beauty of it: the decision to stop is itself a prediction, subject to dice and boldness knob like every other word — which is why the story sometimes wraps up in sixty words and sometimes takes its sweet time.

And since scribes do ramble now and then, there's a safety net: an editor who, past a certain number of words, takes the pen away regardless — stamp or no stamp.

The Scale of the Map

The scribe's portrait is complete. But the problem from the beginning remains: fitting him into a few megabytes. Because a brain fresh out of school is heavy: every parameter is a number with lots of decimals, and takes up four bytes (two, in more recent models — still too many). Millions of parameters at a handful of bytes each: we don't fit.

The solution is the same one maps use. A map doesn't write down true distances: it draws them in grid squares, and puts the conversion in the legend in the margin — "one square = ten kilometers". Quantization does the same to the parameters: you split them into small groups (say 64 at a time), note each group's legend in the margin — the scale, calibrated on the group's largest value — and then write every number as a count of squares: a small integer. With one byte per number you get a fine grid (from −127 to +127); with half a byte, a coarse grid (from −7 to +7). (Symmetric: you give up one value to keep an exact zero at the center — which is why −127 and not −128.) To read the true value back: squares times legend, done.

The split follows common sense: the brain on the desk, small and always in use, keeps the fine grid; the library — the real bulk — drops to the coarse grid and shrinks by a factor of just under eight compared to the four-byte original — "just under" because the legends, with their scales, cost a few bits too. The price is a slight blur: like an MP3, which throws away precision where the ear can't hear. But "where the ear can't hear" is a hope, until you measure it.

The Dress Rehearsal

And how do you measure it? You can't compare the compressed scribe with the original by feel — "well, the stories look the same to me" is not engineering. You need a dress rehearsal.

It works like this: in the lab you build the scribe's stand-in — the original model made to wear the same rounding lenses the field copy will wear: every parameter is compressed into squares and legends and immediately re-expanded, blur included. Then the original scribe and the stand-in are given the exact same exam texts to read, and their rankings of bets are compared, word by word. If the rankings nearly coincide, and the quality measured on the exam texts drops by a hair — not "it looks the same": it drops by this much, measured — the compression passes.

And there's a second dividend, even more precious: from that moment on, the stand-in becomes the master yardstick. The copy headed for the small hardware, written from scratch in another language, will have to answer like him — same text, same rankings, within tolerances declared in advance. Any discrepancy beyond tolerance is a bug, and you know exactly where to look. The field scribe doesn't have to "be good enough": it has to match the stand-in, down to the decimal.

When the Metaphor Becomes Hardware

And here the circle closes with the stubborn idea from the start, because on tiny hardware this metaphor stops being a metaphor.

The chip I'm wrestling with has two memories: a small, fast one (the RAM) and a bigger one that's slow to read (the flash — the USB-stick kind). Now that we know how things work, the layout designs itself: the scribe's brain lives in RAM, always ready; the library with its manuals (MoE) lives on flash, and the manual the librarian picks gets read only at the moment it's needed.

graph LR
  TOK["Each word"] --> RT{"Librarian (router)"}
  subgraph RAM["RAM · small and fast"]
    B["Brain — parameters always in use"]
  end
  subgraph FLASH["Flash · large and slow"]
    L["Library of manuals — experts (MoE)"]
  end
  RT -->|picks 1 manual| L
  B --> CALC["Computing the word"]
  L -.->|just those pages, read on the fly| CALC
  CALC --> OUT["Word written"]

The brain lives in RAM (always ready); the library lives on flash, and at every word a single manual gets "photocopied". The metric that matters: how many bytes you read from flash per token.

At that point the most important design metric is no longer how many parameters you have, but how many manual pages you need for every word you write — how many bytes you must read from flash to generate one token. And it's a metric no data center ever needed to invent, because no data center ever had a few megabytes of RAM.

The title of this piece is a joke, but the story is true. Attention Is All You Need is the 2017 paper that invented the Transformer: every floor a notebook floor, no page limit. It works wonderfully — until you have memory problems.

The big labs respond by growing the notebook with sheer hardware; from the bottom up, you do the opposite: tiny notebooks, almost everything on summaries, the knowledge in the library. Two answers to the same question: how much does remembering cost?

(A side note to clear up a common mix-up: "GPT" is not a synonym for "AI model" — it stands for Generative Pre-trained Transformer. The Transformer is the architecture, the building's blueprint; GPT is a family of models built on that blueprint, the way the "Golf" is a family of cars; ChatGPT is the product that talks on top of it.)

References

The diagrams in this piece were generated by Claude from my prompts, and the analogies were born in a long late-night conversation with Claude Fable 5. But the concepts — and in some cases the metaphor itself, which the field baptized that way from the very beginning — come from here:

• Vaswani et al., Attention Is All You Need (2017) — the Transformer paper: every floor a notebook floor.
• De et al., Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (2024) — the large-scale validation of the summaries + local notebooks mix.
• Botev et al., RecurrentGemma: Moving Past Transformers for Efficient Open Language Models (2024) — the open models built on Griffin.
• Jacobs, Jordan, Nowlan, Hinton, Adaptive Mixtures of Local Experts (1991) — the library of experts is more than thirty years old, and the metaphor is right there in the title.
• Shazeer, Fast Transformer Decoding: One Write-Head Is All You Need (2019) — the shared notebook (multi-query attention), from another title riffing on the same paper.

A Little Glossary