Why Long Context Is Hard: the KV Cache, MLA, and Kimi Delta Attention, Explained Simply

Storing a long chat can cost more memory than the thinking itself. The growing notebook, the compressed notes, and the whiteboard that never grows - from zero.

AI companies now advertise models that “remember” a million words of conversation.

The surprising part is the price. In a long conversation, storing the chat can use more computer memory than the thinking itself.

Part 1 of this series ended with a question: if a Mixture of Experts model is so efficient per word, where does the cost of a million-word chat hide? Here is the answer. Not in the experts. In the notes the model keeps about your conversation.

This is Part 2. Part 1 explained Mixture of Experts, the trick that makes huge models cheap per word. You do not need Part 1 to follow this one.

What is a token? What is a context window?

Models do not read whole sentences. They read small pieces, a word or a part of a word at a time. Each piece is called a token.

The amount of conversation a model can look back on is called its context window. “One million token context window” means: it can consider roughly a million word-pieces of your conversation at once.

The question is how it looks back without re-reading everything from zero for every new word.

The KV cache: the model's growing notebook

When a model writes word number 1000, it must consider all the words before it. Re-reading everything for every new word would be far too slow. So the model keeps notes about every word it has seen.

Those notes have a name: the KV cache.

The notes are useful, but they grow with every word. How bad does it get? Take the original note-taking design (researchers call it attention, and its standard version is multi-head attention, or MHA). If a model with DeepSeek-V3's dimensions used that standard design - V3 itself does not, as we will see - the notes for one conversation of about 32,000 tokens (roughly 25,000 words) would add up to roughly 131 gigabytes of memory [2][3]. That is more memory than most laptops have in total, spent on remembering one chat.

This is the hidden cost of long context. Making the model's thinking cheap (Part 1) does not fix it, because the notebook belongs to the conversation, not to the model.

The standard fix: sharing notes (GQA)

Inside the model, many small “readers” scan the notes at the same time. The most widely used fix, standard in Llama and most Western models since 2023: make groups of readers share one set of notes instead of each keeping its own. Llama and its peers call this GQA, grouped-query attention [1]. It shrinks the notebook a few times over. Helpful, but the notebook still grows with every word.

DeepSeek's fix: compress the notes (MLA)

DeepSeek went further in 2024. Do not store the full notes at all. Store a small compressed summary, and rebuild the useful parts when needed. Like keeping one page of clear written notes from a meeting instead of the full recording.

They cut the notebook size by 93 percent compared with their previous model, and the answers stayed just as good [2]. DeepSeek named it MLA, Multi-head Latent Attention. It shipped in DeepSeek-V2 and V3, the models that surprised the whole industry in 2024 and 2025 [2][3].

Kimi's fix: replace the notebook with a whiteboard (KDA)

Moonshot, the lab behind Kimi, published a bolder move: throw the growing notebook away completely. Keep a small whiteboard of fixed size, and rewrite it constantly. A new word comes in: write, overwrite, or erase. The notebook never grows, no matter how long the conversation gets.

The clever part: different areas of the whiteboard forget at different speeds. A corner holding a key fact from page one can keep it until the end. An area tracking small local details erases and reuses itself.

Moonshot calls this KDA, Kimi Delta Attention [4]. The whiteboard idea itself is one many labs have worked on for years (linear attention, Mamba, the DeltaNet family). Moonshot's published contribution is the fine-grained forgetting, plus a working recipe: three whiteboard layers for every one full-notebook layer. This keeps some perfect memory and still cuts the notebook cost by up to 75 percent [4]. Moonshot says this design is what powers Kimi K3's million-token context window [6].

Two more pieces of the Kimi K3 puzzle

Two smaller ideas finish the design, both described in Moonshot's own materials [6].

First, a giant model is spread across dozens of chips. Every word must travel between chips to reach the parts of the model that process it (the “experts” from Part 1). Kimi K3 compresses each word into a smaller form before it travels, and unpacks it on arrival. Smaller luggage, cheaper trips. Moonshot calls this Stable LatentMoE.

Second, deep models slowly distort information as it passes through their many layers - like a drawing of a cat copied again and again down a row, until the cat is gone. The classic fix, used everywhere since 2015, passes the original drawing along next to the copies; these are residual connections [5]. Kimi K3 uses an upgraded version. Each layer can also look back at several earlier layers and pull what it needs, not just copy from the one before it. Moonshot calls it Attention Residuals [6].

What this means in practice

When a company advertises a million-token context window, the real question is: which notebook design makes it affordable?

One last detail worth noticing: none of K3's parts appeared from nowhere. Kimi Delta Attention was published nine months before K3 [4]. Attention Residuals came months before. The lab tested its parts in public, then assembled them. That is what open research looks like - and it is why a two-part blog like this one can explain a frontier model at all.

That closes the two costs of modern AI: Part 1 covered the thinking, Part 2 covered the remembering. If a Part 3 would help - how these models are trained so cheaply - tell me on LinkedIn.

The headline says one million tokens. The story is who carries the notebook.

Mohamed Kadri operates and transforms Tier-1 telecom networks, and led a NOC's L2-to-L3 autonomy transition that automated 75 percent of Level-1 incidents. He now builds AI and cloud products, and maintains guardplane, an open-source control plane for agent fleets. LinkedIn · mkadri85.github.io

Quick questions and answers

What is a KV cache in simple words?

The notes an AI model keeps about every word it has seen in your conversation, so it does not re-read everything for each new word. The notes grow with the conversation and become the biggest memory cost of long chats.

Why does AI get slower and more expensive in long conversations?

Because the model's notes (the KV cache) grow with every word. For one 32,000-token chat, a model with DeepSeek-V3's dimensions using standard attention would need roughly 131 gigabytes just for the notes.

What is Multi-head Latent Attention?

DeepSeek's fix for the growing notes: store a small compressed summary instead of full notes and rebuild the useful parts when needed. It cut the notebook size by 93 percent compared with their previous model, and the answers stayed just as good.

What is Kimi Delta Attention?

Moonshot's fix: replace the growing notebook with a fixed-size whiteboard that is constantly rewritten, where different areas forget at different speeds. Mixed three-to-one with normal full-notebook layers, it cuts the notebook cost by up to 75 percent.

Is the 1 million token context window real?

The number is real as a capacity, but the engineering question is what it costs to serve. Designs like MLA and KDA exist exactly to make million-token windows affordable; Kimi K3's version is company-described until its files are independently checked.

Sources and references

Numbered by where they support this article.

1. Ainslie et al.: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023. Supports: the shared-notes fix (section: The standard fix).
2. DeepSeek-V2 Technical Report, 2024. Supports: MLA and the 93.3 percent KV cache reduction versus their previous model; attention dimensions behind the 131 GB illustration (sections: The KV cache; DeepSeek's fix).
3. DeepSeek-V3 Technical Report, 2024. Supports: MLA at frontier scale; model dimensions used in the 131 GB illustration (sections: The KV cache; DeepSeek's fix).
4. Kimi Linear Technical Report (Kimi Delta Attention), Moonshot AI, 2025. Supports: the fixed-size whiteboard, per-channel forgetting, the 3:1 hybrid and up to 75 percent KV cache reduction (section: Kimi's fix).
5. He, Zhang, Ren, Sun: Deep Residual Learning for Image Recognition, 2015. Supports: the original residual connections idea (section: Two more pieces).
6. Kimi K3 tech blog, Moonshot AI, 2026. Supports: KDA inside K3, the million-token window, Stable LatentMoE and Attention Residuals - all company-described until the public files are independently checked (sections: Kimi's fix; Two more pieces).

Further watching: Welch Labs, “How DeepSeek Rewrote the Transformer” - the best visual explanation of MLA. Note on the 131 GB figure: computed from the published DeepSeek-V3 attention dimensions (2 x 128 dimensions x 2 bytes x 128 heads x 61 layers x 32,768 tokens) as what standard multi-head attention would cost; DeepSeek-V3 itself avoids it with MLA. About this article: written by Mohamed Kadri; every number was checked against the primary paper or official release listed, and Kimi K3 figures are marked as company claims until the released files are independently verified. Last updated: July 27, 2026.

Part 1 of this series: Mixture of Experts (MoE) Explained Simply. More on operating AI in production: Your AI Agents Need a NOC.