I don't disagree with that. I did add an entire caveat paragraph there.
To me, it's more of a neat visualization, not something that can be used to interpret LLM behavior. Even with a lot of simplification, it can show some interesting patterns.
I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?
This is great, I've read multiple books and watched videos about the attention mechanism. Now that I understand it, this is the clearest example I've seen on how attention works.
I don't know much about LLMs but does that mean you have N^2 computation with the context size since every token needs to track how it relates to every other token?
Yes, except no with the KV cache. Because tokens aren't modified by future tokens you can cache the meaning of previous tokens. This makes the total effort linear over the entire context (or constant per forward pass).
> This makes the total effort linear over the entire context (or constant per forward pass).
This is incorrect. The compute required per forward pass to generate each additional token during decode will scales as O(N), even with a KV cache (without a KV cache, it would scale as O(N^2)). Over generating N tokens, it's O(N^2) with the cache (and O(N^3) without).
It's O(N) for a forward pass because that new token still has to "attend to" to each previous token. That requires N dot products: between the cached key vectors and the new query vector for the new position. You also have N reads from memory (K and V) which is probably gonna be your actual bottleneck. (Decode is memory-bound.)
This is why you should avoid long contexts, if you can, even with a warm cache. You will get charged more, in "cache read" tokens.
This is great, thank you. I have to teach this stuff on Friday so perfect timing. It's hard to explain the attention mechanism in a way that becomes intuitive because the weighting scheme does not help much with the intuition. Having a visualization like this helps a lot. Don't move that page please since I'll link to it!
UX report. I wished to examine attention state step by step, but I found the animation moved along too fast for that. So I tried pausing...
On Chromium/linux, pressing pause doesn't pause, instead resetting the animation to it's pre-play state - the current attention highlighting disappears. Pressing play again, restarts at the beginning. Having a commonplace "pause pauses, and play resumes" UI, could allow more time to look over state. A youtube-like slow playback 0.25? option might similarly help. Or perhaps even better, buttons for single stepping. Tnx for your work.
Is the attention explanation of why the model tells like this? I've seen that there are many discussions about this. (Image attention visualizations were not that good I think)
sva_ | 16 hours ago
[OP] ifz | 16 hours ago
To me, it's more of a neat visualization, not something that can be used to interpret LLM behavior. Even with a lot of simplification, it can show some interesting patterns.
smallmancontrov | 15 hours ago
apnabhidu47 | 15 hours ago
itsnasme | 15 hours ago
wopak | 15 hours ago
are you worried later-layer attention gets drowned out by earlier layers just because there are more of them contributing to the sum?
[OP] ifz | 14 hours ago
Right now only simple correlations are visible.
stared | 15 hours ago
I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?
[OP] ifz | 14 hours ago
When I started, I expected I'd have to experiment a lot to find something comprehensible. But this simple computation can already show some patterns.
stared | 14 hours ago
visarga | 14 hours ago
fuddle | 15 hours ago
ex-aws-dude | 13 hours ago
acedTrex | 13 hours ago
TomatoCo | 13 hours ago
ex-aws-dude | 13 hours ago
Or does it accumulate the relations like A relates to B, so also add in B's relations
libraryofbabel | 12 hours ago
This is incorrect. The compute required per forward pass to generate each additional token during decode will scales as O(N), even with a KV cache (without a KV cache, it would scale as O(N^2)). Over generating N tokens, it's O(N^2) with the cache (and O(N^3) without).
It's O(N) for a forward pass because that new token still has to "attend to" to each previous token. That requires N dot products: between the cached key vectors and the new query vector for the new position. You also have N reads from memory (K and V) which is probably gonna be your actual bottleneck. (Decode is memory-bound.)
This is why you should avoid long contexts, if you can, even with a warm cache. You will get charged more, in "cache read" tokens.
MCP123 | 12 hours ago
fermlon30000 | 11 hours ago
mncharity | 8 hours ago
On Chromium/linux, pressing pause doesn't pause, instead resetting the animation to it's pre-play state - the current attention highlighting disappears. Pressing play again, restarts at the beginning. Having a commonplace "pause pauses, and play resumes" UI, could allow more time to look over state. A youtube-like slow playback 0.25? option might similarly help. Or perhaps even better, buttons for single stepping. Tnx for your work.
scottcodie | 8 hours ago
lhk931122 | 8 hours ago
asd000hh | 6 hours ago
talhaanwar | 3 hours ago