Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.
I get that this is for explaining GPT-2, but I really hope laymen don't use it as an example of how modern models work (ex. absolute positional encoding is no longer used)
edit: I know that it mentions its not modern, but these kinds of details have major implications in terms of the representations a model can learn, which is in many ways the most important part!
tanseydavid | an hour ago
jwpapi | an hour ago
jwpapi | 59 minutes ago
shagie | 40 minutes ago
"Try examples while GPT-2 model is being downloaded (600MB)"
That's a hefty chunk of download and likely compute too.
utopcell | 57 minutes ago
[1] https://bbycroft.net/llm
esseph | 56 minutes ago
andblac | 37 minutes ago
E-Reverance | 25 minutes ago
edit: I know that it mentions its not modern, but these kinds of details have major implications in terms of the representations a model can learn, which is in many ways the most important part!