> A previous project, the Distill Circuits thread, has attempted to reverse engineer vision models, but so far there hasn’t been a comparable project for transformers or language models.
Shocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years
This is one of those papers that ought to have several textbook chapters unpacking some of the insights... I'll just pick a couple things I love:
1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.)
2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.
myself248 | 12 hours ago
greenbit | 9 hours ago
ziofill | 9 hours ago
[OP] Bluestein | 9 hours ago
thatspartan | 9 hours ago
pooyamo | 7 hours ago
greenbit | 6 hours ago
MisterTea | 5 hours ago
amelius | 12 hours ago
How successful was that?
[OP] Bluestein | 12 hours ago
- https://distill.pub/2021/distill-hiatus/
... and, here we are.-
oofbey | 10 hours ago
[OP] Bluestein | 10 hours ago
robrenaud | 9 hours ago
https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l
Also long, but it's detailed and complicated, so there is no escaping that.
[OP] Bluestein | 9 hours ago
thesz | 7 hours ago
https://en.wikipedia.org/wiki/Royal_Road#A_metaphorical_%22R...
_jayhack_ | 9 hours ago
libraryofbabel | 6 hours ago
1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.)
2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.