Interesting idea, but it reads like an LLM write up. For me to engage properly with your written work, I need to see human effort. There are too many half-baked ideas out there, and unless you, as the author, thought the idea is worth putting proper effort into communicating, why should I think of this anything other than something someone said in passing? One way you could communicate effort to me, and likely others, rather than this work looking like the output of some LLM, is to use graphs, schematics and figures to support claims. Metrics are extremely misleading, only really useful when we've verified what is being measured is what one assumes is being measured. Metrics on their own are often deeply bullshit, so without any kind of graphs of trends or other important qualities, I assume it's all LLM output, since LLM's will default to not plotting anything while humans tend to go the opposite way.
Also, more effort into completeness needs to go in. Perhaps it is in there, but you just report an "accuracy" without defining what that means. The more I look, the more it starts becoming apparent this likely hasn't received any real human over-sight at all. "What does Grassmann mixing actually buy us?" It buys nothing, because it is a concept, Claude.
Neat idea, but in my opinion devoid of critical thinking / analysis or the level of rigour required to be suitable for broader dissemination. I'd put effort into making this a more credible body of work ASAP. Slop like this WILL cripple your standing in the field.
Because having an online footprint is a great way to get cloned and scammed? It’s a great way to get tracked? It’s a great way to get addicted to these “social networks”?
It should be unsurprising if comments here, made in public, capable of being challenged are far higher quality than official "peer review" which has really become a bit of a joke. Has been for quite a long time, to be fair. (And unsurprising if not, it's a web forum).
Peer reviewed and published means precisely excrement in a world where papers are not retracted and failures to replicate are refused publication by the "prestigious" "peer-reviewed" journals that published the original.
Intelligent, reasoned, knowledgeable review going back and forth with the author in public makes "peer review" seem just as silly as it usually is. Review by peer professor's indentured servants aka PhD students is often actually higher quality, if wildly dishonest and quite frankly disgraceful.
We now live in a world where LLMs can produce papers. Actual research is a little more interesting, with or without LLM. The journal peer review system doesn't seem to be helping.
I am not against some kind of "public review" and I really like arxiv.org. But I do not see it as a replacement for a peer review. It may start after a peer review when the paper is published, which IMHO should almost always be in an Open Access way.
For AI, the borders seem to have blurred a bit between academic scientists (e.g. at universities) and external scientists at companies or even private people). The problem with the latter is that it is much harder to enforce good scientific practice. Think only about the time it takes to write the state of the art of an ordinary conference paper.
I am also a bit drawn between giving everyone the chance to contribute to research and ensuring good scientific practice. Just writing down your latest ideas and successes is not the same as writing a scientific paper that shows the research in the context of past research and needs, for instance, to be much more mathematically rigorous.
Peer review and scrutiny out in the public is more valuable than only done by a couple of paid guys from a journal.
It's fairly basic. For a work published on a public platform, the same guys paid by the journal can also review it if they so choose to in addition to those people in the public who were not paid by anyone.
In my academic career, I had one instance of reviewers actually reading a paper. Otherwise -- accept or reject -- the comments made clear the paper had never been actually read.
Retractions are non-existent, even with clearly proven academic malpractice.
I am sorry for that. Fortunately, this does not resemble my experience (electrical engineering/digital signal processing).
Usually I got quite critical comments (positive and/or negative). Yes, sometimes a reviewer does not write even a single sentence before giving an average score to complete the review process. But this has not been the regular case.
And as a reviewer for a conference paper of 4 to 6 pages, I usually assigned 1 to 3 hours to read it (journal papers: 1 to 2 days) and to give comments.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
Compute, data, and parameter count are the three logarithmic values that correspond to model performance.
So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.
If you find this fascinating and want to dive more, I encourage checking out nanochat (solid GPT-2 repro; minimal LoC); and trying to make architectural tweaks with a coding agent.
“Learning through experimentation” has never been easier or more fun in my opinion. If you have a GPU, even an older 8GB card (or a few bucks for Runpod), you can pretrain your own small language model and start experimenting / tweaking it.
> So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.
I don't get how changing the attention head (i.e sliding window attention instead of vanilla transformer) is only seen as a "compute" improvement. Indeed, it improves the complexity from quadratic to linear, but there was a reason why people used transformer and not sliding windows for so long (they were already there in 2021).
Making sliding window attention work with performances on par with vanilla transformers is a non trivial problem and is not solved by only putting more compute. This still requires "smart" ideas in that direction.
Yes if you took a transformer network and added connections until it became a full network, then it would still work (?) but it would be a lot more expensive.
(?) We actually don't know that but it is very likely.
Makes me wonder if the brain just does it with brute force.
The neocortex would be akin to something like a 100T neural network, and that’s with the maybe overly reductionist assumption that only synapses are doing anything when there’s a lot of biology happening in there. Every cell is filled with its own computation and state. It could be closer to a 10E model if all that is in play.
The neocortex is nowhere near the whole brain.
Architecturally the brain could be described as a slow but ultra-wide machine with colocated memory and thus an effectively enormous memory bandwidth. That lets it evaluate its whole parameter set continuously.
I keep thinking the brain must be algorithmically doing something super clever but maybe not. Maybe old mother nature just found the “bitter lesson.” Maybe it’s algorithmically inefficient and messy but there’s just a whole lot of it.
I think after you factor it in being on the extreme end of slow but wide (like laptop GPUs) and being a native representation of the neural network rather than a computed silicon approximation, and being more similar to a recurrent/continuous learning model rather than a huge all at once back prop, are there more tricks required?
My thought is you could probably set up something similar with today's technological capabilities but it would probably take decades to train and be expensive to develop which is commercially unacceptable.
Analog computation probably. Theoretically, technology like thermal wells and others that harness physics to find the local minimum energy state allow inference with multiple orders of magnitude improvements in size, energy, and speed. Rough back of the napkin gives SD-card size 1t parameter models on about 2.5 watts using thermal wells (cryogenic cooling required, afaik), achieving density approximately similar to NAND flash per parameter.
They only demonstrate this idea on a laughably small toy problem. An 18M parameter model (which they call “moderate”) on an ancient dataset proves exactly nothing. Plenty of bad ideas appear to work at trivial scale.
Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.
For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.
> So z_t is a token in the input, and z_t+Δ is some later token.
I think you get what I'm saying, but when they compute the gate they do:
h_t_new = alpha_t * h_t + (1 - alpha_t) * g_t
If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))
Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.
So it could just be a sign error and still be right in the code, or I could just be getting it wrong.
If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.
Fair point. If only the paper had a link to the code.
I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention.
Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past).
Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true.
In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.
They also note they kept the proposed model very simple to make it easier to understand (and presumably train):
In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.
No my point is more general. The headline promises me that I don't need attention based transformers, then you see the benchmarks and they tell you they got 10% worse perplexity. How is that going to convince me that I don't need attention?
I mean if you look at e.g Mamba 3 [0] (see page 12 and 14 for benchmarks), the perplexity is roughly the same and the benchmark scores are roughly the same. If you tell me attention doesn't matter you have to convince me by showing that you can get the same performance with a variety of architectures. In short, I would have expected the paper to be a survey across model architectures that all perform roughly the same, which then would give us the impression "hey, they're all performing the same, maybe we don't need attention after all!", instead we get to be shown that their pet architecture is worse, which is the opposite of what the headline conveys.
If I gave a random ML researcher this paper, they would just walk away and work on the next transformer-mamba hybrid.
gnabgib | 12 hours ago
jthomas1 | 11 hours ago
Grimblewald | 11 hours ago
Also, more effort into completeness needs to go in. Perhaps it is in there, but you just report an "accuracy" without defining what that means. The more I look, the more it starts becoming apparent this likely hasn't received any real human over-sight at all. "What does Grassmann mixing actually buy us?" It buys nothing, because it is a concept, Claude.
Neat idea, but in my opinion devoid of critical thinking / analysis or the level of rigour required to be suitable for broader dissemination. I'd put effort into making this a more credible body of work ASAP. Slop like this WILL cripple your standing in the field.
helpyou840 | 9 hours ago
Agreed.
I say: “if you want a human response, then give human effort”
porkshoe | 9 hours ago
random3 | 8 hours ago
ghurtado | 7 hours ago
latentsea | 3 hours ago
kooi | 10 hours ago
;)
RossBencina | 10 hours ago
jacquesm | 8 hours ago
kooi | 7 hours ago
tmsh | 10 hours ago
MYEUHD | 10 hours ago
blobbers | 8 hours ago
AspireOne | 8 hours ago
elendilm | 3 hours ago
It's his/her prerogative to keep an online footprint or not.
khalic | 2 hours ago
drnick1 | 10 hours ago
harry8 | 10 hours ago
It should be unsurprising if comments here, made in public, capable of being challenged are far higher quality than official "peer review" which has really become a bit of a joke. Has been for quite a long time, to be fair. (And unsurprising if not, it's a web forum).
Peer reviewed and published means precisely excrement in a world where papers are not retracted and failures to replicate are refused publication by the "prestigious" "peer-reviewed" journals that published the original.
Intelligent, reasoned, knowledgeable review going back and forth with the author in public makes "peer review" seem just as silly as it usually is. Review by peer professor's indentured servants aka PhD students is often actually higher quality, if wildly dishonest and quite frankly disgraceful.
We now live in a world where LLMs can produce papers. Actual research is a little more interesting, with or without LLM. The journal peer review system doesn't seem to be helping.
hasley | 5 hours ago
For AI, the borders seem to have blurred a bit between academic scientists (e.g. at universities) and external scientists at companies or even private people). The problem with the latter is that it is much harder to enforce good scientific practice. Think only about the time it takes to write the state of the art of an ordinary conference paper.
I am also a bit drawn between giving everyone the chance to contribute to research and ensuring good scientific practice. Just writing down your latest ideas and successes is not the same as writing a scientific paper that shows the research in the context of past research and needs, for instance, to be much more mathematically rigorous.
elendilm | 3 hours ago
It's fairly basic. For a work published on a public platform, the same guys paid by the journal can also review it if they so choose to in addition to those people in the public who were not paid by anyone.
blagie | 8 hours ago
In my academic career, I had one instance of reviewers actually reading a paper. Otherwise -- accept or reject -- the comments made clear the paper had never been actually read.
Retractions are non-existent, even with clearly proven academic malpractice.
hasley | 5 hours ago
Usually I got quite critical comments (positive and/or negative). Yes, sometimes a reviewer does not write even a single sentence before giving an average score to complete the review process. But this has not been the regular case.
And as a reviewer for a conference paper of 4 to 6 pages, I usually assigned 1 to 3 hours to read it (journal papers: 1 to 2 days) and to give comments.
charcircuit | 10 hours ago
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
janalsncm | 10 hours ago
dannyw | 4 hours ago
Compute, data, and parameter count are the three logarithmic values that correspond to model performance.
So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.
If you find this fascinating and want to dive more, I encourage checking out nanochat (solid GPT-2 repro; minimal LoC); and trying to make architectural tweaks with a coding agent.
“Learning through experimentation” has never been easier or more fun in my opinion. If you have a GPU, even an older 8GB card (or a few bucks for Runpod), you can pretrain your own small language model and start experimenting / tweaking it.
addag | 10 minutes ago
I don't get how changing the attention head (i.e sliding window attention instead of vanilla transformer) is only seen as a "compute" improvement. Indeed, it improves the complexity from quadratic to linear, but there was a reason why people used transformer and not sliding windows for so long (they were already there in 2021).
Making sliding window attention work with performances on par with vanilla transformers is a non trivial problem and is not solved by only putting more compute. This still requires "smart" ideas in that direction.
amelius | 3 hours ago
(?) We actually don't know that but it is very likely.
api | 2 hours ago
The neocortex would be akin to something like a 100T neural network, and that’s with the maybe overly reductionist assumption that only synapses are doing anything when there’s a lot of biology happening in there. Every cell is filled with its own computation and state. It could be closer to a 10E model if all that is in play.
The neocortex is nowhere near the whole brain.
Architecturally the brain could be described as a slow but ultra-wide machine with colocated memory and thus an effectively enormous memory bandwidth. That lets it evaluate its whole parameter set continuously.
I keep thinking the brain must be algorithmically doing something super clever but maybe not. Maybe old mother nature just found the “bitter lesson.” Maybe it’s algorithmically inefficient and messy but there’s just a whole lot of it.
ngruhn | 49 minutes ago
captainbland | 28 minutes ago
My thought is you could probably set up something similar with today's technological capabilities but it would probably take decades to train and be expensive to develop which is commercially unacceptable.
K0balt | 24 minutes ago
erickgabriel | 10 hours ago
themagyar | 10 hours ago
oofbey | 8 hours ago
Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.
HexPhantom | 4 hours ago
mapontosevenths | 7 hours ago
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
magicalhippo | 6 hours ago
For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.
mapontosevenths | 6 hours ago
I think you get what I'm saying, but when they compute the gate they do:
If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.
So it could just be a sign error and still be right in the code, or I could just be getting it wrong.
If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.
magicalhippo | 5 hours ago
I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention.
Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past).
Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true.
[1]: https://github.com/Infatoshi/grassmann-flows/
[2]: https://github.com/Infatoshi/grassmann-flows/blob/main/blog....
[3]: https://github.com/kanenorman/grassmann
imtringued | 2 hours ago
Not sure what the point of the paper's title is if the resulting model does not match the transformer.
magicalhippo | an hour ago
In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.
They also note they kept the proposed model very simple to make it easier to understand (and presumably train):
In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.
imtringued | 29 minutes ago
I mean if you look at e.g Mamba 3 [0] (see page 12 and 14 for benchmarks), the perplexity is roughly the same and the benchmark scores are roughly the same. If you tell me attention doesn't matter you have to convince me by showing that you can get the same performance with a variety of architectures. In short, I would have expected the paper to be a survey across model architectures that all perform roughly the same, which then would give us the impression "hey, they're all performing the same, maybe we don't need attention after all!", instead we get to be shown that their pet architecture is worse, which is the opposite of what the headline conveys.
If I gave a random ML researcher this paper, they would just walk away and work on the next transformer-mamba hybrid.
[0] https://arxiv.org/abs/2603.15569
chiharukiryu | 4 hours ago
futune | 4 hours ago