Can LLMs Perform Deep Technical Comprehension of Computer Architecture Papers

78 points by Jimmc414 18 hours ago on hackernews | 23 comments

bob1029 | 14 hours ago

I think the most effective part of this architecture is having multiple initial conditions to sample from.

We've been looking at using gpt-5.6-luna to do hypothesis generation at scale. Running many copies of something approximately as powerful as gpt5.4 just to get a sense of what options exist before we put a stick into the mud.

Single agent loop does not work very reliably for deep research. Especially in domains with complex tool calling and environments. You can get it to perform sometimes (often enough for a demo to work), but the team will rarely adopt it because they want it to work ~100% of the time, not ~40%. The anchoring you get with initial findings makes it really hard to get unstuck without user intervention later on. When all findings occur as part of a deterministic research pipeline (tool), things tend to work better at the edges.

I've been considering a three stage pipeline that does hypothesis generation => investigation => synthesis using luna => terra => sol. This is the first LLM family where I feel like we can actually use the full range.

I have made an experiment with 'idea generation' last year (with much worse llms): https://github.com/zby/DayDreamingDayDreaming

The results were promising: https://zzbbyy.substack.com/p/reinventing-daydreaming-machin...

mahirsaid | 14 hours ago

Actually deep reasoning, can't reason without comprehension. This will make sense for technical uses as it's intended. I can see this being very useful for code error mitigation and fixes.

zeusk | 13 hours ago

Isn't the whole point of attention, some context comprehension in the stochastic parrot machine.

mahirsaid | 11 hours ago

Yes, however I do suspect that at some point comprehension trumps context. Meaning it will be evedent giving the ability to increase comprehension and retaining whatever context you have goes a long way. I dont have the ability to play such large models. Within the means of my hardware I have already playd around with LLM enough to know where balances will start to cause confliction. Priotizing effeciency within the accuracy. This can mean compromising some other aspect of the AI, or this can be viewed as a positive interoperability.

Terretta | 9 hours ago

> comprehension trumps context

Consider context as hyper-dimensional coordinate vectors gesturing at the starting concept cluster of a synthesis chain or thread to unspool.

If the model's comprehensive training activated by your context locates the right thread to pull, this could be considered comprehension? That it "got" it?

aetherspawn | 12 hours ago

The abstract is AI generated and pretty poorly written at that. A paper about grading AI output doesn’t even grade their own abstract.

stingraycharles | 11 hours ago

Can’t believe the abstract has an em dash, a “not X but Y” and a “rule of three” in the first sentence. This is ridiculous.

andai | 10 hours ago

The reference implementation of slop :)

Speaking of which, does anyone know a resource that lists these "tells"? I used to notice them all the time, but now that they've permeated 2/3rds of what I see and hear, I'm starting to go a bit numb.

quackzar | 9 hours ago

manaflow | 9 hours ago

suprjami | 8 hours ago

Another one besides Wikipedia

https://tropes.fyi/tropes-md

tiahura | 5 hours ago

Every time I see people complaining about clear writing with signposts, I feel like I'm in Idiocracy.
I recently just referenced the selective applicative functors paper and let it write me an implementation in scala. There is one already available in github, so I can't judge if it really just read the paper and implemented it, but the result was so minimal quick and amazing.

thejokeisonme | 11 hours ago

Well if you can't judge, doesn't it make your comment completely useless?

nairboon | 11 hours ago

No, I don't think OP's comment is useless. It's not just a blanket statement, "Wow, LLM's quick", but also critically reflective, like a brief limitations section. This opens the discussion, by indirectly posing questions like, "Is it really this fast or "cheating"? How could we measure this experimentally? Etc.

hilariously | 7 hours ago

It's useless because this is not novel at this point, its literally everyone posting on HN - I did a thing with an LLM but idk if its good! That doesn't open up the discussion, it retreads the exact same discussion that both sides are not listening to each other on.

N_Lens | 12 hours ago

In my experience: Absolutely Yes!

thejokeisonme | 12 hours ago

Slop about slop. Already tired of this new chapter of humanity...

suprjami | 8 hours ago

You're not just tired (em dash) you're exhausted!

amelius | 9 hours ago

Why not, most of computer architecture is just plumbing guided by quantitative experimentation (simulation).

Chris2048 | 8 hours ago

Whenever I put a whitepaper in notebooklm and ask it to create a presentation etc explaining the main points, it illustrates the obvious, easy to understand things in the brief, and misses (or oversimplifies) most of the hard technical parts..

elindiosolari | 3 hours ago

You can try https://claude.com/ for this