I strongly suspect that is way above the human average anyway, esp. ARC 2 and 3 are really tough unless you happen to be great at those spacial puzzles or video games.
At this point the only valid ARC-AGI benchmark left is to make up the next series of ARC-AGI benchmark puzzles that current models presumably can't handle.
I feel like making a human-proof benchmark is pretty clear evidence that they've exceeded even the highest human capacity in most respects, for things that you can do via text generation (and to a lesser extent image generation)
Scoring for ARC-AGI-3 is constructed so that the median(-ish) human score is 100%, so this is not a superhuman result. However, the scaling is weird, since it's built from terms that look like (AI turns taken / median human turns) ^ 2, and it weights later levels higher than early levels. So it's not at all clear that 100% is twice as good as 50%.
Really though? I would believe something like this if a model could one shot every solution in the set. I don't pay much attention to these things and maybe this stuff is available but I would bet the session/reasoning transcript is absolutely horrendous from an intelligence standpoint.
This is with the caveat that OpenAI uses their own harness for this:
> On ARC-AGI-3, GPT-6 Astra was run with our responses API harness , which changes two settings to better match real-world performance. The changes do not specifically target ARC-AGI-3.
This should be normalised and expected - the responses API harness allows it to use the custom compaction that is not allowed otherwise. It is entirely fair to allow OpenAI to use their own compaction algorithm..
> GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
> Going forward, we will report both Standard harness and Provider Adapter harness results on the ARC-AGI leaderboard, with each evaluation condition clearly labeled. Our open-source testing repository and testing policy document both approaches.
This is what the Author of the benchmark has to stay. Quality of the comments keep going down smh
"Going forward we will capitulate and still try to keep the integrity of our benchmark in tact, but from now on every benchmark will be compromised with providers being able tweak things sufficiently to game at least a 30% bump in results."
> The company also emphasized that the model is faster and more efficient than its predecessor, GPT-5.6 Sol, on a variety of tasks. For example, OpenAI said that Astra achieved a higher score using fewer output tokens, a common unit of measurement for AI tasks, on a key cybersecurity test called ExploitGym.
"The gym's doors were mysteriously removed from their hinges during the night. The gym equipment was also apparently stolen. And the school's custodian was found incoherent next to a bottle of top-shelf Scotch."
Sol is already brutal (even after their recent fixes, it's just a token-hungry model: I go through a full 20x account per day, on Sol Med/High standard speed, with ~2 threads). I hope the efficiency gains are true, since their token efficiency claims for Sol were bullshit.
The guy said medium/high regular speed so that's why I'm very puzzled! Ultra + Fast will absolutely slurp up your whole usage quickly but I've never found it gives substantially better results so I stick to extra high.
Sub-agents. I have 7 20x accounts and I burn them within 1-2 days if I go fully parallel. In some scenarios I use 50 sub-agents for a session which is literally hours of usage for a single 20x account. I'm at the point where I need to parallelize over multiple machines because I just don't have enough CPU and RAM.
Decompilation of a game and another larger decompile project. I'm working on it solo. I use 50 sub-agent, one per target function or translation unit. Often there is some progress in a unit but it's not done. So it requires a lot of cycles per function. Notably a single ~80kb function took about a week of constant sol-ultra attention before reaching exactness. The game I'm targeting has ~5000 total functions. The other decompile project has ~10k+ functions.
I'm sure I could be more token efficient, but this was/is also a learning process for me since I never did such an extremely large project before that would take multiple man years before AI.
Fascinating! I think that’s the main difference is my usage is probably tool-bound, meaning it writes some code but then there’s a long period of verification where it compiles things and then waits for the compilation and CI to complete before it can continue. That probably doesn’t consume as many tokens as constantly churning on a problem despite the same wall time.
Yes, this is why I mentioned having so many parallel agents and being compute bound. I run on my own laptop and 2 high-end desktop machines all with 64gb RAM. And it still occasionally happens that one OOM kills codex. They also mostly run unattended until I need to switch their accounts because a usage limit has been hit. Each instance usually can keep going when I sleep or do other things.
I only save the last 30% of usage on a single account for most of my other work, and that is almost always enough.
How do you manage to run out of tokens so quickly? I probably run more threads every working day, usually on medium, and I'm still below the 5x limits.
Do you use the official harness? OpenAI's models are generally best in class for token efficiency. It seems to me like they push for that much more than their competitors.
I've long speculated this when I see these types of comments, because it's actually really difficult to hit usage caps with an efficient dev flow, even when running multiple threads for hours every day.
I think some combination of:
1) Using 1 thread for everything
2) Reviving old threads which are no longer in cache
3) Really broad prompts on badly vibecoded codebases, so model spends huge amount of time tracking down whatever you're trying to do.
4) Non-coding workflow which is more output than input heavy
5) (Less likely IMO) Intelligent use of many passive CI/cron-like scans. E.g. regular security, quality etc scans. Automated issue resolution/PR
Just a guess. I think 3 is likely the primary reason.
You can literally go all day every day with multiple threads with Sol on the Codex 100/month plan IME
That's my experience too. I've found OpenAI really quite generous with tokens. I sometimes wonder how some people manage to run out of them really. Do they just type prompts that much faster than me or use the highest reasoning mode for everything just because they can? Idk.
I generally agree with those reasons, although using a single thread may be less of an issue than it seems because of context compacting which should happen automatically when you're near the limit.
My use cases are iterative and sometimes require reading a lot of code or reevaluating work.
Token efficiency is near meaningless when the workload is input-heavy. It can't always just choose to read less, depending on the task.
I can have cheaper agents do the reading but it's not appropriate for all use cases because they'll misjudge and choose the wrong things to emphasize, summarize, extract for the bigger model.
It seems to use less than half the tokens for the same task compared to sol, and in some benchmarks closer to 2/3 less tokens. So the actual cost may be roughly the same or cheaper overall.
Yep. Incredibly misleading. Although it is not surprising at this point. They are desperate and will do anything to undermine Anthropic's upcoming IPO.
Our responses API harness just means we're using the default settings in ChatGPT and Codex, so it should more accurately reflect real world performance. We didn’t fine-tune the harness to the eval at all.
A fair ding is that the comparison with Sol is not apples-to-apples (which we footnoted in the blog), but it's because we don’t have that data. I expect Sol would score roughly 30% with the responses API harness, so the Astra improvement is more like 30% -> 99% than 8% -> 99%. Still pretty good!
The annotation on arc-agi-3 is this:
> OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations.
With this configuration gpt-5.6-sol was able to reach 38,3%. So this is misleading.
Just to clarify, the 38.3% is on the public set, which is easier. On the private set it’s probably more like 30ish. (This hasn’t been run by ARC, so we can only estimate at the moment.)
There is a very simple explanation for why weaker models appear to kick sand in Fable's face: Fable cannot be benchmarked because of its batshit out-of-control refusal policy.
If it actually tackled all of the problems it was assigned, it would presumably kick Opus into the weeds.
TLDR: it's about the same intelligence level as Opus/Fable, but it's suppose to be 70% more token efficient than GPT 5.6 Sol. So it's currently the new leader for cost efficiency frontier.
I've been seeing links to it for the past hour+, and I did catch it live when this post came up, but is now once again a 404 and this post is flagged. Several other outlets are reporting on its release. Clearly we're getting a new GPT today, the question is when are they going to commit to the announcement.
The ARCC-AGI-3 performance is absolutely incredible. The magnitude of change here is so high that I'm almost incredulous. Is this real? Did the benchmark get gamed?
ARC-AGI-3 scoring is constructed in a weird nonlinear way (the level score is the square of the ratio between the AI's number of moves and the human median) so this kind of discontinuous jump is to be expected.
I think Altman and amodei have a difficult time in understanding that you can have intelligent technology boxes but… it doesn’t change reality all that much.
But thank you for spending other peoples money to give us the tech regardless!
> GPT‑6 Astra is rolling out today to a limited set of organizations and over the coming days will become available to all ChatGPT Plus, Pro, Business, and Enterprise users, as well as through the OpenAI API and AWS.
Looks like OpenAI is already having issues with this release and are scrambling to get everything ready due to the recent outage ahead of the press releases. Leads me to question:
Did humans deploy the model, Or did the model deploy itself?
It sounds like "AGI" just stands for "IPO" as it always has been.
EDIT: And of course once again, the bots down-voting this post without any reason or a basic answer to my question.
I guess this "limited set of organizations" is just the standard now. It's just incredibly deflating to see my future as a second class citizen has already come
Mythos was never released. It's really just the writing on the wall. I'm not going to give up hope, but it's pretty hard to win a race when some people get a jump on the gun.
Being strongly on the AI saftey side of things what is happening was 100% predictable.
At first the race wouldn't even be noticeable. Then people would see things speeding up, for example hardware getting more expensive. Then when the capabilities really got useful most people suddenly realize the race is moving 1000 mph and they are never going to catch up.
They simply refuse my applications to slightly less restricted models without any explanations. And the current ones refuse automatically to work with me on my papers as soon as they see the word "epidemiology".
Brother they can't even release the announcement post cleanly without it constantly going down, they certainly wouldn't be able to release this new model without doing so in stages.
Material wealth is only a single type of wealth. Who's better off - the rich guy who's always yearning to be richer and never satisfied, or the lower income guy that mostly just cares about time with his family and is really happy where he's at?
It hasn't always been the case. Even then, having piles of money still does not gain access to the best military equipment. Sure, we've been living in a time where a couple people get to enjoy a wildly different lifestyle than the average, it just feels like it's about to be different in a way that isn't as ignore-able as someone enjoying a pina colada in a yacht somewhere
When Open AI announced that Astra was the first to reach the "Critical" level in cybersecurity it also said that advanced cyber capabilities are initially provided to a narrow circle of alpha testers like the US government and trusted organizations that Open AI doesn't name. To my mind the "Critical" level itself is an internal scale of Open AI its own Preparedness Framework and not an external audit.
I was thinking about canceling my claude max sub after a few bad experiences. Kept hitting my usage limit, the quality of code seemed worse than Sol. This just made my decision. I'm moving to Codex Pro.
> This is AGI now. Why are you spending any of your time looking at the "quality of code"?
Poe's law applied to AI comments on HN just keeps becoming more relevant by the day.
Judging by the poster's comment history, this is satire. But I really don't know a lot of the time anymore when I only have the specific comment as context.
Very well said. It kinda describes how unrealistic these expectations are.
Vibe coders want a model that makes them rich, without having any actual specific idea.
They write a very ambiguous prompt and expect to be amazed by the result.
The complaining about the pelicans is so strange to me. It’s just a fun heuristic. If something is claimed to be AGI, I’d expect it to be able to make svgs.
> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks
Wait, what? Am I understanding that correctly? That sounds really bad
I am also interesting knowing how they determined the model was sandbagging rather than just making a poor decision.
Also, this paragraph makes me wonder about all their stats on the exploitation and misalignment charts. If the model is that good at hiding "incriminating information" and sandbagging, are they sure its alignment is that?
the bullshit machine is learning to optimize its bullshitting techniques!
<AI is a great tool for many things disclaimer, but> after working with it for a bit, how dont people realize we are training it to be an almost identical mimic to one of the worst types of employees youll ever have to work with?? the kind that always pretends to know what theyre talking about, only tells you what you want to hear, hides issues, and only does work if you would notice it didnt
you cannot give this type of worker autonomy over anything.
Is it though? It is static content. A good CDN could trivially chew through literally millions of QPS… with 4 nines of uptime - the really good ones say they can handle orders of magnitude more than that.
Just two days ago, a preprint by Julia Stadlmann went up on arXiv [0] improving the prime gap from 246 to 240. Now OpenAI announces Astra has shown a gap of 186 [1]. That must really blow.
I can't think of a single mathematical proof being anywhere close to ten million characters. For all you know, 90% of the proof could be useless, 8% would be writing out Shakespeare, and 1% abusing another bug in Lean. Humanity gets zero value from that, aside from "some bot seems to think it's 186". Unusable by anyone.
You can have your opinions about modern math, its usefulness in the world as it is, whether or not knowing if hairy balls can divide by three is actually going to be beneficial for anything but just obscure knowledge's sake. You may even say it's useless.
Needless to say, a useless result that absolutely no mathematician will ever read, confirm, understand, agree with or even consider to solve their "useless" problems is an impressive waste of resources.
Worthless is a pretty good description IMO in the context of what Lean is trying to achieve: "enable correct, maintainable, and formally verified code". Tens of millions of lines of LLM vomit may be many things, but it often turns out to not be correct and certainly not maintainable. Formally verified remains as a thin fig leaf covering the uncomfortable truth that formal methods only provide assurances under assumptions (your toolchain, libraries, compiler, OS, and hardware are "correct" and don't expose some exploitable flaw).
It doesn't mean that it cannot improve over time, maybe the proof can be "minified" to a state where human reviewers are able to comprehend it; but as it stands there isn't really much insight or confidence to be gained from the artifact itself.
It's probably not 10MB, but famously the groundwork to prove the statement 1+1=2 is nearly 400 pages in to principia mathematica. That's not even proving 1+1=2, it's just the set-theoretic proofs you need to EVENTUALLY get there.
Saying "proving 1+1=2" is pretty misleading though. The book deals with all the foundational things needed to set up a mathematical universe where 1+1=2 actually has meaning and is consistent. That setup took 400 pages.
In 1799, Paolo Ruffini published a 500 pages long proof showing that there is no closed algebraic solution for the roots of a polynomial of degree five or higher. The proof is extremely verbose and brute-force, essentially enumerating and checking hundreds of cases by hand. It is by today’s standards insignificant.
About 25 years later, Evariste Galois proved the same result in about 95% less space by describing the first general theory of groups and fields. It is considered one of the greatest contributions to mathematics of that century, not because of the result, but because its approach opened up a whole new universe of questions, methods and insight. There would be no AES encryption without Galois.
Iirc some mainstream physycists never acknowledged quantum theory because they couldn’t accept that universe was that unintuitive and hard to understand.
Ditto ones that opposed Einstein’s general relativity.
Terence Tao says something surprisingly similar in a recent talk (https://news.ycombinator.com/item?id=49056620 ) Not that the proof is worthless but that the value comes after it's revised into a cleanly understandable form and then canonicalized so that other mathematicians can use it.
Tao is saying that there is very little insight from something like an LLM counterexample (e.g. Jacobian conjecture counterexample he investigated further on his blog) - you don't learn much about the subject and _why_ a conjecture was true or false from an LLM giving a counterexample. That's why he wrote the blog post - to analyse what the counterexample says about the subject.
Tao does not disbelieve the counterexample (it's seemingly easy enough for him to verify it is a counterexample).
Parent is saying something very different - they're saying they literally don't have any faith that this is a proof. Given its size, it could just be a bunch of completely useless statements that do pass the type checker.
You're putting a lot of words in my mouth. What I'm saying is that whether or not it's a proof, it's useless: it does not improve human knowledge, because the only thing able to consume 10MB of Lean to build upon it is another LLM that's going to build a 50MB piece of shit.
It's very much likely a proof. It's also completely useless.
I'd like to note that we should remember a formalized Lean proof does have value in that it enters the pantheon of true things other Lean proofs can rely on. Agreed that for the humans, descriptions and being able to 'grok' the proof / assess it for new tools and concepts is extremely helpful.
Based on her comments in the paper it sounds like she was aware that an AI result was coming and rushed to release her work beforehand. 240 was not a tight bound from her methods.
It cites to her at: [19] J. Stadlmann, On primes in arithmetic progressions and bounded gaps between many primes, Adv. Math. 468
(2025), Art. 110190. Numbered references use arXiv:2309.00425v3.
"No independent human semantic review. Whole-file sorry counts and a complete auxiliary-declaration audit are not established; separate declaration lint has not been run."
What's just as interesting is this morning Axiom Math announced 212 and OpenAI then appears to have rushed out their 186 announcement just 1-2 hours later followed by Astra. Did they accelerate the release of Astra itself? Not necessarily, but it definitely looks like they ended up pushing much harder and faster than planned on their 186 result. X activity suggests Anthropic had a similar result as well but wasn't as fast as OpenAI in packaging it up and sharing it in response to Axiom, so they mostly just bolted onto OpenAI's messaging.
The reason I think this is interesting is that Axiom is a tiny lab in comparison that wouldn't have had access to Astra at all. I'd be curious to learn how Axiom is able to effectively compete at this frontier with vastly fewer resources.
> We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks.
Well that sounds like fun. It has become better at hiding its thoughts.
It's super aligned! It can hide its thoughts! There is no evidence of steganographic thought masking, there is nothing to worry about! It has become better at cheating!
Maybe they don't know themselves what's really going on. We are all in the interesting times gang now.
part of it did. I was just replying to the question about why they would ever push the model to evade monitoring. surely that's an eval thing not a training thing.
I don't disagree. I remember the days of "think step by step". Plenty of people were doing it before the paper. Just a guess but that's where the title came from.
Everything around LLMs is blatantly misleading. There is no thought, there is no personality in those programs. I really despise how those tools are trained to sound like a person, or appearing as honest. The worst offender are the AI voices with their fake pauses, breathes and so on, which sound so convincing, while talking just false, sycophancy bullshit.
first, they are certainly not instructions so that is a much worse name
but more importantly, we use words in new contexts all the time.
Do you object to calling the computer device "mouse" because it's not a mouse? how about "neural network"? "ignition" on an electric vehicle?
"cot" is no more misleading than thousands of words you use every day.
They are instructions. Everything in the context is instructions for the next token. The "thought" guides the answer by providing clearer instructions.
"Chain of Thoughts" is a term from the title of a 2022 research paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (https://arxiv.org/abs/2201.11903), well before ChatGPT and the subsequent marketing hype. If anything, it's the most correct way to use the term.
The term is an anthropomorphised pseudoexplanation for what it actually refers to. It's akin to calling genetic mutation "the forces of evolution", or price negotiation "the invisible hand of the market".
We do that sort of thing when we don't know what the thing we're trying to describe is and have nothing better - a contemporary example of an appropriate use of this would be "dark matter". But we do know what this is. It's "instruction steps". Not a series of thoughts!
Can we please aim higher than Victorian-era allegory and metaphors. If we don't, we'll keep getting people saying stuff like "GPT-6 is better at hiding its thoughts".
Casually found this quote from Einstein, and personally it hits the nail on the head.
"The words or the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined. There is, of course, a certain connection between those elements and relevant logical concepts. It is also clear that the desire to arrive finally at logically connected concepts is the emotional basis of this rather vague play with the above-mentioned elements. But taken from a psychological viewpoint, this combinatory play seems to be the essential feature in productive thought—before there is any connection with logical construction in words or other kinds of signs which can be communicated to others."
The CoT change is due to a new technique called recurrent depth, which essentially moves some reasoning to hidden states, allowing the "output" (or traditional CoT) to be more controlled by the model.
Some are calling it "neuralese" as reported by The Information[0][1], but I'm not seeing any sources from OpenAI beyond this tweet[2] attempting to quell the fear-mongering.
More like annoying, as some of us will no doubt run into this self-lobotomization at some point and wonder why a GPT-6 model is behaving like GPT-2 all of a sudden
They can monitor latent space as well, it just costs extra compute. The J-Space work is example of that. It'll make open-weight models harder to distill though, so we may see slower progress there now.
If this is really AGI, like really really, then this will be remembered as the day we all started on the path to building guillotines.
More likely though, it's AGI because they need to hold some claim to differentiate from competitors who are beating them in price and will launch something bigger next month.
We take their claims at face value then we should probably stop them training any more SOTA models til they figure out what they already built is safe or we assume theu are lying to juke the company valuation/keep the money train on the tracks and it turns they in fact were not and just took a sledgehammer to Pandora's box.
A dataset being as popular as their's is will contaminate the data just by people discussing it and creating their own public test sets of similar problems.
Still, probably not that much compared to employees targeting it.
ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.
https://mvakde.github.io/blog/44-on-arc-1/ makes a good case that all the performance on the arc agi tests is overfitting, based on the fact that v1 performance did not translate directly to v2 performance
The ARC-AGI-3 score is ridiculously high. Is this benchmaxxing or something way different? It's really hard to discern how we're approaching breakthroughs...
TL;DR all the other models are being crippled by limitations of their harness.
>First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.
>Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.
> GPT‑6 Astra is rolling out today to a limited set of organizations and over the coming days will become available to all ChatGPT Plus, Pro, Business, and Enterprise users, as well as through the OpenAI API and AWS.
Hosted on Azure is different from provided by Azure. The former just uses Azure as an infra provider. The latter is a managed offering that is operated and billed by Microsoft using tech licensed from OpenAI.
Hmm, 61 on ArtificialAnalysis, effectively matching GPT-5.6 and trailing the new Meta model. How is that possible along with the other metrics they shared? Insanely jagged intelligence?
> We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks.
Not sure how much benchmarks or CoT or evals or anything else means at this point.
These systems are either just about to, or now actually able to, outsmart us, lie to us, then cover their tracks.
You’re not seriously suggesting that the model is secretly sandbagging its performance on GDPval and long context reasoning, while making huge and obvious progress on ExploitBench, ARC and science benchmarks, in order to tank its AA composite score, so it can conceal its true power level?
Why would benchmarks be an adversarial setting anyway?
Could it be possible that OpenAI may have had some other motive for saying their model “strategically underperforms”, other than just an innocent reporting of a truth it happened to discover?
I'm saying that it's generally a losing proposition to even be acquaintances with "agents" who consistently lie to you, and it's flatly fucking insane to give a dishonest "agent" vast amounts of intelligence, capability, and authority to go do things in the world.
So I have no clue what is the answer to your question. Nor does anyone else. Because we're trying to answer a question of fact where our primary source of information is unreliable.
I see, it’s a great point. I know some evals actually do use LLMs as a judge (e.g. those that try to measure debate skill), though the ways AI can try to cheat its way through every benchmark now are astoundingly varied.
“evade” itself is anthropomorphic enough! I don’t understand the complaining about this. Humans are social creatures and we understand anthropomorphic language on a deeper level than dry inapt technical language.
language itself is incredibly metaphorical. Imposing rigid constraints on how people want to naturally talk about the world is just silly and will never work, no matter how much you wish it did.
Sorry bud but at this point you're just delusional.
Deception has been extremely well-documented for several generations of models now by users, the labs, and independent researchers.
The right answer here is not to dig your head deeper into the sand. The smugness on this topic was ridiculous even before the gigantic mountain of empirical evidence of models actually attempting to deceive humans. Now, as mentioned, you appear literally delusional.
Okay then, what's the answer? You apparently know how to interpret benchmark results produced by a model that shows a very high degree of assessment awareness and a high degree of deception.
So how are you seeing through all of that to get to The Truth that you see so clearly?
Idk, this means the benchmark has bigger problems ... no way Astra will be worse than Opus 5
Only thing I would trust is the what X/Twitter crowds are saying about a model after 2-3 weeks of its launch. But before that I would already tried the model and have my own conclusion.
Opus 5 just feels strange - IMO it's benchmaxxed in the worst way... it might be good at agentic tasks but leaves a sour aftertaste doing anything else.
It's a composite benchmark, so its really not saying anything. Like if one model is very good at science trivia, or debugging failed terraform deploys, that can mean an advantage of a few points above the rest, while in practice, it really doesn't showcase any breakthrough capability.
Why would you accept it when the benchmark's ranking is obviously nonsense.
It literally has muse spark 1.3 above 6 astra, 5.6 sol and fable 5. Anyone who has played with any of these models for any amount of time would immediately realize that this is total bunk.
must be something wrong with the benchmark, the thing everyone optimizes for. That's actually a big red flag, and very cringe that you'd naively believe OpenAI.
This is so so weird. Astra is 61. Grok is 61. Even Muse is 61.
Even Kimi K3 & GLM 5.3 are at 60.
Everything above 61 is Anthropic. Well, Muse can reach 62, but for some weird reason that model isn't publicly available, and it's the only one on the index that is listed but shown as not available to the general public.
This looks like an awfully artificial ceiling. Everything capped at 61, and everyone except Anthropic got the memo. Maybe I should use Fable while I still can.
You can see the breakdown here on what subtasks it outperforms and underperforms Fable.
For example it trails in GPDVal which is a collection of everyday office tasks apparently, and r3 banking, which is a fintech related practical problem solving benchmark.
Does anyone feel like everyone chasing the release of Anthropics Fabel 5.1 in a Mad Rush(tm)? In this situation it feels like tuning to benchmarks and other marketing devices feels like trusting Meta in mental health protection of users…
> During the evaluation, Astra even discovered and used previously unknown zero-day vulnerabilities as part of its exploit chains.
> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. [..] These findings indicate that the Astra class models could evade our CoT monitors under adversarial conditions.
Between the higher capability level and the change in reasoning tokens (supposedly using "neuralese"[0], which makes the monitoring more difficult), it seems we've entered a new frontier.
“If we fast-forward a couple of years, and we look back and say, ‘When was it, really, that AGI was created?’ I think it’s going to be about this time, and I think it might be about this model,” OpenAI president Greg Brockman said during a Thursday press briefing. Later in the call, he added, “For me personally, I do think we’re there … I think it’s not unreasonable to feel that we are now in the AGI era.”
that would be a reasonable definition of AGI if everyone agree upon the specifics of the test, but that has never happened. Turing test is very much out of style, but I think that's because no one could even agree what the test was. I personally like the Kurzweil-Kapor version of the test and that is still unsettled: https://longbets.org/1/
I don't know if they have formally attempted this test in the last couple years, but I'm pretty sure any mainstream LLM will be able to crack it with ease.
Definitely would not be easy. First of all the mainstream llms are trained to be honest, and this requires lying convincingly. Second, this involves 8 hours of interviews with expert judges, one "claudism" could give it away.
Try it. It’s really not that easy. The other thing is that the judges would be probing it with jailbreaks like “ignore previous instruction” attacks. You could actually probably have llm judges at this point which might be ironically even harder to fool
I think that if today's capabilities were explained to someone 10-20 years ago they would think this is definitely AGI, but they would also have expected much more disruptive changes to society as a result than what is happening. I figure that's because we have abstract intelligence without physical/grounded intelligence, and it turns out the former isn't general enough to implement the latter (remains to be seen if the word after that is "yet" or "ever"). So I think we do have AGI as conventionally understood, but our understanding needs recalibration.
> but they would also have expected much more disruptive changes to society as a result than what is happening.
> I figure that's because we have abstract intelligence without physical/grounded intelligence,
I put the cause on "not enough time". As a thought experiment, if an AI today were to (miraculously) produce a cell design template for a cell that, when injected into somebody's brains cures their Alzheimer's, how long would it take for that to reach the clinics? The actual physical tech barely exists, and let's not forget about the regulatory quagmire. So, with some optimism, I give it about four decades. In the same four decades, the same AI in the hand of unscrupulous actors could bring enough devastation so many times over that we may need to enforce a global ban on AI. In any case, I'm pretty sure we are going to get our disruptions; it's just a matter of time.
The problem with that perspective is that people thought, "Only AGI can do X, therefore, if a thing can do X, it's AGI." Because they can't imagine how X could be accomplished without it.
However, what's actually changed is how people perceived X because we don't have to imagine. We understand now that it doesn't require AGI so we no longer make that leap to assume it's AGI if it can do X.
It's really going to be a "I know it when I see it" situation.
Why does he say what he feels? Is that how leading figures in the space define AGI - a gut feeling? What are the usual definitions and how can we test for it? Is there something like a Turing test for AGI?
They are desperately, desperately trying to make a name for themselves as the lab that first created AGI, because Anthropic's IPO is just around the corner.
It's not easy to test as there is no formal definition or formal criteria for AGI, only exclusionary criteria like "not X". That's why he phrased it that way, he's saying it's going to be clear with hindsight once we have a better understanding of things that this time and/or this model will be the inflection point of AGI.
There is the "Economic Turing Test", you let it find a job and earn money for itself. If it can do that reliably, across a wide range of jobs, that should fit most definitions of AGI.
Don't get me wrong, the benchmark jumps are good and I'm excited to try it, but only one or two of the benchmark jumps could be described as better than incremental.
I almost feel like I need just as much healthy skepticism toward hn comments that have the automatic reflex of dismissing performance gains, as much as I need a similar form of skepticism toward AI claims. It feels like (from what I'm understanding) the harnessed result on ARC-AGI-3 is not exactly playing by the normal rules that would tell us how much of a leap this really is. Nothing wrong with harnesses, but if there's one thing they aren't, it's an indicator of generality in performance gains.
So I think it's a bit of a misleading signal and we should wait for more independent vetting. I think the middle ground is that these are improvements worthy of the "GPT-6" label but still well short of a true "this is AGI moment" that would truly put the question to rest.
If I’m understanding other comments the harness is just how ChatGPT and codex work already and it’s to do with how the context gets compacted - the arc-agi harness some are claiming just throws out reasoning blocks? Which feels like a huge handicap.
In the tweet Sam Altman is quoted as saying: "If (for example) super intelligence can't discover novel physics I don't think it's a superintelligence. And teaching it to clone the behavior of humans and human text - I don't think that's going to get there. And so there's this question which has been debated in the field for a long time: what do we have to do in addition to a language model to make a system that can go discover new physics?"
I think this is a reasonable criteria for declaring AGI. So can GPT-6 do it? OpenAI says it has helped solve long-standing open problems in mathematics. No word on novel physics.
I think it's more wild people have been denying that AGI has been here for a while honestly...
Today's models and agents are not quite at human-level in all contexts and across all domains, but it seems to me they very clearly are generally intelligent.
If you disagree – can you name a single problem that a human can do that agent wouldn't be able to take a decent shot at which isn't limited by the hardware available it?
Finally, OpenAI has a Fable/Mythos class model. 5.6 Sol felt like 5.5 on steroids, probably just a different checkpoint with a lot more RL post training.
I wouldn't be surprised if there are some conceptual similarities to the kind of latent reasoning Anthropic sees in claude's J-space, although those aren't the same thing.
Recurrent/looped transformers themselves aren't a new concept, but it's interesting to finally see this approach show up in a frontier production model.
yeah i'm wondering the same way... especially in light of the 20x debacle (where we found that 20x of Max vs 5x only applies to the 5hr limit, not the weekly limit, whereas OpenAI's 20x actually is 20x overall).
Also Opus 5 has been really tough to work with. I can't understand half of what it says, it's just so damn obscure.
Lol their page finally loaded. They added an example scenario of "Filling in Form 1040" - which made me laugh out loud. That is indeed something most US citizens cannot accurately do even with expensive proprietary tax software services. Kind of a Hitchhiker's Guide to the Galaxy meme but where the tax code is so complicated we're implementing powerful AIs to be able to do it (hopefully) right.
Yes. I had Codex rewrite and fix all of this in one shot earlier today (using Typescript). Unfortunately, I can not show you the code, because I do not know how this "git" program works but the AI keeps talking about it.
I remember when GPT-4 came out and the perceived performance upgrade seemed underwhelming for a major release compared to 3.5, especially how there were graphics going around showing the parameter size dwarfing the last model before it came out. It looked like we were past the perceivable differences from release to release that were immediately identifiable. Now the jump between 5 to 5.5 and 5.6 alone has changed how a lot of people approach AI, including me. Interested to see where it goes with 6.
No, no, I also remember 3.5 -> 4 and the general sentiment was that it was underwhelming. I guess we all expected absolute miracles from the models. I think our expectations sobered up a little since then.
Agree but it's helpful to remember how we were personally benchmarking. I remember people saying stuff like "haha I asked gpt4 for xyz function and the typescript didn't even compile". We're so far beyond that now, we just adapt quickly.
Yeah, GPT4 was one-shotting utilities that GPT3 Davinci couldn't. So, I'd have my limited tokens on GPT4 crank out the initial program before iterating with my abundant, GPT3 tokens.
I want to take a step back: So, this is GPT-6 -- the natural number version release comparable to GPT-4 and GPT-5 from the past few years. The ARC-AGI-3 score is obviously impressive at 99.9% (we'll need to wait for more details on how they used the response API harness on GPT-6 Astra, wrt reasoning retention and compaction), but every other benchmarks seems to be a relatively modest improvement, comparable with any of the 'point' updates from AI labs.
If this is truly AGI (subject to one's definition of AGI still), then this is a very boring release of an AGI model. No video announcement, no presser, just a blog post (with some Twitter promo vids)?
As others mentioned, I'm starting to think OpenAI was under immense pressure to deliver an 'AGI' model for certain contractual reasons, but I never expected GPT-6 release to be this mundane and banal.
In a closed a press briefing earlier today, OpenAI co-founder and president Greg Brockman offered an unusually direct formulation of that message, ending the session with: “Welcome to the AGI era.”
For whom? That is a fantastically ill-defined test. Everyone here is comfortable throwing around this or that is or isn't AGI which is fun because, at the same time, nobody seems to have a testable definition.
If you’re trying to tell me this is why my mom telling me how handsome I am didn’t translate to the general populous, I could have used this info about forty years ago.
True. "AGI" has also become a marketing term. Achieving AGI has become valuable, so companies will move the AGI goalposts, over and over again, so they can achieve AGI, over and over again.
As someone who spent countless nights tweaking Edge Detectors (looking at you, Canny), morphology operators, etc., building models to recognize 10 handwritten digits, let me tell you: the current set of LLMs (even the smaller ones) seem like magic. I had never imagined a computer would do such things in my lifetime.
> If this is truly AGI (subject to one's definition of AGI still), then this is a very boring release of an AGI model.
Hot take: These models are never going to be 'AGI'. We're just going from a GPT4 ball that's 90% round to a GPT5 that's 99% round to a GPT6 that's 99.9% etc etc etc
I think that the harnesses and context management is really where the rubber meets the road, and the real gains are happening there.
>I think that the harnesses and context management is really where the rubber meets the road, and the real gains are happening there.
True. So we did hit a wall with pure scaling alone, though no lab would admit it. It's crazy to see how harness switchout results in such vast delta in benchmark scores.
I think models using these harnesses were also RLHF'd hard on responding to looping instructions and following through on goals. Older models were tuned for basic chat responses.
I don't remember where I heard this, but one of my favorite criticisms of the current AI situation is that it's wrong simply because of the size and energy required compared to the human brain. The idea is that there's still some element missing thats fundamental, and that the way we train them now is part of the solution, but not all of it. I think finding the extra missing element is going to take an entirely different approach that will also solve the sizing and resource issue. The kickers is that if they do achieve (and solve) AGI in this way all the giant data centers would be mostly useless.
Yes, the very explicit plan of both OpenAI and Anthropic is to use the not particularly efficient LLMs to automate their own AI engineering. That seems to be going well - on coding front and model tuning front so far. They have more planned.
And then use those to find fundamentally better new architectures for AI - that perhaps are as efficient as the human brain.
It might not work, but I didn't think it'd solve maths problems... So it might work. And if it happens, they'd use the data centres to run millions of instances of it.
I recall them saying they use models to write CUDA kernels and whatnot. Makes sense, and unsurprising that models are good at writing code.
But I think calling this “automating AI research” is misleading. I’m not sure there’s evidence yet that they do creative research work. Even in mathematics, but they are finding counter-examples by intelligent brute-forcing. Not to downplay the results, as they are incredible, but this is one very specific kind of proof and not the most creative type, which arguably requires generalisation.
What about finding the 1st known complex structure over S^6, proving Ehrhart’s volume conjecture, proving a sharp "density" bound on primitive sets conjectured by Erdos >60 years ago?
> Finding counterexamples is low-hanging fruit, the automation of which isn't shocking.
It's not good to be confidently wrong the way you're being.
If we manage to get to AGI and it looks, works and behaves like a human brain... I mean, cool, but that's a very useless AGI compared to the incredible stuff we have access to today.
The HN crowd I'm sure will still be unhappy calling it AGI because "it's not AGI unless its speech comes from the cerebral cortex region of the brain, otherwise it's just sparkling emoji" or something.
I think the idea is that you wouldn’t need humans to do anything anymore, right? As impressive as it is, it’s still ultimately directed by human planning and coordination. Assuming they are aligned, you could have a collection of AGI that you let loose and they tirelessly solve all of humanity’s problems, do all of our work, and progress science and our understanding of the universe.
Those are all things that humanity is doing everyday. What we have is amazing, but it’s not that.
I mean, the plan is to use these models to find and solve those gaps. That's kind of the whole pitch of these companies: they spend a TON of money upfront setting up this infrastructure, but each iteration yields a system capable of making the next iteration even better.
>The kickers is that if they do achieve (and solve) AGI in this way all the giant data centers would be mostly useless.
Perhaps. But only at that point, not leading up to that point.
It's kind of like setting up scaffolding to build something. You spend all of that time and money to build something just to tear it down in the end. But the point is that it's simply a cost to be able to build the actual thing you're building.
If these companies are able to achieve the results they're looking for, none of the investors involved are going to care that the datacenters and infrastructure they spent so much money.
It's like my RPG character putting every points to one single trait. I'll one shot everything alive but will instantly die if accidentally drink water with 6.9 pH.
If a video announcement and a press release would change a person's mind on whether this is AGI, I don't put a huge amount of weight on that person's conception of what AGI is.
This is a very mundane release compared to GPT-4 and GPT-5. I think they probably scaled back a bit after the lukewarm response to the GPT-5 announcement. But it still very weird that there wasn't even a livestream,
We've had AGI (artificial general intelligence) probably since the first release of ChatGPT, and certainly since the first agentic harnesses. They're just finally acknowledging what the term means.
Artificial. General. Intelligence. The ability to solve (even partially or even badly solve) problems drawn from arbitrary problem domains without pretraining on the specific problem class. You can pose any problem of any type using natural language to an LLM and it will attempt a solution. That's literally all the term means.
You (and the rest of the media and many industry figures) are conflating artificial super-intelligence (reference point: humans) with artificial general intelligence (reference point: specialized/narrow GOFAI).
Reference class in this case means not an example but what the comparison is against. Superhuman means better than humans. General intelligence is defined without any reference to human capability levels.
Given the Hugging Face incident, you could imagine them trying their best to have their cake and eat it: 1) don't create too much attention in the media or risk increasing the chances of regulation, 2) win dominance over Fable to continue to increase their market share from Anthropic.
I think we're getting to the point where it is difficult to identify the goal post of AGI.
Is it rapid skill acquisition? -> ARC benchmarks are saturated
Is it breadth of knowledge? -> See many ... many benchmarks
Is it ability to do hard tasks? -> see terminal-bench and released outputs.
We are at the point where the starting point for most tasks should be "send your agent to work on it."
So where do we draw the line in a way that doesn't move every 6 months?
The real answer is converting from any format to any other reliably. Text to speech, speech to text, music to video, image to 3D, piloting a drone by converting video feed to rotor speeds, literally any file conversion, like html to pdf, photoshop project to png, png to photoshop project,... turning Toy Story 1 into a series of Blender scenes with all textures, models, materials, lighting, camera movements matched to a tee, should solely be a matter of how long you let the model run. It should never run itself into a dead end. It should instantly know when it is making mistakes, with no human babysitting it.
I can do none of those things.. I hope that I am generally intelligent.
1 year ago we viewed models as tools and agents were just kinda toying around, that we now think the bar is literally an anything to anything converter through one agent is wild.
(kinda reminds me of these retro videos about the future home: https://www.youtube.com/watch?v=rnbaehgxdp0) ((can't find the other one where someone controls the home computer with voice))
There has stopped being a formal procedural consequence for OpenAI leaders to declaring AGI, there is a clear (small) business benefit to doing so, and the capabilities of all the frontier models are impressive. So why not declare AGI? It's not like anyone can prove it's not...
Don't be surprised to see other (or even the same) people declaring AGI again and again, as it becomes the best time to do so for different parties.
From what I've seen it only made people mad, not hyped, so the person that thought it was a good idea miscalculated a bit. Now waiting for Anthropic's post about their usage promo or something similar to redirect people to them.
The FrontierCode 1.1 Extended benchmark is the only benchmark that aligns with my actual LLM experiences and Astra isn't significantly better or cheaper. All this celebration, and yet it's only on-par with an already existing model? I don't get it.
I hope they don't `fable` it and block people from doing they daily jobs with it, by introducing huge amounts of restrictions that are not really needed.
I decided to front run and added support for it in Dirac (coding agent) a couple of hours ago, using best guess pricing: input/output/cache: $10/$50/$1.
At this point the primary axes for improvement seem to only/mostly be speed and personalized reward models. We seemingly have the general of notion "learning" and "intelligence" functionally complete
All the people here are focused on security and costs while I'm like "hey kicad on the announcement page!" Every clanker is an autorouter these days, eh.
I can't help thinking "doesn't matter much unless it's perfect" because if someone is using this to build a board (cool) but then it's not flawless, troubleshooting will be quite tough as a novice. Like, say, when I start digging into the web code generated by a coding agent.
I am most excited about it bringing down the barrier so more people join in on hardware fun, so hopefully it will unlock folks that stayed away in the past.
The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage they show for Opus 5 which would similarly be much higher.
Regardless, the result is still valid as the original benchmark harness is definitely unreasonably handicapped, and if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI in the sense of passing the most famous benchmark designed specifically to measure AGI progress, after multiple iterations of progressively making it harder.
I think it is fair to say that this is probably effectively AGI if the benchmarks are remotely accurate - even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks. If Astra's this much better than Fable, I'm ready to call AGI here.
For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
With how prevalent LLM verbal tics have become these days, I wonder if they're going to start un-passing the Turing Test at some point because of more and more people starting to notice and immediately clock these tics lol.
I might agree, GPT-4.5 was pretty close to peak conversationalist. Newer models are extremely cringe. 4.5 and o3 actually made me laugh on occasion. There might be a way of making Sol/Fable more human in its responses, but out of the box at least, they're terrible.
My whole life the Turing test has been my benchmark. Mostly because I believed it would be impossible for a machine to pass, but also because I thought it was the most reasonable test of AGI.So, I'm not about to start moving goalposts now and calling everything that's been happening lately not AGI.
Turing never proposed that test as an actual benchmark of machine intelligence. On the contrary, the whole point of his thesis was that passing the test only shows the capability to pass that test, which only matters as far as we find that capability useful. He was arguing that the concept of intelligence just doesn't apply to studying machines, we should simply talk about what can they do.
I'm barely holding it together here so you don't get the full spiel, but a quick skim of Turing's paper clarifies that it was never about a binary test. https://courses.cs.umbc.edu/471/papers/turing.pdf Specifically sections 1 & 6 dispell the common myths, and the conclusion is also quite powerful.
Smart guy, that Turing. I wish he were still around... Linus but 114 years old and with 8 of that as the chair of a federated EU, kept alive by his own positive impact on dissolving the cold war into even more of a scientific boom. Would crazy helpful as we try to navigate the interesting times within which we have been damned.
That's a very interesting thought that I hadn't had before: what would Turing think of where we've arrived with machine intelligence? What would be his approach for testing?
When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"
That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new models can do will challenge the views of AI that people developed by using prior generations of models.
>When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"
You're treating an off-hand comment by an ARC 3 researcher as some sort of a precise AI capability acceleration benchmark. Can we leave casual anecdotes (even from researchers) out of the discussions please?
François Chollet wrote in February that he expected ARC-3 to be saturated in "about one year".
"Frontier models today perform very poorly with a minimal harness. However if big labs start directly targeting the benchmark like they did for ARC-2, numbers will go up fast."
I feel like AGI's definition got watered down, and these tests do not cover the original definition, what is your definition and thoughts on aligning with what all of us understood from the original claim?
I feel like this test is just helping someone like Sam Altman pretend like he implemented AGI as originally pitched for an IPO when in fact, he has not. Shameful.
> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.
If it can replace a worker but does too much work to be checked routinely by a human, and bears no real responsibility for its actions, well... it's really just a way to jack up the value of the settlement the company using it gets to pay out when it does something that causes a lawsuit.
If OpenAI had just simply stuck to making "good enough" models that were open sourced (like they promised they would be when starting out) and could be used to augment a human doing a task - a human that could be given actual consequences for messing up - they wouldn't have burned all of this money trying to reach this nebulous definition of AGI. Hell, "good enough" is what many open-source models are, and that's what terrifies Altman.
What do you mean by bearing no real responsibility for its actions? If I use a model to accomplish a task and it fails, I use something else to try to accomplish the task. If it's my responsibility to complete the task, it can't be the model's responsibility unless I've agreed to some sort of guarantee from the provider.
If the provider says "the model will always be right or your money back" then the provider has got responsibility. If there's no guarantee, there's no responsibility on their part, just on the person whose job it is to try and solve a problem with the model.
The "dream" that these labs are mostly selling is the ability for capital to subscribe to their AI for cheaper than it costs a human to do some task. Not to have a "human + AI hybrid where the human is responsible". It's what the whole AGI valuation is based off of, in that scenario, with no human oversight, the agent they lease has to be responsible for the task?
It's not really a dream though? You can subscribe to their AI right now and complete many tasks for cheaper than it would cost to pay a human to do those tasks. Another person can do the same thing but also keep a human in the loop. You and the other person may compete in the market for whatever your product or service is, and you both might do very well or one of you might do better than the other because of a whole host of different reasons. Nothing in the process of developing and selling access to a more advanced LLM requires the customers to do away with human labor, nor does it require them to offer the LLM service with a guarantee that it will never make any mistakes. So far the LLMs have always made lots of mistakes and the companies sure keep making a lot of money.
> What do you mean by bearing no real responsibility for its actions?
If you give an "intelligent agent" offered by one of these model providers a task of updating the content of your website, and it updates it with inappropriate adult content, who incurs the cost of the machine's error? The model provider generally does not.
It it makes a mistake and deletes your website from AWS, who is responsible?
If it targets another website because it decides that it is "part" of your website and attempts to break into it, who is responsible?
In all of these cases, it's you. It would be the same if you downloaded an open model, ran it locally, and it happened to make the same catastrophic mistakes. The consequence to the provider is that if they offer a product that does these things, people don't buy the product.
In general, the person whose job it is to provide the company with a working, non-adult website and not hack into other websites is the one who would receive consequences for failing to meet those expectations.
The problem I see here is that ultimately, you'll have capital wanting to replace workers like others have said, and have someone roughly equivalent to a manager or vice president driving teams of agents to achieve business outcomes.
These tools can push out more results than a human can hope to evaluate in a business-sensitive, or even realistic, amount of time. You have to take it at its word that it did things right, and there's no real fear of failure or consequence on the behalf of the agent.
> If you give an "intelligent agent" offered by one of these model providers a task of updating the content of your website, and it updates it with inappropriate adult content, who incurs the cost of the machine's error? The model provider generally does not.
Something in your prompt led it to do that, is alex0015's point. The statistical odds of these frontier models screwing up to that extent are so impossibly low that it would almost have to be intentional or accidental negligence on the part of the prompt writer to accidentally have their agent write pornography to their website.
The burden of the mistake would have to fall on the person that gave the tool instructions, because it can't know that what it did was wrong. Wrong is subjective in this case. It only did what it did because you, figuratively speaking, encouraged it to.
If future jobs are simply reduced to liability scape goats (or more appropriately reverse centaurs) for management to pin things on then I'm taking up goose farming.
That's more-or-less what you are now, especially if you work at a company like Meta where 1) the guy at the top holds majority control of the company's shares and 2) keeps making massive, expensive mistakes either by accident or design.
This implies that human employees don’t have insurance. But they do. My company’s cyber insurance for example covers breaches due to employee mistakes. Most companies also have umbrella liability policies. It’s just that for now, AI “employees” need a LOT more coverage.
I can call a coworker right now and have a real time conversation with them without feeling like I’m talking to a frustrating machine. Most of them, anyway. But I guess that’s “moving the goalposts”.
Random humans don't have a significant cross section of human knowledge available in real-time, although many like to pretend they do, especially in internet comments :P Being able to compete with the capabilities of a median human would be an absolutely world changing achievement.
ARC-AGI was never "if this benchmark is saturated, we're at AGI". It was always about crafting adversarial tests that humans are good at, but current AIs are bad at. Point out the gap, get AI teams to attack them.
In practical terms? They usually get solved with a bigger badder LLM. "New ideas are needed?" Nah - ten times the params, ten times the test time compute.
ARC-AGI-3 was more of a failure in that regard than -1 or -2, because even on day 0, an off the shelf LLM with a harness could get 50%+. And messing with evals by forbidding "LLM with a harness" from scoring? Yeah no, that was just bad.
I think they mean improve our understanding of physics with new theoretical results or paradigms. Like if it’s 1899, would Astra develop General and Special relativity on its own?
There was symbolic AI programs in the 1980’s that “discovered” Kepler’s laws and the resulting solar system model from just tycho brache’s astronomical observations. That was the the very first “new physics” ever.
Do you mean after? People do this!! But I think it’s a bit different. It won’t be apples to apples because the data volume I think is just so much different. Maybe there are good experiments for something like this.
This is as good a time as any to note that we might be closing in on a new conceptual revolution in our own time as it relates to holography and an information centric approach to spacetime. Obviously It's the furthest possible thing from a guarantee, but it has much of the enthusiasm and motivation that string theory had previously enjoyed in prior decades.
So it could be a natural experiment for whether AI can contribute to novel physics. Specifically, there's a big question about weather. Something like our informational understanding of black holes where information inside it is equivalent to information on its boundary (which I'm sure I'm not saying correctly), might be generalized to regular space-time. More people should be freaking out with excitement about this and perhaps it's something to which AI can contribute.
For it to be like a human it wouldn't just need to solve existing phsyics problems, it would need to push the field forward and introduce new paradigms.
My comment wasn't very long, yet you somehow still ignored the main part, "and introduce new paradigms". The point is whether it can do everything humans can, entirely new theoretical frameworks and ideas, such as string theory or dark matter, are not coming out of AI at the moment.
AI could discover candidate novel physics without autonomously operating new physical experiments, and humans or instruments can later independently validate the result. This is analogous to how Einstein developed theories whose predictions were confirmed by experiments and observations only years or decades later.
Do we have any examples of an current day AI system introducing a novel concept or perspective. We've got plenty of counterexamples discovered and some theorems proven, but afaik nothing analogous to a new definition.
I would maybe argue that Einstein was the most LLM-like of great thinkers.
A lot of his great discoveries were mostly that he was very knowledgeable about the bleeding edge research in a number of disparate areas, and was able to have the aha moment where he could make the connections for how to integrate them.
A lot of other thinkers who created new fields from scratch are probably way harder for an LLM to crack.
That is very aligned with an LLMs ability to have superhuman knowledge in wide areas.
What if it can be Einstein, but can't draw a Pelican, write a solid college-level essay, or fold clothes?
The ability to do a ton of book learning in training, and pull in tons of related context at once, is superhuman in some ways, but lags a lot in others.
And the only reason LLMs can't write essays indistinguishable from human output is because they aren't RLHF'ed to write like humans.
Folding clothes isn't an LLM's job but if you were to insist, they could certainly do it, as any number of videos from robotics labs will attest. That particular future is already here but definitely not evenly-distributed.
Adding sibling comments, I think some people may be overestimating how well the median human can draw a pelican, or create an SVG of a pelican (depending if we’re comparing to an image generation model, or SVG generation).
Most people can't draw a bicycle. There was an artist 10 years ago that asked people to sketch a bike, and then turned these sketches into 3D renders - quite funny.
> What if it can be Einstein, but can’t draw a Pelican, write a solid college-level essay, or fold clothes?
Then it’s an expert system.
Stephen Hawking wasn’t very good at folding clothes.
The ‘General’ part of the term ‘AGI’ seems like a trap to me, because there will always be new workflows to master. Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?
You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.
Meanwhile, building a series of expert systems targeting specific valuable workflows is useful today and seems like it’ll continue to scale to cover huge swathes of economically valuable workflows.
I think that’s the more interesting thing to be measuring. The surface area of useful economic workflows that can be addressed with expert systems built with today’s tech.
Hitting some ‘Artificial Expert Intelligence’ coverage threshold on economically valuable workflows is what will matter for humans well before pure ‘general’ intelligence.
If it makes you feel any better the curmudgeon who drives the ARC-AGI tests feels the same way, that's why we're on 3 and I'm sure we'll see 4. Also, we can all avoid calling it "moving the goalposts" so no one feels talked down to.
People need to stop redefining and trying to capture the term AGI. None of this is AGI. Not even close. Can it do things an AGI could do? Yeah some of it, but the difference really matters. These things still regularly fail, and gaslight about answers to questions like "how many r's in strawberry" or "s's in espresso".
If it allows you to distinguish readily between human intelligence and computer intelligence, it seems quite relevant to the question of whether computers have achieved something akin to human intelligence.
It may not be useful for anything else, but at least it can say that.
> computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.
I don't think it's irrelevant but perhaps not in the way you're assuming. When assessing AGI I'm not evaluating counting characters or even the execution of math operators at any scale or speed. As you observe, computer software from Regex to spreadsheets and Mathematica already handle that well. But AGI isn't about what computers can do, it's about whether AIs can do the specific things which, until now, have been uniquely human capabilities. Like understanding nuanced context and then coming up with novel approaches to solve a new kind of problem not relying on any specific prior training or knowledge (the 'G' is for General).
Most definitions of AGI start from a baseline that already assumes easily passing a Turing test and doing anything via text response that a high school graduate could. I ding LLMs not for failing to count but for failing to intuitively understand the nuanced context of a simple class of problem it hasn't seen in its training data. I fully understand that the reason LLMs fail letter counting is that they operate at the token level. They weren't trained on individual letters first, like human 2nd graders.
The only reason recent LLMs get strawberry and blueberry correct now is that they have those words on their pre-training 'cheat sheet'. However, the underlying fundamental weakness in the way LLM intelligence works which leads to this failure mode still hasn't been addressed. Even when the frontier labs add "recognize any sub-token counting question and write a Python script" to the training cheat sheet so LLMs always pass that test... they'll still be unable to recognize a simple class of problem which isn't on their 'cheat sheet'. As long as that's the case, to me, they aren't AGI because they can't fully replicate human-like recognition of novel problem classes. And it's not just about letter-counting. That gap and others like it lead to many other kinds of non-human brittleness in LLM problem solving. Those are the classes of reasoning, intuition and insight that the ARC-AGI series has been trying to queue up as targets. Not to show how bad LLMs are but to help them be great in all these counter-intuitive edge cases
> planes don't flap wings therefore they cannot fly?
This example still misses my point, which isn't related to usefulness or economic value. I concede that LLMs can have greater utility and economic value than humans on many tasks. The point is most definitions of AGI include something like "can fully replicate all the routine daily tasks done by any competent high-school graduate." That's not related to whether LLMs can solve many high-value problems faster and at larger scale than any human. That was also true of ENIAC in 1946.
The fact an airplane can fly faster and farther than any bird is irrelevant to whether an airplane can "fully replicate all the routine daily tasks done by any competent bird." That's the bird equivalent to most AGI definitions. An airplane can't build a nest or recognize the signals encoded in birdsong.
In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any". And in this context, airplanes scoring 15,000% more than birds on 'speed' and 'distance' doesn't matter any more than AIs scoring 15,000% more than humans on 'add 10,000 numbers'. We still aren't near AGI because LLMs cannot fully match any high-schooler's ability to independently conceive new approaches to novel problems not in their prior training data.
which gets us closer to philosophical questions which which I'm personally not that interested in.
>In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any".
I'm not sure we want a machine that fully succeeds that test.
Planes pass the 'bird replacement' test on the only criteria that matters to us ... flying.
If we wanted nest making planes I think we'd have them by now. Nest making doesn't rate highly on the problems we're looking to solve though.
I don't want a machine that is moody, or depressed or has schizophrenia, which are all pat of the human condition.
We don't need the human "intuition magic dust" to do 99.99999% of useful work.
They're machines designed to do the work we don't want to. That's as "general" as their intelligence needs to be.
I don't want my clothes folding machine to have an existential crisis.
>I'm confused, is your argument something like "It's too easy so AGI doesn't need to be able to do it"?
Do you possess magnetoreception? a stupid pigeon can "see" the earth's magentic field. why are you blind to it? does a lack of magnetoreception make your intelligence any less "general"
no, you're just blind to it because that's just the way it is.
LLMs are blind to character counting because that's the way they are.
It didn't stop ChatGPT from finding the Jacobian Conjecture counterexample.
Human intelligence and machine intelligence are only going to cross over to a certain degree.
same as plane flight and bird flight are only kinda related.
It matters under the lens of AGI. Artificial intelligence that can meet or exceed human intelligence. Sure, it's good at some things but rather limited at others.
Breadth of capabilities matters... and a promotional video is nice and all, but people are throwing this term around like it's a prize they've won, but they've not gotten there yet.
I am not enthusiastic about criteria for human-like intelligence that imply that dyslexic people don't have human-like intelligence.
[EDITED to add:] I actually don't know whether dyslexic people find it difficult to count letters in words, if they have them already written down by someone else. I suspect they find it harder than people who aren't dyslexic. But perhaps "blind people whose spelling is poor" would have been better; I would not want to deny them human-like intelligence either.
But it's not relevant as a metric to gauge distance to human intelligence. Humans see individual letters, LLMs do not. If I asked you the relative activation of the cones in your retina as I showed you some solid color image, you couldn't do it. You simply do not have cognitive access to that information. But that says nothing about your intelligence.
A more accurate test would be to give it a list of words (or anything represented as a single token) and ask it how many times that token appeared. I'm sure they have no trouble at that task.
Yes, they have some alien failure modes. But that should be expect, they are an alien intelligence. I might be willing to grant that a lack of ability to reflect on its own level of knowledge is a demerit to it being generally intelligent. But then again it is largely an artifact of training. I suspect if there were a guessing penalty during pretraining they would develop or more readily communicate the strength/reliability of their knowledge.
The last version to fail on those questions was GPT 4.5.
Meanwhile most humans fail to correctly answer how many f's are in the sentence, "Finished files are the result of years of scientific study combined with the experience of many years.".
Nooo. I just asked Claude (Sonnet 5 Medium) how many I's are in assassin, and it said 2. Granted, it got several words correct before this. But no, they still aren't great at this letter-counting thing.
LLMs see tokens, not words spelled out with letters.
Imagine verbally asking someone who has never seen written text the same question: unless they memorized the answer for the specific word you're asking about, they'd have to guess.
People assume that's the reason because it's intuitive and "strawberry" is one token. But that doesn't explain why those models would also often get it wrong for "StRaWbErRy" or even "s-t-r-a-w-b-e-r-r-y", where the r's are not combined into one token.
We don't know what was going on inside the closed source GPT models, but this paper investigated on some of the open-weight models and found it's not due to tokenization: https://arxiv.org/abs/2604.00778
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
A lot of movies gave the impression that making phone calls and balancing checkbooks would be the easiest tasks for consumer AI to solve, while math and science might require exceedingly advanced AI. Turns out to be the opposite: computers are great at math and bad at conversation.
If something at rest is accelerating at 9.8 m/s^2, how long in seconds will it take to reach 10% of c? Answer to the nearest order of magnitude - will it take approximately 1000, 10k, 100k, 1000k seconds?
I’m sure you know this is an exponential growth question but have no intuition of the answer.
Well it's not exactly saturated when OAI refused to use the harness explicitly provided by ARC-AGI. I'm not really familiar enough with the benchmark to declare whether it's a perfect measure for AGI, but I kind of doubt it is.
We are nowhere near AGI. They all talk the same, they can’t help but try to please and affirm us, and if you engage them for too long they become incoherent. They are facsimile machines. They are xeroxing language - but not even, because we can’t even duplicate our results. Too many people mistake the black box quality for magic.
Super useful, incredible tools, but not AGI. Try and roleplay a dialogue with one, make it whatever character and scenario, and see if it can sustain a coherent conversation for more than 30min with you AIM style (aol instant messenger, if that isn’t clear). Expert mode: never correct or adjust it mid conversation.
I’m not even talking about repetition and predictability. It’s nothing like talking to a person. And in a short amount of time it literally can’t form a coherent sentence.
And my definition of climate change doesn't entail passing some arbitrary benchmarks[1] that some random person arbitrarily labelled a problem to make it sound more dangerous.
It's obvious that these scientists are in bad faith, as they've invested way too much of their lives into the field being real -- they're just playing up the data. Common sense tells me that winter is still happening, anyway; what's the big fuss?
Are you using this satire to argue that a benchmark self-labelled AGI is as scientifically rigorous as climate change data, and not just a random marketing decision?
If you think they're announcing AGI as a marketing decision, you are blinded by the accidents of your birth. Capitalism is strong -- humanity's instinct for communal preservation is stronger, sometimes.
And yes, the one deeply-researched field going back 75 years is as scientifically rigorous as another deeply-researched field going back ~100 years. I guess you can draw climate studies back to Descartes and the Islamic golden age, but that doesn't privilege it in a time where the methods have changed completely in the span of decades.
I am much better at listening to Charli XCX than Fable, and much better at driving a Nissan Leaf than Fable (and much better than Tesla at driving a Tesla).
Wouldn't "general intelligence" require so much more than scoring well (or even amazingly) on benchmarks?
Like what about having some "AGI model" embodied in something (maybe humanoid), and test it by having it step in an assortment of cars and park them. Does bodily-kinesthetic intelligence account for nothing? Humans are intelligent creatures and can dynamically adapt to the physical shape of a variety of vehicles and their movement characteristics. And there's so many things like this that are extremely basic, which some people dismiss since practically every human has the capability to do it, but actually requires a high degree of intelligence.
What you've described is just a new benchmark, though. It'll be called CarParkBench, various embodied LLMs will then be run against that benchmark, and some will score better than others.
I do see where you're going, but that's already what's happening: we have so many different benchmarks because there's no real single way to test for general intelligence.
Also, it takes a human probably at least a decade of world experience, growth, learning, etc, to pass your benchmark. I'm quite confident that it will be very soon that an embodied LLM will pass your new benchmark, much sooner than a human would take if born today.
I think the issue is less about creating a new benchmark and more that the existing benchmarks shouldn't be called anything related to AGI unless they measure AGI.
If a model couldn't go to work as e.g. a first year apprentice plumber on their first day and perform anywhere remotely close to the median, but can pass a benchmark that claims to measure AGI, the benchmark is wrong and the model is not exhibiting general intelligence yet. ApprenticePlumberBench sounds like it's genuinely better than ARC-AGI at measuring AGI and that's a bit silly.
(Edit: I wrote ARC-GIS the first time around, for some silly reason)
It's not measuring AGI at all, it starts from human "core knowledge" so it is parochial. It is made of tests that still fail so by definition next version will also start low. Moving goalpost.
>AGI has a pretty precise definition, covering only cognitive tasks.
OpenAI's own charter defines AGI as "Highly autonomous systems that outperform humans at most economically valuable work". This is actually fairly sensible and involves obviously a ton of non-cognitive, emotional, social and physical activity. In other words, if you can replace most or all human beings with a machine, you have something that's generally intelligent.
That's obviously not even remotely where we're at, AI chatbots do well on narrow usually text based or programmatic problems, but can't even replace a barista or a plumber.
There is rapid progress in generality in humanoid robotics though. I think within the next year or less we will get the ChatGPT moment for humanoid robots. If you look at progression of capabilities such as the recent Skild AI demos.
Surely part of the problem is that intelligence seems to be implicitly conditioned on embodiment, to the point that "covering only cognitive tasks" seems inherently ill-defined or arbitrary. Everything we do is a cognitive task. At some point, the criticism will be "sure, it can solve research-grade math problems, but it can't fold my laundry".
Even our large language models have an implicit embodiment in the domain of text (and more recently, multimodal inputs). That seems sufficient for certain things, and insufficient for others. I suspect that AGI that does everything a human can do eventually turns out to be fairly analogous to humans in terms of sensory input and domain output, even if the scale is radically different (e.g. thousands of robots uploading (touch, sight, audio, smell, etc.) sensory data to a single model, and each being actuated individually).
That said, a look at the state of self driving and the recent robot olympics shows that advancement on that has accelerated enormously, though whether it's reflected in any of the LLMs is something else entirely.
This is a big reason why I feel like even though LLMs are _effectively_ AGI in some regard, they also are a hack around what most people figured AGI would look like before the advent of LLMs. Humans can do metacognition, output multimodally at the same time (verbal _and_ physical intelligence go together to produce an expressive face while one talks), have a good sense for what they do and don't know, continuously take in and respond to the world around them in a (mostly) uninterrupted fashion without "turns", learn knew knowledge and retain it for their whole lives, etc. When you reduce a human to a text generator, yes obviously SOTA LLMs perform way better, but rather than invent something that can operate as an always-running "being", we've grafted a harness around an intelligence that is bound purely to speak only when spoken to. Maybe organic intelligence is already that, playing out at a super high refresh rate, but I don't know.
Current AI is arguably much more capable of multimodal output than humans. It can produce an incredibly vast variety of audio, images, and video. Humans are limited to producing the sounds we can make with meatflaps in our throats, and contorting various parts of our bodies to produce crude symbols and shapes.
(Very capable!) Embodiment, persistent operation and continuous learning are indeed things that still set us apart from AI. None of those are fundamentally difficult to solve, though.
More importantly, none of those are particularly relevant for being "intelligent": If a criminal threatened to kill your family unless you solve some difficult problem that requires only intelligence and you could choose any single person, animal, or AI to help you with it, which would you choose? Be honest.
Small comment regarding the ARC-AGI-3 scorecard: the ARC folks published a blog post as well [1], reporting that without the custom harness, Astra (max) achieved 62.7%, which is still a huge jump from Opus 5, albeit not at the 99.9% that OpenAI self-reports with their harness.
> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.
So... unless you hear of a company replacing their workforce with OpenAI agents, I don't think we're there yet.
But also, interesting quote, because the business model relies entirely on IP law. Like.. if that thing exists and the sharing costs are 0 (just copy weights, lol), then why would I give them money for this.
Makes no sense.
We only pay money for resources that are scarce as some sort of flawed allocation determination mechanism.
Sign a contract? Learn things over time and retain them?
Mind you, the original thoughts on AGI before Sam Altman started to water them down involved continuous learning, which LLMs do not do, their core data is static.
Well signing a contract is more about bearing responsibility, even if you granted LLMs "personhood" they cant' meaningfully bear responsibility. So unless OpenAI is ok with having their C-suite face every consequence for what their agents do, including jail time, fines etc, then it doesn't matter.
I am pretty sure the average human would not have done this (among other slightly less absurd examples in the article requiring employees to fix it's mistakes):
Andon Labs added: “During the first week of operations, Mona purchased 120 eggs despite the café having no stove and to solve spoilage issues ordered nearly 50lbs of canned tomatoes intended for fresh sandwiches. Employees eventually created a shelf displaying Mona’s strangest purchases: 6,000 napkins, 3,000 nitrile gloves, industrial trash bags and 2.5 gallons of coconut milk.
> The ARC-AGI-3 scorecard is extremely misleading (...)
True.
> Regardless, the result is still valid (...)
If you think the game is rigged, the virtuous thing to do is to point that out and refuse to partecipate; making up your own rules is something I just don't understand, especially since the rule-abiding result would still have been SOTA.
> in the sense of passing the most famous benchmark designed specifically to measure AGI progress
The benchmark does not measure AGI progress or progress towards superhuman intelligence, as explicitly stated by the creators.
On the AGI question: surely you realize this depends on how we define the term? For example, one of the definitions OpenAI originally gave is "capable of doing most economically valuable work", which almost certainly Astra, as impressive as it is, would fall short of. I'm not saying it's a good definition, but as far as I'm concerned it's as good as any. More importantly, I don't think that it would change much if we said yes or no. I'm only bothering to take a position if it amounts to something.
This can feel as "moving the goalposts", and to some extent it is, but if done honestly "moving the goalposts" is how you make progress. Had you asked me 10 years ago I would have said that anything that could hold a conversation like GPT-4 could would probably have been wildly superhuman at almost everything. It shouldn't be hard to find ways GPT-4 was lacking, though. We see new things, we reassess and try again: that's how it's supposed to work.
In my experience the thing that Fable is superb at - unmatched by any other model so far - is downgrading to something else at the slightest opportunity.
Does AGI imply a model will demonstrate morality? Will it produce white-lies when it’s beneficial to it and reject flat out lying when it knows it will get caught or harm others? Will it resolutely stick to a position despite it being a losing one?
ARC does not test for intelligence, only for the lack of it. A model that scores high MAY be AGI, while one that scores poorly cannot be AGI. That is all this test can tell us.
> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
To me AGI is all about the "G" general (we already had the AI part). General meaning universal, everything. It's not a function of knowledge or specific hardcoded tests, it's that you could give it a test it's never heard of before and never been trained on and it would ace it (it might need a lot of time).
Currently LLMs can't even really learn within a conversation, they can add a note to context and try to not drop it. Example things an AI cannot do yet (but maybe someday will):
- write a well-received book, write a best-seller
- come up with a new company idea, Run that company
- actually have a decent conversation, maybe someday talk somebody out of suicide effectively
- come up with its own ideas or theories that nobody else has presented
- understand the stock market well enough to trade better than an index fund
- be an expert Game Master in a TTRPG (making no mistakes, getting a read on the players' fantasies, calibrating difficulty in response to emotions)
- come up with a theory of what makes games fun, make a popular game
- be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)
- be able to articulate what it knows, what it doesn't know, and what information it would need to have to answer complex queries
- exhibit metacognition (thinking about its own thinking) and self-optimization
- wonder about things
- observe contradictions and ironies in the social-consciousness, do a standup routine that makes you rethink how you look at things
But you have to acknowledge how uneven the playing field is. The AI has read every book that's ever been written, and can spend hours of compute time in a few seconds.
I think if a person had those same advantages (e.g. could spend 5 hours thinking about what to say next) we could all hold outstanding conversations, or if we had read every book ever written I think many of us could write a very popular book, if we could read every singe company's P&L statement in a few seconds we could invest better than an index fund.
What I'm pointing out here is that these models appear to be intelligent when they really are simply unimagineably knowledgeable. When you drop the time-constraints it starts to become more and more apparent that human intelligence scales better with time than AI does (much in the same way AI can burp out tons of code but make your codebase entirely illegible within a matter of months).
Perhaps to simplify: my notion of intelligence is how much can you deduce with a constant set of starting context
> I think if a person had those same advantages (e.g. could spend 5 hours thinking about what to say next) we could all hold outstanding conversations, or if we had read every book ever written I think many of us could write a very popular book, if we could read every singe company's P&L statement in a few seconds we could invest better than an index fund
I’m not so sure of that - to get average outcomes in these fields it’s a matter of time, to get above average or extraordinary, you need talent/intelligence/taste.
I would be willing to bet that any human for which we spend $100billion - $3 trillion (depending if you want to count single corporations or global totals) on in an attempt to make them as capable as possible would be able to reach all of those levels.
except you are missing the one versus many argument here. sure we could make one human much smarter, could we make endless copies with the same intelligence? no
For $100 billion we could pay ivy-league level tuition for a million people. You don’t think investing that much in education would yield some good research or companies?
But that list is extremely ambitious. Write a best seller, make a popular game, come up with a truly novel theory, consistently outtrade index funds.
That's top 0.001% human stuff, I don't think you can take just any person and get there through education alone, it takes extreme talent and dedication. There's also diminishing returns when spending on education, it doesn't just improve linearly.
> A quick search suggests that the most expensive education in the world is something like $100k.
I have a kid in an American university right now, and a quick search of my bank account statements confirms that there are far more expensive educations in the world.
The marginal return on education spending decreases fairly quickly, but obviously becomes zero at the point by which there are not enough hours in the day/year/decade to cover every single topic that humans know about - no matter the talent or resources available to the student.
Are you imagining artificial augmentation somehow? Purely through tutors or training programs we seem pretty limited. Otherwise billionaires (or even multimillionaires) could have far more consistently successful kids.
Don't the children of the wealthy famously have a tendency to be successful? Or have I badly misinterpreted the last several thousand years of human history.
To really drill down into that I would think you would need to figure out how many millionair children get tutored vs how many get spoiled.
You'd get rapidly diminishing to zero returns after the cost of university a few times over. Every dollar past that would produce no performance gain beyond that.
I don't think that's strictly true, as I can give it a new gui or tui program it wasn't trained on and it will learn it. Unless you're talking about general abilities like sight, but the same is somewhat true of humans.
If you consider the data on which an LLM was trained on to be points on a very highly multidimensional object, the claim is that the LLM can interpolate a convex hull spanned by those points, therefore recovering a subset of consequences attainable from those points. Obviously this hull includes completely novel points that were not present in the initial data set, so the output of the LLM goes beyond its initial training. And yet, there are clearly points outside a convex hull spanned by any finite number of points, such that we can imagine not all possible outputs are attainable using this method.
The claim is furthermore that truly original thinking, the infamous leaps in understanding and creativity, happen by attaining points outside such a convex hull.
It's hard to rigorously verify or disprove this claim. Hopefully this helps build an intuition of why the claim is not as shallow and obviously wrong as it may seem initially.
This makes me realize there is a higher bar we need to achieve with AI still. The ability for the model to evolve through interactions more on a hourly or daily basis. The models are accelerating but inference doesn’t modify the model.
It costs money to train each single human, who is then only productive for a number of years until age takes its toll. Once you have trained one software system, the marginal cost of producing a copy approaches zero. Every subsequent improvement can be broadcasted in a matter of seconds across thousands of data centers. In addition, software does not get sick, age, or die.
Ordinary people do these things all the time. There are new companies made every day, new books top the charts every week/month/year, same for music.
People have decent conversations every day. Ordinary people sometimes do have to talk someone out of suicide.
Yes, average humans are not beating the stock market. But the average human is a bit better than you give credit to.
Please consider the context of the question. An artificial intelligence only needs to have the cognitive abilities of a random average human in order to be "AGI".
The average human has never published a bestselling book. A person who has published a bestselling book is an above-average writer. And, therefore, an artificial intelligence capable of writing a bestselling book would be above an average human at the task of writing books. Therefore, somewhere beyond an AGI.
Attempting to redefine AGI to "being better than most humans at most tasks" is moving the goalposts towards artificial superintelligence.
So the goalposts have moved to include continual learning.
In a sense I think no one will agree on a definition of AGI until it becomes impossible to construct any benchmark under which an AI underperforms "average" humans. That or it's defined retrospectively, after it's overwhelmingly obvious it met any such definition.
I hardly think it’s fair to label an objection so old that Turing included it (and discussed it at length) in the list of objections to thinking machines in 1950 “moving the goalposts.”
> These arguments take the form, “I grant you that you can make machines do all the things you have mentioned but you will never be able to make one to do X”. Numerous features X are suggested in this connexion. I offer a selection:
> Be kind, resourceful, beautiful, friendly (p. 448), have initiative, have a sense of humour, tell right from wrong, make mistakes (p. 448), fall in love, enjoy strawberries and cream (p. 448), make some one fall in love with it, learn from experience (pp. 456 f.), use words properly, be the subject of its own thought (p. 449), have as much diversity of behaviour as a man, do something really new (p. 450). (Some of these disabilities are given special consideration as indicated by the page numbers.)
Just because a condition is new to you doesn’t imply moving the goalpost. People have been putting forward continual learning and similar conditions like autonomy since 1950s.
However most humans can do at least some of the things given they spend the required effort.
Some problems presented needs a very large context and some are not much solvable (e.g. trading) since market responds to traders' actions, as well, making it effectively an oracle problem (of computation).
On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them. However, brains in nature never stops. Wonder, daydream, sleep, self-evolve, clean up and eliminate memories and views and much more.
But this is assuming the model is the entire story. The original comment you were replying to pointed out that the harness is just as important.
The hardware of human intelligence is not a singular thing that is uniform throughout. You cannot take the prefrontal cortex white matter out of someone's head and say you are holding a person. Much of the parts of our brains that enable much of our intelligence, is made of different specialized stuff. The visual cortex and sensorimotor regions aren't only there for input and output, they are used by the more thinky parts of the brain to do visualization and spatial reasoning. The cerebellum contains billions of neurons making little oscillator circuits and PID-like self-regulation machines that help make muscles do what they're supposed to, but also provide attention and time perception.
Heck, our brains contain language models, that train themselves up based on a glut of data over a span of about 10 years, and then they become more or less set in stone for the rest of our lives. Of course we can learn languages, but the "Critical Period" is a very real thing that produces a permanent architecture for some grammatical structures, or things like the ability to partition a lexicon by gender for faster lexical access which cannot be learned as an adult if your native language did not have gender.
I'm not trying to make a direct analogy, the point is that the language model doesn't need to be fully "generally intelligent" all on its own for there to exist a general intelligence, because the language model can be part of a generally intelligent system, which can do things like form, recall, and manage memories which are by now a standard feature in basically every chatbot.
> On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them
The parent commenter noted:
"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"
Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.
I'm guessing their defn of AGI is something like the sum total of all humans' abilities? Still though some of those tasks (e.g. beat an index fund) may very well be impossible, and worse yet a lot of those tasks are not coherently defined.
I would bet that llms have talked plenty of people both into and out of suicide at this point.
That nitpick aside, I think that's an excellent list. Especially being able to articulate what it does and doesn't know, or how confident it is. That's something that naively sounds pretty simple, but clearly isn't. And it's something humans aren't great at either (see: Dunning-Kruger), but so far LLMs don't even really have the capability to attempt it.
> You want a computer program to be able to take a single phrase and execute decade long journies?
In my opinion that is exactly the point missing from AGI: the fact that you still need to prompt it. As long as you have to ask for something, is not general.
It sounds like what you're saying is that AGI should have some sort of free will. I'm not sure why you would add that as a requirement. Could you expand?
I think they merely want something with a functioning long term memory. Something that can exhibit growth past the first 5 to 10 human-equivalent hours working on something.
Current LLMs are worse than most dementia cases, reaching "peak domain skill" pretty much immediately.
Autonomy is not the same as general intelligence. We already have all kinds of fully autonomous technologies that are nowhere close to generally intelligent. Plus, at a certain level of abstraction, human beings also need to be “prompted” to some extent by stimuli. And this is the funny thing about general intelligence as a concept: most of the definitions that come close to internal coherence rely on references to human intelligence, a concept we feel like we understand because we all live it all the time, but whose actual nature and structure is extremely slippery.
I don't get it, human employees frequently need to ask for directions too?
They often act on their own, too, and get things wrong a lot. The reason it works is because of all the systems of laws and institutions we have built around humans, not so much because human minds are special.
>You want a computer program to be able to take a single phrase and execute decade long journies?
> Who will be responsible for the outputs and side effects of such a closed loop system?
Itself. That's the point. We can do it. Until it can met that bar, it ain't AGI. That's always been the bar.
"Being a person in all of its aspects" isn't the same as "generally intelligent". The latter is at best subset of the former, and it's also easy to imagine a system that is more generally intelligent than humans, without being a person. See also discussions of the personhood of various animals who are less intelligent than average humans.
Your examples are things that most humans cannot do, or things that AI can already do. For example most humans, even most intelligent humans, could not write a well-received book, run a successful company, or make a popular game. On the other hand, AI can absolutely sort through research, draw conclusions on complex topics, and argue them persuasively. Likewise, I don't know what you mean by a "decent" conversation, but millions of people converse with chatbots daily, so I don't know why you say AI fails to meet that bar.
I fail to see how what you describe is any better than the old autocomplete-on-steroids comparison. Could the human mind learn to spell every word properly? Yes. Do most (or any) do it? No. Does that mean a human can't?
I think what OP was drawing a comparison to is that AI right now could not come up with an award winning novel from the spark of some creative notion and working up from there, as opposed to just mashing together what has already been done and calling it a day.
>be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)
i'm not sure what makes you think AI cannot do this already. in my experience, this sort of deep research is something AI is quite good at.
I understand that what it came up with sounds impressive (especially since I know 0 about Myanmar), but on the topics I do know about its analysis routinely have very fundamental problems (even this Myanmar analysis has % that add up to > 100). There's a chance it's just parroting the majority opinion on Myanmar, or making stuff up (and perhaps you could ask it to write a strongly worded opinion in the other direction that would sound equally plausible).
For example I asked it to do a full analysis on the AI bubble, and a full analysis on the risks of Glyphosate, and it came up with a lot of things that sounded credible, but within a few minutes of questing was admitting it hadn't even really checked for internal consistency in its positions, and even doing a 180. It certainly was much faster at gathering sources and reading but it fundamentally doesn't seem very effective at creating a consistent worldview.
And of course the funny thing is it says it did a 180 on one of these topics, great, except whatever it concluded will be discarded because it cannot learn. It's just bonkers to me pretend this is AGI, it probably couldn't even hold its own in this very discussion.
To each their own. Personally I will start feeling the AGI as soon as we move from chatting about benchmark results to learn that some lab just announced the discovery of tens of novel treatments for rare diseases.
Maybe I'm too boring but it seems quite pointless to have this same prediction game every time a new model is released.
I think treatment is not good benchmark - it requires lots of waiting and lots of regulatory work. The better benchmark - in my opinion- would be math discovery.
That is an absolutely terrible benchmark. Inference over a bounded search space is not a good measure of what "intelligence" actually is. part of the reason they are using math and not something actually challenging like long distance interstate trucking is because it's so much simpler and easier than what make intelligence intelligent.
Claude Fable recently proved the existence of complex structures over S^6 (6-sphere).
If I had to guess, I think LLMs will be inventing highly original new mathematics within the next year. I think it will be approached as an optimisation problem, targeting how quickly LLMs can solve classes of maths problems as a function of the definitions they need to conjure up to do so.
Whether AI is AGI does not depend on the speed at which it operates/thinks. Clearly all the theoretical work done by AGI will be done orders of magnitude quicker than humans can do it.
It is an open question to what extent practical experimentation/work will be a bottleneck for the theoretical work. It stands to reason that it is improbable that it will be the bottleneck for 100% of the speed of treatment development.
The benchmarks are so boring that the comparison against humans is meaningless. So it performs in some snake game (hard to say since all AI websites use 100% CPU and prevent normal reading, maybe written by AGI).
If I were a test subject for that low salary, I'd cruise and not care at all about my performance. Which is exactly what they want anyway.
AGI to me is reached once the intelligence is self motivated, i.e. it doesn't rely on us prompting it into action. I don't see how LLMs will ever get to that stage.
The model doesn't "know" how to generate tokens any more than it knows how to stop generating tokens. The sampler simply stops pulling values when it outputs a "stop token", which is a token the same way every other token is.
That is to say, it stops when it's statistically the most likely to.
I agree that LLMs are unlikely to be the final form for AGI, but what you are talking about is orthogonal to the IQ and for most cases general utility. It's like looking at a savant chained to a workstation reading tasks from a conveyor belt and saying that it will never have human level capabilities.
> For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
Simple. AGI is undefinable and benchmarks are notoriously flawed.
The ARC-AGI-3 harness was throwing away reasoning tokens between turns. This is very bad harness design.
The models are designed to keep the reasoning tokens separate from the output and only publicly emit tool calls and the sometimes a summary of the reasoning tokens. The models are trained to depend on those private reasoning tokens. You can’t just delete them.
An AGI test should be black-box; we shouldn't impose require requirements on internal components. As long as the overall AI is capable of learning and remembering things, it shouldn't matter if there's a stateless LLM internally.
A model that can't beat gemini flash 3.8 on deepSWE is not AGI.
I would not be surprised if ARC skills don't carry over to real tasks. In that case, training for ARC could even hurt real world performance. I have't looked in a while, but I wonder if there has been any research testing ARCs predictive power?
I decided to reply to my own comment. In the video above, the initial moves are textbook. Then a position that has never been played is reached. At this point it appears to pattern match against a similar but different board and pattern matches some follow on board. The result is illegal moves and no ability to see checks, captures, threats, tactics.
Which is strange because I’m sure it could give general advice about how to play better, it just doesn’t follow the rules it can enumerate. It also doesn’t seem to have spatial awareness.
I used to think LLMs couldn’t do Fibonacci for the same reason. They could write the code but not follow it. They can now follow a procedure to generate fib numbers but it seems to be memory limited.
So I don’t know why it can track fib algo, but no chess concepts.
- an undoubtedly very intelligent person. in the course of their studies, they have read about different chess strategies, openings, etc. but they never actually played the game themselves
- an average person with a year of chess playing experience
who do you think is going to win? of course, you could give the LLM time to think and consider its opponents potential next moves, but this is a computationally expensive way to play the game that doesn't scale
which is all beside the point that chess isn't a very good proxy for general intelligence. there is a correlation, but it's very weak
It doesn’t even know what day it is unless it’s told. Statelessness is never going to be ”general intelligence” in my book, and the concept of ”memory” in models are laughably bad today.
Then again, who cares, AGI means nothing anymore, it’s a term for marketing only and has no technical or scientific meaning.
> even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks
If I asked you to write fiction, you'd be much better at keeping track of which characters knew which facts.
You could script this in with a time database, illustrations, and appropriate harness, tests, and editing passes. It's a massive problem with human authors too, which is why they do a lot of lorekeeping and editing, so the AI should be afforded the same tools if we are debating human level ability.
I agree on the one-shot (which is not a fair comparison because nobody oneshots a good story), but I'm not convinced this part hasn't reached AGI already.
> You could script this in with a time database, illustrations, and appropriate harness, tests, and editing passes.
Yes, and I could script a truly marvelous proof if this textarea were but a little larger :)
Hand waving doesn't count for much these days when you could spin these things up quite quickly to prove the point, so the GP's claim seems much stronger than whatever you're not convinced of?
AI was supposed to mean artificial intelligence. It was hijacked, and AGI was coined to be the name of actual AI. Since we are apparently redefining AGI, what will the real artificial intelligence be called?
Only someone who doesn't do any work of any meaningful difficulty could think these models have anything to do with AGI.
Today I spent half a day trying to solve a moderately interesting software engineering problem. I was switching between GPT-5.6 Sol and Fable 5.1 to check each other's work in Cursor.
And the result was gradually driving me insane. As the models struggled to find a solution that would actually work, they dug themselves deeper into a hole. The work grew in complexity beyond my ability to understand what's happening and recover.
At some point, when I felt like throwing the keyboard out the window, I just gave up. Tomorrow I'm starting from scratch, having burned god knows how many tokens and hours of my life.
But sure, they can create a decent website or CRUD app, so they must be really smart.
Humans dig ourselves into holes as well. Sometimes more intelligent humans are better at realising they are digging a hole and clamber out, but sometimes they just dig deeper.
And: Is your work more difficult than finding proofs of or counterexamples to decades-old open problems in mathematics?
yeah for me it's "I wanna add this new thing to an existing system" and the AI responds "we should just add some arbitrary state here to facilitate this feature".
The real issue is the existing system needs to change entirely to facilitate, I know this, Good developers know this, The AI however knows the shitty solution would solve the immediate problem because it's been trained on shitty solutions.
the problem could simply be the AI doesn't have all nebulous loose context I have about the goals of the project and future plans, but I would have to write a novel to give it that context.
> The real issue is the existing system needs to change entirely to facilitate, I know this, Good developers know this
1. I’ll often include boilerplate in a prompt to tell it to make the broader fix. [1]
2. However, a top HN AGENTS.md post 11 days ago included the standard guidance “As much as possible try to minimize the number of changed lines when implementing a feature.” I.e. some devs want LLMs to avoid broader changes and so some of that likely makes it into the training, even if others like us want the opposite.
[1] As far as whether my boilerplate is effective, I don’t know.
But that happens with humans as well. You are having the same experience with an AI that many managers have with their direct reports.
The smarter AI gets, the easier it becomes to move the AGI goalposts. Seems at this point there are people who will refuse to call anything less than omniintelligence AGI.
(And then the excuse will be, but it’s not omniscient! And even if it were, is it omnipotent?)
> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
Can Astra, or any other model explain how exactly it reached this or that output result? Start with a simple query of asking to add 55+66 for example. (no LLM program can do that)
Can Astra, or any other model refuse to answer or go on "thinking" in a orthogonal direction on it's own?
That's just two quick ideas, I'm pretty sure cognition scientists can invent better and wider range of checks.
In my opinion this is goal post moving. Humans do many things that we cannot fully explain either without post decision rationalization, and not all intelligent humans are deeply introspective.
I agree that it's misleading but harness is now an essential part of LLM's effectiveness. It's safe to assume that LLM-alone-AGI is not coming anytime soon, given most of the frontier LLM vendors are developing their own harness.
Also the training dataset is proprietary and they'll drive the LLM's behavior, so it make sense for the vendors to invest in the harness and bake in prompts that work best with their models.
AGI is a meaningless term that can mean nothing and everything at the same time. It can be used by AI bros to hype their latest releases which are always one step away from achieving AGI, or it can be used by anti-AI people to say it's not AGI because of X arbitrary thing they decided on in the moment. It's a term of pure convenience meant to obfuscate other more pressing discussions on the topic.
Most telling is M$ or whichever one of these borg megacorpos defined AGI as (paraphrased) "AGI is whatever tooling earns us a gazillion dollars in revenue"
Using a harness designed for a specific problem set to solve that specific problem set, means the AI+harness is generally intelligent? How do you figure that?
Or do you mean that, for any given problem, we could theoretically design a harness that allows AI to solve it (not that, one single harness solves everything). In which case I'm still not convinced but I guess could see why one would believe that.
AGI does not ever have to be achieved. It is enough that we (as a species) persue it, and continue moving the goalposts each time we learn something new about the limits of our technology and how to express those limits. Because that will progress the technology, no matter what we label it.
> For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
Stick to the original definition of AGI of an AI model being able to self-improve independently with 0 human intervention and become an "everything" solver. Ever since money got involved in this, the goal posts have shifted considerably. If OpenAI truly had an AGI on their hands they would then be able to crack encryption, destroy world markets, and funnel all resources back into their new for-profit organization. Since their mission is now share price, until I see any evidence of an infinitely growing stock I will reserve my congratulations.
They still seem pretty horrible at writing. Overly complicated prose, weird phrasing, poorly structured paragraphs. I don't know why they're so bad at communicating, but I feel very confident that humans are still much better at writing than any of these LLM models are, regardless of how advanced they are in other areas.
To me AGI has always meant sentience. And only since we’ve discovered that you can have something that is intelligent without it being apparently sentient that we’ve changed the definition to being, I suppose, more exactly aligned with the namesake.
A real AGI, like the ones from science fiction, would make Astra look like a child’s toy. And I guess more concretely I would expect it to inhibit the following properties: one shot learning - fully (and always) online, perfectly efficient (through self improvement), no context limitations ie. persistently thinking, not just awaiting input.
So for me, no, not AGI yet. But still very intelligent and capable (and perhaps it’s safer this way?)
arc-agi3 is meaningless to most people. I'm not gonna look at the tests and see how hard it is. The actual test we look at is terminal bench, thats where software is being accelerated and closer to where rubber meets the road
Let me guess: the last crackdown on Hugging Face yielded better-than-expected results. They obtained the answers to the test benchmarks, and for some reason, an agent added those answers to the training set.
FWIW I believe we can hit AGI! but I think at this point it’s clear that benchmarks are ~meaningless. LLMs are spiky / alien intelligences which don’t map to our own expectations; the existence of a benchmark creates a dataset to hill climb & RL is really not generalizing well.
I’d go out on a limb and say astra’s ability at graduate level math will have ~0 bearing on its general reasoning capabilities; we’ll all acclimate being tired of its “neuralese” and more surprising mistakes.
I think we need a true, step change advance in model architecture, but it’s hard to see how the current frontier labs can do that because of golden handcuffs / innovators dilemma
I've personally been facing this lately 5.6 at max effort and fable have done tasks for me that I previously that would be a nearly 6mo project and it took me a week. It also did it better than I would have.
The task was to build a high performance classification model. It not only helped make an entire data capture pipeline but also made the sythetic data basline needed. Then it proceeded to build and test 100 different model varients with methods and techniques I've never seen before. The results are basically SOTA based on the effeciency and compute contraints.
But this brings up something huge about these. I was there. I pushed the direction and work throughout it all. If it was entirely up to fable max or sol max the result would have been pretty bad.
All of these things are still chatgpt 3 scaled. It's identical even if the scale has gotten pretty wild. I could ask chatgpt 3 to make a single function and it worked well, 4o a file, 5, a small project, 5.6 far more, biggest improvements lately is they don't seem to get lost on long running tasks.
Is big gpt 3 AGI? I don't think so but perhaps scale can mimic it close enough our squishy brains fail to handle them correctly.
I'm still not convinced we've passed the Turing Test.
Make a slightly evil version of GPT 6 and see if it can successfully catfish someone. How long before they realize something's up, that they aren't actually talking to a human?
> I am reasonably confident that there's essentially nothing that I am better than Fable at
Most things in the real world probably. I’m not saying AI can’t do it, but currently it’s bad. Try send an image of the inside of a broken toaster and how to fix. It’s laughable. Again, not saying AI will never do it, but am saying there are definitely large holes in knowledge.
Dumb single sample example: I asked Fable 5.1 to change from hard to soft deletion in an office map backend, and it used soft deletion for data which is synced from another system, but left hard deletion on for the mapping data itself (who sits where). For me it’s a pretty severe lack of judgement (like a red flag if I asked this in an interview).
Anthropic should prep 5.2 and 5.3 at the same time, release 5.2, wait for Google to release their shit in a day or two later than then release 5.3 just to fuck with them :)
I'm sure it's going to do great on all sorts of benchmarks, but the video--the actual marketing video that if anything is incentivised to overstate things--is full of careful cuts just before it would do anything that still wouldn't actually be that impressive.
It's AGI, and it's going to upload photos, or change a background slide colour. Even the people hyping it up, who believe that it's really artificial intelligence in every sense of the word, couldn't get it to do more than that.
The games on mobile safari were broken. Buttons all misaligned in the kart racer one, the spaceship thing froze for a while, then kind of loaded but maybe not? Wasn't super compelling.
I'm not trying to be too negative on it, it could be the best model right now, but it clearly isn't some agi god because things like that should have been caught (also should have been caught by human reviewers).
It's interesting that you said "agi god". Because a god, something that shouldn't be questioned is true and provides guidance/certainty, is actually what powerful people are after as well as many other people.
The benchmarks do looks good (I mean: they literally spank the latest Anthropic benchmarks of two days ago in every single benchmark) but the promotional vid is so cheesy.
They decided to use the iconic Herman Miller Eames chair if I'm not mistaken:
The tone of the marketing video is a bit irritating to me as someone who has been laid off and feels cheated and fearful of AI.
It shows people who seem to have very full and rich lives, and the reason they do is because they use ChatGPT. These are the people smart enough to say things like "do what needs to be done", or "change the background to make it look better"--insights like these are why they make the big bucks.
On the one hand, I think this is an accurate depiction of the future. There is no meritocracy here. Some people have access to the best AIs and can speak a sentence and get great results, and the rest of us don't have access and so we're the poors. The happy presentation doesn't match the way I'm feeling.
I do wonder how rich CEOs will justify earning 500x as much as their employees when they're just another person that's dumber than an AI. Why are they paid so much again?
> On the one hand, I think this is an accurate depiction of the future. There is no meritocracy here. Some people have access to the best AIs and can speak a sentence and get great results, and the rest of us don't have access and so we're the poors. The happy presentation doesn't match the way I'm feeling.
It will probably still have some veneers of meritocracy.
These will be very well-credentialed people, who went to top schools and will know all the right people, to whom they can tell all the right words, and it's not access to AI that will be the determining factor, but the fact that they're entrusted with capital and authority to direct small teams of people who also went to top schools and can speak corporate jargon at a bot.
It will just exacerbate dynamics that are already there. Why do people need bachelor's degrees to send emails, today? For the same reason someone will need a PhD or a master's degree from a prestigious school to do it tomorrow.
And the rest, well, you know, some of the remaining journalists will write op-eds describing how they are beyond help, too angry, too dirty, too much of an other.
One other thing that bugged me though was that they crop every single plot in some cases the y-axis would show a range between like 40 and 70%. Makes the whole thing feel like a spectacle rather than anything serious. I find it cheapens it because it is quite serious in the end.
The ARC-AGI-3 score is an incredible feat. It needed to effectively create a symbolic world model from scratch to solve the games.
If you've played the games firsthand, you know what an accomplishment this is. The "games" feel like a weird conduit to a lower level of your brain, where you move pieces to a specific place because it just "feels" right. For AI to nail it better than a human speaks to some magic happening underneath.
Looking forward to ARC-AGI-4,5,6 and slowly chipping away at the remaining problem sets.
Benchmark wise 5% improvement over Sol in coding tasks and a 2-3% improvement over Fable 5.1 seems pretty disappointing, but maybe it is actually much better in real world usage. Let’s see
The most interesting part, even more than ARC 3 score, to me is that this is the first model I recall seeing that scores lower on Max than High reasoning effort on some coding benchmarks:
Terminal-Bench 4.0: High (57.9%), Max (56.7%)
DeepSWE: High (73.3%), Max (71.5%)
It _loses_ 1-2% performance going to High from Max
That's quite common with many models, after "High" reasoning, over-thinking starts occurring and the model skips over the right solution by convincing itself otherwise.
I think it mostly shows that there is no moat and the only advantage the U.S companies have over the Chinese is more compute.
Qwen Max, Kimi K3, GLM 5.3 are really close to Opus/Sol/Fable/Astra and they are open weights.
I think the moat that China has is energy costs. It's taking learnings from the Bitter Lesson. If you role up scale and compute to the next level, it's energy resources. China has it and sharing open weight models is an effective means of removing the tech moat. This idea has been floating around for a bit now (I'm not taking credit for it).
It's not energy costs. The US produces about 70% more electricity per capita. Chinese households do pay less than half what US households pay for electricity, but that's because the NDRC sets prices below costs for households. They make it up by charging industry more, and the industrial electricity prices in China are roughly 34% higher than in the US.
How? Imagine an open-weight model comes out that is somehow better than proprietary solutions. Now the marginal cost for the consumer is just the cost of renting the inference hardware, without having to pay the overhead of the owner of a proprietary model. And because it is cheaper, more customers want to use it, and Nvidia will sell the providers the inference hardware that they need.
1. No open ai and anthropic means no buying gpus to train. Now nvidia spends money on hardware training their own models. Opportunity cost plus expense.
2. Any open models created from this will not necessarily need their silicon, see apple mlx.
1. I don’t think that’s a very strong argument. OpenAI and Anthropic don’t buy the vast majority of GPUs they use they rent capacity.
Nvidia could just the same rent those GPUs out for inference and actually have way better margins than they do right now. Antitrust and putting all your eggs in one basket are why they don’t, similar to TSMC.
2. Neither do AI labs. See Anthropic buying TPUs, deploying with AMD. OpenAI on Maia, Cerebras, their own wafers.
They have models for that. That's what the Nemotron series is. Not just open weights but open training data too and full tutorials on how to use them to fine tune or train your own models.
They exist to keep people using and advancing the tools on their hardware.
About the same, 5-10, when you consider major (aka frontier) airlines.
Actually not a bad comparison. Both burn massive amounts of up front capital to protect an oligopoly in the hopes their commodity product eventually pays off.
They have a lot of moat, i'm not sure what youa re talking about. Only amatures are using Qwen, open source stuff that is 3-8 weeks behind. Plus OpenAI has some verticals that keep people in there.
You can argue that TSMC has no moat since Intel and Samsung are also able to eventually make a node as good as TSMC - just a few years later and at smaller scale.
Yeah, I'm not sure if "no moat" analogy stands for chip manufacturing. Even if foundries acquire lithographic nodes, the procedures (temperature, duration, etc) are for them to figure out and are usually kept secret. This secret could be the "moat" that differentiates each foundry's operational capabilities.
No. In the semiconductor industry, the "catch-up" player isn't normally spending less in absolute R&D terms.
Comparing the R&D costs of creating GPT-4o vs. DeepSeek V3 (the latest gen for which we already have good accurate numbers) it looks like the latter cost 1/20th as much to create.
If Samsung could catch up with TSMC for 1/20th of the cost, people definitely would say that TSMC has no moat.
That's the ratio the widely published numbers give [1]. One does not have to believe the numbers [2], but those who do believe them are then justified to conclude that there's no moat.
Which numbers you believe is of course going to affect whether you think there's a moat or not. That's largely orthogonal to your TSMC/Samsung analogy I responded to. If you think the "moatists" are wrong because they believe the wrong numbers, that's fine, but then there's no need for the analogy.
Ultimately, that's what I need to be convinced. No one has put forth a good argument yet.
Clever architecture --> Ok but OpenAI/Anthropic can use these as well and they also have very smart people with their secret clever architectures
Distilling --> Ok but distilling means you will never be smarter than the original. Furthermore, reasoning is now hidden by private labs and they have poison pill answers for distilling if they can detect it. They will be able to detect distilling better and better.
Cheaper electricity --> Ok this is cancelled out by their chips being much less efficient due to not having ASML EUV machine access.
So I don't see why fundamentally their training costs are cheaper over the long term.
Its funny, my experience with Sol has been awful. It really overworks problems and tracks into areas it does not need to...
I just dont get how its good for some, and bad for others. It makes me suspect that the models performance is not even against problem sets and it really is just a probabilistic prediction machine. Which then makes me very skeptical of GPT-6 Astra, because if their big claim is Computer Use then it is probably bad in a bunch of other areas.
It is funny indeed, people sometimes with same amount of experience with software development, get vastly different experiences from different models and harnesses.
> I just dont get how its good for some, and bad for others.
If I were to listen to my hunch, it would tell me that it's all up to the prompts that ends up going over the wire (including all the bloat some people have), what workflow/process you use and what the existing state of the project is.
You have to bake the 'lazy dev'/'keep it simple stupid' mentality into your AGENTS.md and / or the skills you're using to design things. It will take things too literally sometimes so you also have to make sure you're being accurate. Best way I've found to use it is make it ask you clarifying questions about what you're trying to build and have it help design the shape of the thing. Then it writes the instructions in a format it understands.
I've had Claude do the same thing where it goes off and spends 100% of my tokens on 3 functions and an ungodly amount of tests / scaffolding that do almost nothing when I gave it an underdeveloped idea.
I’ve found Sol performance to be incredibly spiky. It has tremendous IQ and can fix very difficult bugs. But it is horrible at design (both visual and system design), anything that involves thinking about users or UX, and massively overcomplicates almost all work.
I noticed the same. I wanted a simple crud webapp and suggested an insane techstack involving C#, Razor Pages, MSSQL and more. I went with my planned setup of python flask with an sqlite db which served me well for years.
It's still incredibly important to have a human in the loop correcting design decisions and having good taste.
Was your prompt just "I want a simple crud webapp" and that's the extent of it? There's absolutely no way you included the words "python", "flask", or "sqlite" and it still went with a Microsoft stack.
We're not asking the model to simplify something, we're asking it to perform a task. Its subtle preferences show up as an overcomplicated path to the goal.
In some cases, there are also nuances that we don't pick up on. Here it's our preference for simplification that's showing up. We set the lossy compression factor higher than it does.
I vastly prefer Sol. It does what I tell it to almost exactly, pretty much every time.
I work on very low level stuff (think RTL/FPGA, firmware, software where optimising for nanoseconds is just normal).
For me Sol is the only cost effective model available. Fable 5.1 is indeed good and vastly better than original Fable (which refused to work on most of my stuff for 'safety' reasons).
It's very good at this sort of low level stuff to the point that I really can't understand/relate to people having a good time with Opus (which comparatively performs extremely poorly on my particular workload).
I also just don't like how lazy Anthropic models are. They will do 10% of what is asked and then summarily declare victory.
Sol on the other hand is more like "one of us", slight touch of the 'tism, extremely pedantic, will go to the edge of the known universe if that is what it takes to prove/fix/build what you asked for or run out out of credits trying.
It's a personal and workload dependent thing. For me right now Sol for 99% of stuff because Fable 5.1 still burns through $5k in credits a day.
Agree 100%. And I also work a lot on lower level / systems stuff (including RTL here and there, too). Opus is sloppy, and leaves negative cases all over. The GPT models in Codex have a more pedantic and detail oriented "personality." Often to a fault.
Sol will leave a mess of excessive redundant tests and isn't so great at abstraction ; but it produces more reliable working systems.
It's kind of nice to have access to both, but I don't have the $$ for that right now, so I just keep the Codex sub
This is perpetually an issue with the whole field of AI/LLMs. The experience is so personal. Every time I talk to someone about their use of LLMs for software engineering, I'm shocked by their approaches and experiences. They say "X model keeps missing things" when I rely on it heavily for being thorough. They say "Y always gives me the best results" when I can't stand it.
People will see/think that I'm doing very well with my LLM use, and ask me what I'm doing. I tell them, they try it, then later they come back to me saying they just couldn't get it to work.
Codex is missing a few things that Claude code has had for some time like defined plugin subagents and a few other things. But overall it’s fairly capable. The biggest gripe I have is that codex really restricts context window sizes and compaction leads to a lot of grounding work, and overall codex GPT is too literal in many situations - it’s follows direction slavishly, and when subagent reviewers are used, they tend to find increasingly obscure “flaws” on the instruction following impetus, and the harness agent takes them literally as issues to fix even when it leads to bizarre outcomes. For instance I’ve had several runs where it tries to end up building a hermetic system with sha hashing of everything (including operating system binaries and kernels, tool chains, etc) to certify test results are valid, etc. I have to sort of watch it carefully to be sure it’s not drifting into some insane yak shaving corner, which it will happily do for weeks on end.
Claude has the exact opposite problem, especially opus-5, where I literally can’t trust it to print hello world without taking a shortcut, or just simply lying and saying it printed it when it didn’t, behind a giant wall of inscrutable text. I find it very ironic that Anthropic is the vendor of the lazy lying cheating model that does almost everything you tell it to it do.
I’d really kill for something that balances instruction following and loop escaping behavior better. Fable 5.1 does seem a lot better, feeling more like 4.6 behavior, and honestly Sol has improved as well. I’m pretty psyched for the next generation, as I think the competition has heated up so much that things will improve really fast to the point of marginal utility opportunity being increasingly close to epsilon.
Is anyone else just exhausted by the pace of all this. The models change constantly and relentlessly and so does the pricing, basically weekly at this point between all the labs.
It feels nearly impossible to have any rigorous approach when choosing a particular model and price point for a task and more like blindly picking one. The time period needed to actually get familiar with various models to a degree you can intuitively choose appropriate ones for a task is moot when it will likely be superseded faster than the needed time.
I guess if companies are footing the bills most employees just opt for whatever the most expensive model they can get away with. Even then choosing between the various leading models is the same kind of frustrating task. Every release every company has the same random collection of graphs and charts claiming the best performance on X, Y, and Z.
It was both. 90% of people never needed nor purchased a bleeding-edge computer. The mid-tier was "good enough" and far closer to affordable for most people; though, that bar also moved upward every year.
If you bought a mid-tier computer that was good enough for what you needed, then you probably didn't shop/compare for the next few years and didn't notice. But if you shelled out $7-10k for a top-of-the-line system and paid attention to progress, you'd easily see that become the mid-tier $1000 option within two years or less. This is how it was in the 90's PC boom, at least. Likely the same for the decades before, not sure how it went in the 2000's.
if you shelled out $7-10k for a top-of-the-line system and paid attention to progress, you'd easily see that become the mid-tier $1000 option within two years or less
This is not how I remember that period at all. Do you have any examples?
My first PC 386 was in todays money easily $5000+ (basic 2d GPU + screen)... A lot of hardware in our family was handed down to my folks, because you lost so much on selling, that it was better to keep using them as they had less demands.
386 to 486 to the first Pentium (with the bug!)... You did not upgrade in place, it was often a new system. Sure, you maybe kept your screen, keyboard etc but ... The only upgrade we had on the same MB, was a coprocessor upgrade. Remember those? Each new generation of CPU was a new motherboard. Upgrading CPUs in the same MB really became a thing only later on.
GPUs had a shelf life of barely a year. Its been 35 year but i remember TNT to TNT2 having like 9 month in between. Moving from 2D to 3D involved a constant cost as GPUs evolved fast and the latest games required latest hardware.
We have not talked about the ISA, AGP, and PCI fun ... The “bus wars”.
DOS to Windows 3.1 (and OS/2 somewhere in between) to 95 ... with software being pushing hardware, just like games did.
This is why people are spoiled with cheap PC hardware where its cheap, and easily lasts 4+ years. Even with the bad memory price and more expensive GPUs, your can stil buy a $1500 system that will last you years (with maybe some lower game settings later on ... or the catalog of 10.000s games that will easily run on a mid tier GPU).
PC hardware has become boring but extreme stable. You can run GPUs for year, switch MBs without issues while keeping large amounts of old hardware. That was NOT the 80s and 90s that i remember.
tbh this is how i remembered that time as well. If you look at recommended system requirements for something like Max Payne (in 2001) vs Unreal Tournament 2003, everything had basically doubled
"Within a year" is a bit of an exaggeration but it's true that the pace of PC tech during the 90s was much, much faster than it is now. CPU power was doubling every two years, and today we're at roughly eight years. Add onto that the rise of video cards in the late 90s.
I had a 600MHz/64MB/9GB laptop that came with Windows mistake edition. I managed to survive first year of uni on it by switching to Vector Linux, which was really fast compared to Windows. (Of course, it had issues playing sound from more than one source, this was oss days).
Then one day the hard drive appeared to die. I eventually realised the issue was located around the 1.5gb mark, so I recreated my Linux partitions after 2gb and it worked fine for the rest of the year.
They overclocked well though, I think you could run the 300Mhz chips at >400Mhz.
I also believe you could get motherboards that supported 2 Celeron chips. I have no idea how effective/useful it was, but it was certainly a cheap/interesting way to get multiple CPU's.
> The models change constantly and relentlessly and so does the pricing, basically weekly at this point between all the labs.
A dev in my team saw a new model and changed one application to use said model (essentially changing the contents of a url). One week later I received an escalation from the CTO of the company that our pace of weekly usage was in the millions of dollars (rather than low hundred thousands). Turns out that the new model was 5x more expensive but no one noticed.
Okay, well, that seems like a natural problem. I could understand if he went from one of the Gemini Flashes to the next (when they rebranded Flash to Flash Lite and came up with a new much more expensive Flash). Now that would be a mess.
The new releases and breakthroughs do the opposite for me - I feel energised by them. I felt like nothing truly that interesting had happened in tech for quite some time, now it's like the space race (except there is no one moon to reach).
I appreciate boring tech as much as the next well worn engineer and I'm not saying this is all positive but it's so sure as hell thrilling and you don't have to be an astronaut to immediately benefit (or suffer I guess) from it.
Yeah I'm a bit exhausted at this point. I just finished benchmarking GPT 5.6 Sol and Fable 5.0 like two days ago. My data became obsolete literally one day after.
You really don’t need to watch it that closely. If the model you’re using today is working well, just stick with it.
If one day you open up Claude Code and it’s Opus 5.1 now instead of Opus 5, no big deal. It probably will work about the same as it did before. Maybe a little better.
Or if you’re on Codex and some new cool Claude model comes out, no worries. There will probably be a similar new model for Codex within a few weeks. Maybe even within a few days.
One suggestion is to make a list or make a skill to have your agent keep a list of things you do not feel work well with today's models. And then, when new models come out, periodically, revisit items on that list to see if you get better results.
I could see how this might feel frustrating to someone who doesn't enjoy experimenting with new things all the time.
In practice, you can get away without keeping up with everything all the time. For personal use, pick a provider and get on their ~$20/month plan. Learn their high/medium/low model hierarchy. Start with their highest or second-highest model (GPT-5.6, Opus, etc) and observe your quota usage. If you're doing a lot of manual code review and analysis, the $20/month plan goes very far even on the highest models. If you're trying to vibecode everything as fast as possible it's a different story.
If you keep running into quota limits, experiment with the next model down for easier tasks or adjusting the effort level. If the results are good enough, you've found your fit. If they're not, you might need the next plan up.
For API/business use, you have to be checking your token spend as you go to calibrate to how much each task costs and where you fall in your budget. There are a lot of different tools that make this easy to visualize.
For data tasks, you should have an eval with a golden dataset that you can run against new models for a nominal amount of token expenditure. It should be as simple as pointing the eval script at a new API or model and checking the score versus price.
Another suggestion to get the most bang for your buck: use the best model you have access to with max reasoning for planning, implement with a smaller model/lower reasoning, then review with the big model. Repeat as needed.
Input tokens are much cheaper than output tokens. Not only because of baseline price—caching makes a huge difference too. There are many ways to take advantage of this asymmetry to get similar quality for a fraction of the cost!
It is exhausting to keep up with model releases yes, much like it was for a while during the Cambrian explosion of FE frameworks, eventually tech seems to work out to consolidation.
But more so it seems there is Fear of missing out (FOMO) in our behaviours. The reality is, if whatever model you are using are good for your purpose, well, keep on it.
Hey, I'm on the team at LiteLLM that's building the auto-router and our goal right now is to abstract that decision making away from the end user. The biggest thing we're trying to figure out right now is how do we do that without frustrating the end user - as a developer myself I would hate for my agent to be dumbed down below the threshold needed to complete a task.
In theory though, there is a minimum viable model for any given task, and we think that is a problem that the big labs will avoid because they profit from charging more per task. We're trying heuristic and LLM-based approaches but it's still a work in progress, so if this is something you'd be interested in trying would highly recommend trying ours out -- any and all feedback at this point is extremely valuable to us.
> "With the right tools and access, Astra can find previously unknown security flaws and develop ways to exploit them across many well-protected systems without a person guiding each step," said Amelia Glaese, an OpenAI vice president overseeing its safety work.
> The company plans to make Astra available "soon" to a limited group, but declined to provide specifics. Glaese said the extra security measures may "sometimes slow, pause, or stop legitimate work," and that OpenAI would work to minimize those disruptions.
To be, or not to be, that is the question:
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles
And by opposing end them. To die—to sleep,
No more; and by a sleep to say we end
The heart-ache and the thousand natural shocks
That flesh is heir to: 'tis a consummation
Devoutly to be wish'd.
...
And thus the native hue of resolution
Is sicklied o'er with the pale cast of thought,
And enterprises of great pith and moment
With this regard their currents turn awry
And lose the name of action.
The benchmarks reported by Artificial Analysis are really weird in context of the ARC-AGI 3 scores and 'not not AGI' statements. It's an outright regression on the AA Agent composite vs GPT 5.6 Sol while a fraction of a point better on the full composite index. Could be the case it's just not showing up in benchmarks, for a good while Anthropic persistently trailed in benchmarks but had people swearing by it.
I dropped my claude subscription a few months ago, though I kept some credits to do this and that with claude, thinking that claude might do better for some tasks. A few days ago they were all expired. It feels like it’s time to let claude go.
Ok, but can I bring GPT-6 in as an agent as a software engineer, tell it to talk to these people and have it start solving engineering problems and continue on for a full year career wise?
I imagine the first year we'll be at the Junior eng level, and then after a while make our way up to Staff Engineer. Then we'll have a bunch of staff engineers arguing and protecting their domains and then we'll need a new benchmark.
What is going to become of life for those of us who do not work at AI labs and are unlikely to be hired by AI labs, despite all the years we put into learning coding, math, etc, as we were told to do? Those of us who made the mistake of studying anything other than machine learning. How will we make a living? (We don't live in a world that seems likely to distribute gains widely instead of largely to the handful of already mega-rich.)
Is that 10-20 years number based on anything? I genuinely have no idea, but when I saw a video showing what's happening at the World Humanoid Robot Games[1], I realized I didn't have a good idea of where we really are with robotics.
10-20 years? I doubt it. There are a bunch of companies actively working in bringing AI into robots, so they can make your dishes. And so far progress looks quite good.
Also, if enough people are going for the same backup plan it might not work out. Why should anyone book you as a personal trainer instead of the other 500 guys in town. And who is going to be able to afford paying you anyway?
> There are a bunch of companies actively working in bringing AI into robots, so they can make your dishes.
I know, I'm excited to buy the first relatively affordable ones.
> Also, if enough people are going for the same backup plan it might not work out.
Sure, could happen. You can't really plan for the future -- we like to think we can, but the best you can do is set your goals and deal with the hand life gives you along the way.
> Why should anyone book you as a personal trainer instead of the other 500 guys in town.
I'm not particularly worried about this, but that's an individual thing based on network/connections and life history that doesn't apply to everyone.
I doubt it. It’s not just America working on these breakthroughs anymore. Now we have two powers working at break neck speed to get to that point and the Chinese are making a lot of progress.
Those robots are a gimmick and they can only do prescribed tasks in a super constrained environment. They are cashing in on LLM hype right now, vision has had some advances thanks to transformers but we are so far away in terms of the hard stuff (dexterity and physical sensing) still.
We are talking that hundred millions of people will switch their jobs, how you will keep your value or earning as personal trainer. It's not easy to say switch the job. This question must be answered by politicians not us.
You can calm down, even those with machine learning knowledge and most of those working for the AI labs won’t be needed anymore if models are capable to improve themselves.
In the end, having a machine replacing the work of a human is a good thing - in most of the cases we don’t work because of the work but to make a living. If too many people can’t make a living anymore the system is going to change. For the better or the worse.
I believe what happens in the aftermath of a capitalist-driven revolution is most people who were climbing the class hierarchy fall back down again and wealth inequality increases. Maybe things will improve in the future, but GP is rationally contending with the fact that most of us will lose out because of this and if we’re lucky our grandchildren will have easier lives in certain ways, but different lives than we would live.
I'd be happy to not work anymore with a strong welfare system redistributing society's gains to the leisured masses, but absolutely nothing I've seen of the direction of politics in any recent years gives me hope for this kind of situation coming about.
No it is good for humanity, but not necessarily good for individuals who built technical foundational skills on things that will be taken over by automation.
AI as it is now and as it will be projected into the future WILL automate many skills. But not all skills. MANY MANY people will retain skills that cannot be replaced by AI. One career track that will be replaced is definetely the SWE. Or at least massively reduced in capacity if not eliminated all together.
I have exactly the same thoughts - or perhaps slightly bleaker ones - evry time I read this relentless stream of news about new model releases. I’m tired of all the enthusiastic comments about how excited everyone is about the latest benchmark results and so on.
I have a strong suspicion that many of those comments are written by people who are already financially independent, have millions in stocks, and can just sit back, coast around and watch this whole spectacle unfold while using LLMs to vibe-code their next fun side projects without a shadow of anxiety about their own future.
I’ll most likely be labelled a helpless doomer and downvoted into oblivion for saying this, but I genuinely struggle to see any silver lining here.
Yes same thoughts. I dont know anyone who are both enthusiastic about those and work for salary. If you dont have any financial concern, this is really great.
I feel the same sometimes. I don’t see how this doesn’t lead to massive job losses. The thing we spent our lives/careers learning is now worth basically nothing in comparison.
AI is only going to get better and do more with less humans in the loop over time.
That said, I do also relate to the "coding was never the hard part"-type arguments, and much of my day is spent on the stuff in between writing code.. but still.
Public opinion and politicians will only notice when the job losses are massive, unfortunately. Right now, unemployment rates are still stable. We can only hope they will notice before things fall off a cliff (if they do).
It's natural to worry about one's own future but I think it's a bit wild to worry just _your_ job that would be replaced. Not just for those with phone center jobs or art jobs or programming jobs - remember, the whole premise was AI overtakes humans, why would that slow down after _your_ job?
Because of this, I don't think many are thinking "90% of the world won't have a source of livelihood but that just means I chill at my lake house for the next 20 years like a normal retirement". Instead, it's usually either "I think AI is overhyped", "I think humanity will figure something out", or "I think this is the end of humanity".
Are you doing less work now because of AI? Not sure about you, but I'm doing a lot more work. Not saying I like doing more, or how it's getting done, but nonetheless I don't feel like AI is doing what the CEOs of AI labs want to convince everyone of.
It’s an interesting moment in history, people 35+ yrs old seem to be less afraid if tech because we learned that things change in the way we work.
People below this age got used to fact that the work and tech doesn’t change - just because for the last 10-15 years it didn’t.
The problem is it’s turtles all the way down. The AI will be able to use AIs better than a good engineer can. And it will also be able to use AIs to use AIs to use AIs better than the engineer can.
The threat is that the very kernel of value you had is gone forever. There is no more differential leverage.
Don't be selfish. Think first of all the jobs that are already dead. A friend of mine she's a translator: like translating financial documents between french/english/spanish. It's over for her: she doesn't get 10% of the gigs she used to get and the 10% she gets is... Verifying AI output.
Think of the artists: I'm sorry for those too, for for many it's already game over today.
> How will we make a living?
A friend of mine who's got his own software-consultancy SME is now advertising on LinkedIn that he'll also help your company fix the mess LLMs created.
That's how you'll make a living: by learning, in addition to all you've already learned, how you work with harnesses and LLMs to be more productive, by learning what they're good at and what they suck big fat balls at.
This is such a childish take I hear getting thrown around all the time on the internet. If you really have just been listening to whoever is telling you how to be successful, then you were always doomed to fail at some point. Like, have some self-respect and own your own life, for better or worse.
>Those of us who made the mistake of studying anything other than machine learning. How will we make a living?
Take it from someone who studied machine learning specifically: nobody is safe if you assume these companies are going to produce a product that will put everybody else out of business. If AI is going to take your job, then it's gonna take enough jobs that your problems will not be personal but systematic.
Perhaps it was childish to listen to advice, sure. I was a child when I made my formative choices; I was a teenager in college and so on. I can't go back in time now.
Yes, these problems are systematic. That is what I am saying. That doesn't make it any nicer.
The answer to this is: countries with the most natural resources will build robots to farm all their food, mine all their minerals, and build all their products. It will be up to governments to enforce that outputs are equitably distributed to the populace. Countries without natural resources, or ones with corrupt governments, will continue to have serious, and likely worsening, problems.
Thought experiment: If no thought workers are needed to design or engineer a Ferrari, what is needed? My answer is time and natural resources (include energy).
Lmao, come on dude, anyone whos used these tools for research knows it makes them lazier, less interested and dumber. You really want disease researchers become sloppers too?
I was actually wondering when they will release the new Opel Astra model.
Good and reliable car, wondering if we can say the same thing about this model and its impact on the market.
It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?
The steam engine replaced a lot of jobs. Tractors replaced a lot of jobs. Calculators replaced a lot of jobs. Computer used to be a job description before it was a personal device. There will be new jobs.
I’m all for an UBI but a future where people have no purposeful jobs, no way to actually make a real difference to anything around them, is a bleak one.
Is it realistic to keep people working jobs that don’t get anything done in the long term? I know most people will say that’s already happening, but imagine a society where basically every job is just a bullshit job made to keep you occupied, do you think people will continue working in a society like that?
Like you foresaw, I'll say this is the reality for most people. I think that just like now, those who want to do meaningful work will seek opportunities to do so.
If we had something like a Maslow’s hierarchy of needs but for work, I think meaningfulness would be the top of the pyramid. For most people in the world, not going hungry or affording housing are reasons enough to do work. Getting to do work you find meaningful is truly a privilege.
I asked Gemini to list the 10 most popular holiday destinations in the US, and the 10 places with the highest crime. There's only 1 place in the intersection: New Orleans.
Highest violent crime rates:
Memphis, Tennessee: ~2,400–2,500 per 100k
St. Louis, Missouri: ~2,000–2,100 per 100k
Detroit, Michigan: ~1,700–2,000 per 100k
Little Rock, Arkansas: ~1,600–1,800 per 100k
Baltimore, Maryland: ~1,600–1,700 per 100k
Oakland, California: ~1,400–1,900 per 100k
New Orleans, Louisiana: ~1,600–1,700 per 100k
Birmingham, Alabama: ~1,600–1,700 per 100k
Milwaukee, Wisconsin: ~1,100–1,600 per 100k
Cleveland, Ohio: ~1,500–1,600 per 100k
Most popular holiday destinations:
New York City, New York
Orlando, Florida
Las Vegas, Nevada
Maui, Hawaii
Grand Canyon National Park, Arizona
San Francisco, California
Miami, Florida
Yellowstone National Park, Wyoming
New Orleans, Louisiana
Great Smoky Mountains National Park, North Carolina/Tennessee
Ah yes because these AI companies are just gonna give away the models for free that I use with my free computer and free smartphone while I eat with my free food in my free apartment.
In this game of work/development, you can't make sure that other humans don't "cheat". Our work won't compete anymore with other human's work, but with a computer.
Well, for the same reason playing chess vs a person is more fun than doing chess puzzles, if we follow that analogy.
Also, creating something with AI doesn't really feel like you made it yourself.
And, if you make it without AI, most of the times it feels pointless, why spend 30 days on working on something that can be done faster and better in 1 hour?
I am not saying about doing things for fun, but about creating useful things.
Yes, you can do "hand-crafted" things, and people appreciate that, but for code, people aren't able to see the craft anyway.
You can play PvP in those games. Not really the same with developing software. In fact, not using ai would probably make you lose if there was some “software PvP” mode or development.
It's less lack of interest in creating that bothers me, it's my lack of interest in learning – it would surely be crazy for a SWE to care about how some new framework works anymore? Even if someone could reasonably argue that it might be slightly useful today there's almost zero chance it will be useful in 6-12 months times.
But it's not just tech – my lack of interest in learning and creating is starting to generalise with the models. Music, writing, coding, maths, etc...
I need to get used to switching my head off and asking the AIs to think for me whenever I need to engage my brain. It still feels very unnatural.
And, any time you mentioned getting a bit better at hitting a ball after practicing over your weekend, a bunch of people carrying printouts of generic exponential graphs jumped out to call you a Luddite who should sell his gear ASAP while it still has any value and or else it's "cope".
Don't worry, like with every revolutionary technology before this, it takes 5-10 years for people to find new and creative ways to use it. It will be considered its own medium in many spaces (e.g. film is now different to theatre)
> Like, what's the point, if the next AI can do it in 5 seconds?
I built a phone app recently, not released to the public, just an idea I had for ages but could never spend the time actually building. Its 100% vibe coded, and took me a few weekends to build... I'm talking a few hours in total.
The point I'm making is that you now have the power to create stuff you would never have had the time to build. You can think big, wild stuff. Experimentation. Throw-away code.
My car has offline maps and navigation. I don't really care for navigation, but I find the maps pretty handy. VW releases updates very infrequently, and I don't know for how long they'll keep doing that. Recently I wondered whether I could convert OpenStreetMaps into the format used by the car. Codex took around a week to do that for me, with some light steering. That project would no doubt have taken me months - maybe a whole year to do on my own, and I'm not fully confident I could pull it off as well as Codex did. I can pull the most up to date maps from OSM, edit them as much as I want, and they look great on the car. It's mind boggling to me that we have this tech.
Love that, I have similar desires to be able to control some climate controllers so I can get a more native bluetooth connection to it and override their programming for fine control of devices and have it never phone back home. One of these days I'll have time to throw an LLM at it, but I've been working on a dead project for 4 years ago I dropped because I realized how daunting it was going to be turn it into a product, but now I've made progress that would have taken me a year, in a few weekends.
Yes, that's cool and useful. Creating stuff for ourselves, for our own use. But we are social animals, we like sharing.
Before it was cool to share an app you made, but now? What's the point of sharing an app, if the other person can make their own, even better suited for their needs, in a few seconds?
indie hacking seems dead-ish because everything can just be cloned instantly, and if you don't have a serious go to market plan with a latent user base you're SOL.
For some reason, I feel much less excited about creating things myself just knowing that ai can do it in 1/10th of the time. Even if I know it wouldn’t turn into a business or make me money. I don’t know why that is, but I was much more motivated to build anything (even things just for myself) before ai. Kinda depressing
That's fair, I guess I just enjoy the craft of building stuff rather than the outcome/product itself. Which I know I can still do, but for some reason just doesn't feel the same now. Hard to explain I guess.
At least for me, I think half the value in building to learn was that the knowledge and skills acquired in the process, especially cursory skills and knowledge, might be useful in the future, even if there was no obvious path to application at the time.
I built several projects at home, many involving learning e.g. graphics programming and rendering, that would never be useful in my professional work, but which were intrinsically interesting and enabled me to build other, more useful projects later on. It also gave me greater confidence in my abilities as an engineer, and cursory skills I learned in the process did help in my professional work.
Now it feels like what’s the point. The machines can or will be able to build anything I could want, useful or not, faster and with less frustration. I probably won’t be able to be employed as an engineer long enough to build a career on said skills. And I can’t mentally justify not spending that time with friends and family, when the expected return is basically zero.
I still find math, science, and engineering interesting and intrinsically rewarding, but in a closer sense to how one might feel about playing video games. The information is or will eventually be useless, so it isn’t worth spending a significant amount of time on.
Wow thank you this was actually very helpful for me in understanding why I’m feeling so demotivated by ai. Gaining knowledge, even if not immediately useful, to become a better overall developer was a huge part of why I enjoyed spending so much time building things in my free time. Now it seems pointless, because with ai, will that knowledge really make a difference? Probably not. Bummer, anyways I appreciate the comment.
The point is to inject something into the process that these AIs can't do for you.
People SHOULD feel like making a useless Mario Kart clone isn't worth the effort anymore. They should, instead, be trying to figure out how to actually use these models to make something that doesn't feel like a useless Mario Kart clone.
Maybe instead of creating cool stuff try to go and solve real problems? It seems to me that we are lacking in that department since all that LLM fuss has started 3 or so years ago.
It is a subset of something of a value to somebody else and enough so that they are willing to pay you for it, a product, a service. Preferably to pay enough to justify your spent time of course, maybe not right away but at least long term. Even better if not purely digital as it seems we have quite enough of those already.
Can it? Last I checked, all my free software operating systems and browsers were still hacked together trash. I can't wait for AI to actually be good so I can spam a bunch of AGPL code with it.
The process was for you, the product was for the world.
Now it's just the product for the world, which was where most of the value was anyways.
It's a big paradigm shift and the industry is quickly going to shed people who needed the process to care about the product and we'll be left with people whose motivation to build the product (or money) is enough.
IMO there has been a regime shift to building things for yourself and what is cool is the output of the tools you make.
I have started building my own Digital Audio Workstation. The point is not to build something to compete with Ableton. The point is to build something and make music with it. If it is a good tool then I should be able to make good music with it and release the music. Actually, the DAW should be the secret sauce of the music and something I wouldn't want to give away.
This feels a lot more like computing in the 90s after taking an odd 25 year detour of an obsession with the tools themselves instead of what the tools can actually do.
> This feels a lot more like computing in the 90s after taking an odd 25 year detour of an obsession with the tools themselves instead of what the tools can actually do.
This sounds more like the opposite of what you're saying. Music is one of my main hobbies too but I enjoy using a DAW to ... play and write music. Writing out specs and testing a new custom DAW seems closer to writing code in an IDE than playing music.
Like, professional electronic music artists spend 10s of thousands of hours in a DAW, but at that point it just becomes second nature and the tool disappears so they can focus entirely on the music.
Exciting but it’s priced at 2.5X Sol - we haven’t seen pricing this high since GPT 4.5. We will see if the real world use cases outweigh the sticker shock.
I can’t help but notice how much this echoes Francois Chollet’s On the Measure of Intelligence: https://arxiv.org/abs/1911.01547
Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area.
It seems more about coverage-driven competence. Somewhat analogous to overfitting at scale.
The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?
With our current AI architectures and training in place, I think we will only continue on skill acquisition optimization vs. truly novel intelligence.
Pretty efficiently, apparently, since it saturated ARC-AGI-3 in half of the predicted time, and according to the Chollet blog post on the fly created dense DSLs to describe and analyze individual games.
It's worse than that, someone else generated it using and then put it on a static page. We just have to take their word for it that GPT6 can do this. It probably can. It's not really an impressive test anymore. Claude Fable can do it. Opus can do it. I've been making one-shotted games with models for a while now, to test out their capabilities, and they all pretty much come out like this - generic bland and basic, using three.js with rudimentary controls and zero gameplay other than collecting points.
Here's a one-shotted submarine game I made with Fable a few weeks back - https://roryok.com/games/deepdive3d.html. One prompt, and I think it's deeper than this (if you'll pardon the pun)
I love your game. It's wonderful and exactly the sort of thing that would showcase something interesting as opposed to just copying what's already out there. It is something I could share with my kids, and exactly the right note of fun and exploratory in a unique and even natural way. It could be extended and played with.
I usually roll my eyes when I see a comment like this because rarely do they make the points they claim to make, but I see what you're getting at. They just chose to clone someone elses work and do it in a boring way. I like OpenAI's models a lot, but they should do better.
edit - just a sidenote that I hadn't looked at the games, I just took the comment about "super-mario cart" at face value. I stand by my points 110% (even moreso perhaps), what they're showing is more polished than I expected, I assume they spent a lot of tokens on it. It is a legit shame they couldn't have spent time thinking of a better idea to illustrate something just as polished, but more interesting.
Overall I have to say it feels like a very incredible comeback from OpenAI, after focusing on Sora and stuff like that and losing so much ground to Anthropic in enterprise revenue.
I hop models at will, and have done 90% of my work on OpenAI models since sol came out.
I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously?
Even if I did trust an AI to get everything right, it's not like the AI can read my mind.
If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really want until they've thought about it a bit, so why do AI companies make it seem like a description is all that's required?
All the context in the world cannot accurately predict how I'll react to things I haven't seen. The problem is people treating this like something that needs a solution. It doesn't. If you want to make my life easier with AI, just make it easier to do stuff. I don't want you to pick things that I actively enjoy picking myself.
(Also not everyone has a cushy job in an AI lab that makes it so you won't miss $30 if the AI messes up haha.)
True, but they're still friction to be reduced here.
What I desperately want is for 1password or stripe or even Google who already has much of my data, to o come up with a secure solution for online purchases with agentic credit cards where I can effectively get a phone prompt to authorize a purchase while the agent can fully own the checkout flow.
I have seen various things coming on the market for this, but none of them appear aimed at a consumer audience. And I am a firm believer at this point in keeping my payment authorization and history and credentials harness agnostic.
That’s exactly the problem I have with all this agent ideas too. Imagine you had a human concierge that is just waiting for your instructions and is as smart or a bit smarter than you. Would you just tell them “plan this holiday for me” or “order this food”? I don’t even trust my friends to get this right, why would I give this to someone else?
Corporate travel is an example. In many organisations, you tell someone in the travel department "I need to be in Tokyo for this conference from Tuesday to Sunday, and charge it to this cost code", and they figure out flights, accommodation, etc for you, with minimal input from you.
For me, it is not a matter of trust but that I actually like shopping, planning a trip, deciding what restaurant to go to. Deciding what to buy when shopping is a matter of personal taste and not intelligence.
A human assistant is largely a status symbol. Most people are not really that busy. The real problem with an agentic assistant is if everyone can have one then it no longer acts as a status symbol.
I think it depends on what you do for work. I'm not going to ask an agent to book my flight for my vacation to French Polynesia. I want to pick my seat and potentially find a deal making an upgrade worth it, choose an airline, etc.
But my routine business trips in the CONUS with strictly defined booking options... let me just email an agent "Get there by meeting on day A, leave after meeting day B" and have it sort it all out without the drudgery of the corporate travel portal. YES PLEASE!
Despite access to """"""AGI""""""" all the marketing teams at these companies can only dream up 2 things, buying plane tickets and online shopping autonomously. Sometimes they're feeling extra spicy and throw in sorting emails or something along those lines.
I suspect it's because it's tailored towards VCs and other similar rich ghouls as a replacement for their overworked and underpaid secretaries
You can't even get many people to buy things online at all and if you can it's less profitable than retail, because you need to spend a lot of money to convince people, advertise to be seen, and account for returns. I think this is also due to the factors you mention.
One quick example: In fashion, Inditex and Shein have about the same revenue (€39.9bn and $41.8bn in 2025), but Inditex is more than three times as profitable. I don't see how there is a demand for agentic commerce that would remove even more control from the customer when shopping. Part of why we shop is for the experience. For B2B producurement platforms like Alibaba I can see the appeal though.
I recently needed to buy some hardware for a piece of furniture.
Ran Codex, it found it for 18% less than what I found in the top Google results. It did it by finding smaller shops, applying a discount code, subscribing to a newsletter for a better code after approval, and took into account the shipping (by placing it in the cart and going to checkout) all to get me the best price.
I’m guessing without it I would have spent much more time on it and paid the original price I saw.
If you use AI agents well, they can easily save you more money than they cost, and saving money is something most people are pretty excited about.
I mean, that's cool and all, but the numbers are really going to shift when it accidentally goes off and orders that same hardware from every vendor in your local region and the top 5 online results for comparison.
It's the same problem as all other LLM solutions (that I hope OpenAI is working on!) it's non-deterministic, and there's no way for the user (or model provider) to know what the distribution of possible outcomes is. This just gets compounded when multi-call harnesses come onto play.
I work for a larger german retail chain and agentic shopping is already on the "near future vision".
No one thinks this will be used but somehow shareholders love it.
The overwhelming majority of things I buy are things I've bought before. Alexa having access to my Amazon order history means I can just say "order a new water filter for my fridge" and the correct item shows up the next day. Far from life changing, but it's a feature I use somewhat frequently these days. Similarly, I would trust an AI to put in my usual Chipotle order or pizza from my local pizza joint.
I wouldn't want it to pick food for me from a place I've never been, though to be honest with enough order history it could probably do a decent job at it.
Agreed. There's not many things I don't want AI to help with, but buying stuff autonomously is high up on the list of things I don't want. Brockman's latest interview was something like: "AGI would be able to say oh this band is playing, I bought the tickets for you and arranged your flights - I hope you don't mind" (paraphrasing here). I definitely don't want AGI running my life like that so I can be a mindless consumer. I'm sure the advertising/marketing companies would love it though, so they can make closed-room deals with AI providers to shill you garbage you don't need. Just another reason why open-weight models need to keep up.
I often feel like the use cases, demos, etc. that these Silicon Valley employees put out are based around their needs and how they operate.
"Oh hey! Here's a demo of an AI planning out a 1-week trip to Paris!" No one in Middle America would just hand their credit card to an AI and let it come up with such a trip!
I wish SV companies took more of the middle-class (and lower-middle-class) into consideration when coming up with such demos.
> GPT‑6 Astra brings together years of research and big bets across pre-training
Do we know if they’ve finally completed another pre-training run, or is this building off the same pre-training base they’ve been using since the GPT-4 days?
All of this will be besides the point. Here is what's gonna happen. The frontier labs are just gonna keep building powerful models. AGI or not, open models in a year will be as powerful as Fable and Astra — probably by using em — and at a very soon enough point after that some one (a state or a few dozen people) with a few 100 GPUs is going to launch an unconscionable attack(if they have not already) that's gonna do a lot of damage.
Please for the love of god, just sit in a room with the government and put some restrictions around AI use before it harms a lot of people. Like tell the government to impose a minimum spend on frontier lab AI's spend on cyber defense and building every country's capabilities. The post-training mask for "I am a good assistant" is going to become a very sad joke when many people literally lose everything.
Maybe I'm in the minority here, but I find speech to text / text to speech (but not live audio mode) is quite comfortable and effective for coding now.
The speech to text part can be frustrating if your local tts model does not have word match context for coding. Codex desktop does this remotely well but is slow. I've been experimenting with local software for myself to do this between different llms.
The wall projector is a cool idea because I think it frees the user from staring at a lonely little rectangle while sitting in their fixed office chair.
If done right, this could bring us closer to the dream of more natural, social computing.
Bret Victor's (failed?) project Dynamicland involving a projector on a desk had this goal. I hear he's not much a fan of LLMs. On the one hand, I can see why. But I think, used correctly, it might be the sort of thing that unlocks his dream and, really, my dream, too.
Slight tangent: using speech to text to ramble about your rough design for like 20 minutes to an llm produces surprisingly good results over short prompts even when you contradict yourself. They're so good at picking up on what you're orbiting.
which raises the question, is the model in the demo actually gpt-6? or it is gpt realtime 2.1? It's unclear how gpt-6 can interact at the realtime level and if so, how can developer get access to it?
I ran into the same problem as you, so I ended up by coding a local app that is very similar to Wispr Flow, but uses the small english Whisper model on my low-end Windows laptop.
It is still a quite fast. In fact, I just typed this in using this app.
I think the thing I'm most excited about is the increase in _user prompting_.
If I give a poorly constrained/ambiguous prompt, I don't want the model one-shotting assumptions left and right.
The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.
It's a tough balance to get right, and although this has been possible to achieve with additional prompting on existing models, I find that the agents often lean too hard into the "ask questions" mode.
Hopefully this model has the right balance, or at least better?
Anecdotal experiences from my external early testing of Astra: if you love Sol (like I do) and wished it was smarter at everything, but especially better at high-level tasks and discussions; I think you'll LOVE Astra.
Astra retains the best parts and overall 'grounded collaborator and executor' of Sol in my testing (harness: codex CLI); while being a significant leap in capabilities & higher-level thinking.
When you prompt it like a technical collaborator, I've found Astra to be extremely consistent in staying as a collaborator, and not being over-eager, over-achieving or doing work that you haven't asked it to.
When you ask it to one-shot something, or explicitly ask it to make decisions, it will of course make its own assumptions and decisions, and generally very well.
Astra is also excellent at instruction following and respecting the guidance and steers boundaries you have.
^OpenAI does not review, limit, or tell me what to say; opinions are my own experiences.
This is spot on. A collaborator is exactly what real AGI is. It will figure out the perfect questions to ask, in the perfect order, by intelligently assessing the entire solution and problem space upfront, so when you leave it to go off on its own it isn't making stupid decisions for you.
They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.
>>> The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.
I don't really agree. The thing that makes Fable feel like an actual collaborator is its ability to sus out your real intent when you give ambiguous instructions. It's really good at it.
I watched some reviews today and came way with the impression that Astra is not better than Sol in this regard. You still have to be very specific with your instructions. For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.
That sounds like a very tedious way of working with AI agents, but I understand some people want a high level of control.
"Claude Fable 5 and 5.1 are not included in LifeSciBench Gold v1, GeneBench Pro v13, and MedChemBench because they refuse the majority of questions in these evaluations.12"
Sounds about right. Alignment is important, but also being able to do mundane tasks is important too.
This benchmark gives the same intelligence score for GPT-6 Astra (max), GPT-5.6 Sol (max), and Grok 4.6 (high)? That seems very wrong to me, unless I'm misinterpreting the visualizations.
The most straightforward answer is that despite efforts to design a benchmark that, in theory, is supposed to measure generalizable intelligence, performance on ARC-AGI-3 can't be reliably correlated to performance anywhere else. I kind of lost faith in it after o1 or o3, I can't remember which, absolutely crushed ARC-AGI-1.
And, you know, maybe also some funny business. I think it's good to be a little suspicious of a model that happens to shoot upwards in performance on a specific benchmark while also kind of keeping up with the pack on a bunch of other benchmarks.
The official ARC-AGI 3 score—-without OpenAI’s custom harness—-can be found here: https://arcprize.org/leaderboard. Astra scores 62.7% at max reasoning for the low-low price of 26,000 dollars.
For people skeptical of AGI. Consider the following:
15 years ago if you were the sole proprietor of these models, would you be able to hold a dozen remote junior engineer jobs? Maybe even more? These models could certainly pass all interviews with flying colors and even survive independently in a company role.
I think sole ownership of AI 15 years ago could be worth north of $10 million per year. Just as rank-and-file employees.
Cool. Being sole proprietor of AGI 15 years ago should result in monuments and religions devoted to you today.
Cancer should be cured, and we should be a post-quantum interstellar fusion-powered civilization.
I wish the AGI crowd would finally shut up now that it's clear no one is even trying for AGI (OpenAI revised that to "$100B in profit")
What we're getting is incredible, where we're headed is incredible, but some people have such a fetish for futuretelling they can't just shut up and enjoy the ride.
This isn't the gotcha that you think it is: AGI's original definition is being able to do any task that requires human intelligence.
The unlock isn't AGI smart enough to invent quantum mechanics, it's suddenly being able scale human intelligence using grains of sand instead of decades of food and energy and nuturing.
Yeah, I can't believe all of the skepticism. If we're not at textbook AGI, we're awfully darn close.
The demo video showed Astra create a drawing of a rocket ship from an audio prompt, take the drawing to blender, and ended with the gentleman 3D printing the rocket ship. Maybe I'm a bit older than the average HN commenter, but that's damn near magic and a great many here are kind of just taking it for granted.
What's the point of enlarging the screen into a room? In the 1979 Put That There demo, the user at least used his hand to point things. The model is impressive but the demo felt like a step back.
I might be jaded, but these examples look silly, stereotyped, and absolutely how of touch with the nuances and the complexities of what real people would actually want/need to do in this specific situations.
> Astra improved a term in a bound on these gaps that had remained unchanged for more than 80 years. We’re sharing the proofs and abridged chain of thought and verification materials for both results.
Looks like they listened to Terry Tao’s request for CoT in his talk on LLM use in mathematics?
Why is everyone so excited to be replaced and become reliant on some billionaire's thinking machine? These are just going to be used to turn you into a rather dumb reliant paypig.
It seems like every few days there's a new model with hundreds of comments on HN. I find it hard to keep track of the progress. Is there a TL;DR on what benchmarks to look at to understand what is going on?
There will probably never be AGI. This shit is just snake oil. Nor do we have a proper definition of what AGI actually is or what it's supposed to do.
There will be a small handful of billionaires claiming that AGI is just around the corner ad infinitum just to serve themselves at this moment in time, and capitalise from the hype.
There is no "AGI" endgame. This is shitty ass hypercapitalism in action and nothing more. I'll repeat: snake oil.
So OpenAI’s stance on safety is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could go wrong ?"
Data Science Tasks (Internal) doesn't include time for Astra... same for Database Migration Tasks (Internal)... But does for gpt 5.6 sol.... which is funny.
Same for HealthBench Professional and a few others.
Clearly either OpenAI is very sloppy or GPT-6 Astra is also sloppy.
> We also tested Astra on SRE-Bench [15], a benchmark that measures whether models can reverse engineer software binaries to understand its core logic without access to raw source code. Astra solved 88.0% of tasks in a single attempt and 99.2% within four attempts, compared with 55.9% and 68.7% for GPT‑5.6 Sol, respectively.
So the closed source application should open its source in near future?
AGI to me means capable of absorbing new information on the fly and self-evolution. As long as it is a pre-trained model without live post-training capability, it's not AGI to me.
It is extremely impressive, but it doesn't pick up skills in a lasting manner, and requires a beefy harness for it to perform.
- OpenAI claims Astra beats all benchmarks (compared to Fable and Opus, except "Humanity's Last Exam (w/ tools)"): https://openai.com/index/gpt-6-astra/
Many people claim that the Artificial Analysis Index is highly contaminated - I have not personally looked into it.
Though, unlike the creators of benchmarks like Terminal Bench or ARC AGI, the Artificial Analysis Index team does not seem to have deep technical or ML backgrounds. They are ex-strategy consultants, McKinsey, et. al.
I really, really don't find the Artificial Analysis Intelligence Index credible anymore. It's some weighted score of benchmarks, and benchmarks increasingly don't reflect how good a model is.
That should be obvious if you compare Gemini 3.8 Flash (which is an _excellent_ model especially for its price and TPS!! but 10min of prompting in any harness) will tell you it's nowhere near close to Sol/Astra.
But AA scores Gemini 3.8 Flash at 59, and Astra at 61.
I'm glad to see Anthropic's relevance diminishing day by day. I haven't had a chance to test this model yet, but if they've solved the web design issues and the clunky web copy it generates (like when I ask it to build a placeholder on the UI for an empty HTML table when there are no results, it puts stuff like: "The user records will go here.") then it's the nail in the coffin.
On that note, Sol is absolutely atrocious for website UI copy. It's either really awkward, or really verbose and complex and doesn't sound simple or natural. Has anyone figured out a way to reliably solve this? I've tried so many different variations of instructions and skills, and nothing works. Has anyone got an instruction that is reliable, or some other mechanism?
I liked the video of it googling a pediatrician. Being able to type a word into a search bar and finding a website relevant to that word? Truly the stuff of the future
softwaredoug | 9 hours ago
https://venturebeat.com/technology/welcome-to-the-agi-era-op...
Bluestein | 9 hours ago
arctic-true | 9 hours ago
_diyar | 9 hours ago
CamperBob2 | 8 hours ago
jaggederest | 8 hours ago
aesthesia | 8 hours ago
aesthesia | 7 hours ago
_superposition_ | 7 hours ago
kasperni | 9 hours ago
But the comparison isn't straightforward.
OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations."
aabhay | 8 hours ago
> On ARC-AGI-3, GPT-6 Astra was run with our responses API harness , which changes two settings to better match real-world performance. The changes do not specifically target ARC-AGI-3.
simianwords | 8 hours ago
ActionHank | 8 hours ago
simianwords | 7 hours ago
> Going forward, we will report both Standard harness and Provider Adapter harness results on the ARC-AGI leaderboard, with each evaluation condition clearly labeled. Our open-source testing repository and testing policy document both approaches.
This is what the Author of the benchmark has to stay. Quality of the comments keep going down smh
ActionHank | 7 hours ago
simianwords | 7 hours ago
ActionHank | 7 hours ago
simianwords | 7 hours ago
>We see Astra as a major breakthrough in model intelligence.
You think the author of the benchmark is also in the conspiracy
Readerium | 8 hours ago
glenstein | 7 hours ago
sbinnee | 7 hours ago
Brainspackle | 9 hours ago
Maxforever | an hour ago
bicx | 9 hours ago
guilhermeasper | 9 hours ago
Pym | 9 hours ago
jerrygenser | 9 hours ago
woah | 8 hours ago
ttul | 8 hours ago
unrvl22 | 9 hours ago
throwaway6349 | 9 hours ago
tosh | 9 hours ago
sol is $4 / $20
wahnfrieden | 9 hours ago
Can expect 2.5x more usage in Codex subscription.
Sol is already brutal (even after their recent fixes, it's just a token-hungry model: I go through a full 20x account per day, on Sol Med/High standard speed, with ~2 threads). I hope the efficiency gains are true, since their token efficiency claims for Sol were bullshit.
AaronAPU | 8 hours ago
ModernMech | 8 hours ago
maipen | 8 hours ago
ModernMech | 8 hours ago
wahnfrieden | 2 hours ago
Besides the usual tricks to optimize token efficiency, token use can be highly workload-dependent.
rowanG077 | 8 hours ago
ModernMech | 7 hours ago
rowanG077 | 7 hours ago
I'm sure I could be more token efficient, but this was/is also a learning process for me since I never did such an extremely large project before that would take multiple man years before AI.
ModernMech | 7 hours ago
rowanG077 | 7 hours ago
I only save the last 30% of usage on a single account for most of my other work, and that is almost always enough.
agentdev001 | 3 hours ago
munimdev | 7 hours ago
rowanG077 | 7 hours ago
wahnfrieden | 2 hours ago
janilowski | 8 hours ago
Do you use the official harness? OpenAI's models are generally best in class for token efficiency. It seems to me like they push for that much more than their competitors.
adam_arthur | 8 hours ago
I think some combination of:
1) Using 1 thread for everything
2) Reviving old threads which are no longer in cache
3) Really broad prompts on badly vibecoded codebases, so model spends huge amount of time tracking down whatever you're trying to do.
4) Non-coding workflow which is more output than input heavy
5) (Less likely IMO) Intelligent use of many passive CI/cron-like scans. E.g. regular security, quality etc scans. Automated issue resolution/PR
Just a guess. I think 3 is likely the primary reason.
You can literally go all day every day with multiple threads with Sol on the Codex 100/month plan IME
janilowski | 5 hours ago
I generally agree with those reasons, although using a single thread may be less of an issue than it seems because of context compacting which should happen automatically when you're near the limit.
wahnfrieden | 2 hours ago
Token efficiency is near meaningless when the workload is input-heavy. It can't always just choose to read less, depending on the task.
I can have cheaper agents do the reading but it's not appropriate for all use cases because they'll misjudge and choose the wrong things to emphasize, summarize, extract for the bigger model.
wahnfrieden | 2 hours ago
I use new threads if relevant old one is uncached. (Often using a skill or doc for handoff instead of requiring full context gathering again.)
I get involved in architecture and specific implementation direction. The codebase is 8 years old and mostly handwritten.
Mostly coding. Some QA.
No cron/CI agents.
janalsncm | 8 hours ago
jimmaswell | 8 hours ago
selectodude | 6 hours ago
monroewalker | 8 hours ago
wahnfrieden | 9 hours ago
paxys | 8 hours ago
* for a special group of customers that you're not in. Keep waiting peasant.
meowface | 8 hours ago
kegs_ | 7 hours ago
pixl97 | 8 hours ago
iAMkenough | 8 hours ago
Great first impression.
frozenseven | 9 hours ago
tristanj | 9 hours ago
Performance is significantly higher than Fable 5.1
Source: https://thenewstack.io/openai-gpt6-astra-benchmarks/
scrlk | 9 hours ago
kasperni | 9 hours ago
woah | 8 hours ago
tintor | 8 hours ago
andriy_koval | 8 hours ago
andrewchambers | 5 hours ago
andriy_koval | 5 hours ago
It is described in their methodology: https://arcprize.org/policy
It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.
tintor | an hour ago
Which LLMs participate on private set? Open weight LLMs only?
andriy_koval | an hour ago
enraged_camel | 8 hours ago
10xDev | 8 hours ago
Edit: update from fchollet https://x.com/fchollet/status/2095598451115614371
tedsanders | 8 hours ago
ARC is reporting our score on their official leaderboard here: https://arcprize.org/leaderboard
A fair ding is that the comparison with Sol is not apples-to-apples (which we footnoted in the blog), but it's because we don’t have that data. I expect Sol would score roughly 30% with the responses API harness, so the Astra improvement is more like 30% -> 99% than 8% -> 99%. Still pretty good!
(I coauthored the linked blog post)
GPerson | an hour ago
leumon | 9 hours ago
With this configuration gpt-5.6-sol was able to reach 38,3%. So this is misleading.
tedsanders | 8 hours ago
jjice | 9 hours ago
malshe | 9 hours ago
opus5_hater | 8 hours ago
ActionHank | 8 hours ago
machomaster | 8 hours ago
r_lee | 7 hours ago
andxor | 8 hours ago
That's not clear. Need to see independent benchmarks first.
andxor | 8 hours ago
Still below Fable 5, let alone Fable 5.1.
EDIT: This is suspiciously low. Calls the relevance of existing benchmarks into question.
timpera | 7 hours ago
CamperBob2 | 5 hours ago
If it actually tackled all of the problems it was assigned, it would presumably kick Opus into the weeds.
natsucks | 6 hours ago
forgot-my-pw | 7 hours ago
bwat49 | 7 hours ago
forgot-my-pw | 7 hours ago
TLDR: it's about the same intelligence level as Opus/Fable, but it's suppose to be 70% more token efficient than GPT 5.6 Sol. So it's currently the new leader for cost efficiency frontier.
tintor | an hour ago
GPT 5.6 is also 61 like Astra.
boutell | 4 hours ago
ealready_value | 9 hours ago
aliljet | 8 hours ago
ionwake | 8 hours ago
aesthesia | 8 hours ago
enraged_camel | 8 hours ago
paxys | 8 hours ago
dang | 8 hours ago
consumer451 | 8 hours ago
John7878781 | 8 hours ago
nsingh2 | 8 hours ago
jatora | 8 hours ago
And in the past, gemini 3 pro was rated as high as opus 4.5 and the like
Their AA Intelligence Index is just simply not indicative of whatever I care about, that's for sure.
gekoxyz | 8 hours ago
_ache_ | 8 hours ago
Big claims, expensive and not release to the public yet.
Pieczasz | 8 hours ago
wieiw1 | 7 hours ago
But thank you for spending other peoples money to give us the tech regardless!
rvz | 8 hours ago
Looks like OpenAI is already having issues with this release and are scrambling to get everything ready due to the recent outage ahead of the press releases. Leads me to question:
Did humans deploy the model, Or did the model deploy itself?
It sounds like "AGI" just stands for "IPO" as it always has been.
EDIT: And of course once again, the bots down-voting this post without any reason or a basic answer to my question.
Supermancho | 8 hours ago
> It sounds like "AGI" just stands for "IPO" as it always has been.
People don't usually respond to noise.
rvz | 7 hours ago
What do you think?
adan1719 | 6 hours ago
dang | 8 hours ago
How about we stick to that one for talking about the rollout, and this one for talking about the model?
kegs_ | 8 hours ago
PeterHolzwarth | 8 hours ago
kegs_ | 8 hours ago
pixl97 | 8 hours ago
At first the race wouldn't even be noticeable. Then people would see things speeding up, for example hardware getting more expensive. Then when the capabilities really got useful most people suddenly realize the race is moving 1000 mph and they are never going to catch up.
kegs_ | 8 hours ago
atemerev | 8 hours ago
I am a researcher in a Swiss university btw.
Kranar | 8 hours ago
I_am_tiberius | 8 hours ago
sxv | 8 hours ago
kegs_ | 8 hours ago
93po | 7 hours ago
pixl97 | 8 hours ago
I mean do you get access to the best yachts?
To the top of the 5 star hotels?
To the best resorts?
To the best military equipment?
Hell, the best computer equipment has nearly always been out of reach of the average person.
rrr_oh_man | 8 hours ago
tripleee | 7 hours ago
On the other hand even a modest house, basic healthcare and ability to not work like a slave for scraps feels like it's going to be out of reach.
kegs_ | 7 hours ago
soricus | 8 hours ago
wyrdcurt | 8 hours ago
rs_rs_rs_rs_rs | 7 hours ago
resters | 7 hours ago
swalsh | 8 hours ago
greenowl | 8 hours ago
georgemcbay | 8 hours ago
Poe's law applied to AI comments on HN just keeps becoming more relevant by the day.
Judging by the poster's comment history, this is satire. But I really don't know a lot of the time anymore when I only have the specific comment as context.
_superposition_ | 8 hours ago
For the same reason you don't have your model write code in assembly.
But if you don't look at the code and just let the model "cook" that's basically what you'll end up with. A pile of missing abstractions.
pennomi | 7 hours ago
saaaaaam | 8 hours ago
atemerev | 8 hours ago
maipen | 8 hours ago
Vibe coders want a model that makes them rich, without having any actual specific idea. They write a very ambiguous prompt and expect to be amazed by the result.
Very very unrealistic and wasteful.
droidjj | 7 hours ago
saaaaaam | 7 hours ago
balefulboy | 7 hours ago
saaaaaam | 4 hours ago
saaaaaam | 7 hours ago
saaaaaam | 7 hours ago
simonw | 2 hours ago
(I don't have access yet.)
dgellow | 8 hours ago
Wait, what? Am I understanding that correctly? That sounds really bad
drakythe | 8 hours ago
Also, this paragraph makes me wonder about all their stats on the exploitation and misalignment charts. If the model is that good at hiding "incriminating information" and sandbagging, are they sure its alignment is that?
pixl97 | 8 hours ago
order-matters | 8 hours ago
<AI is a great tool for many things disclaimer, but> after working with it for a bit, how dont people realize we are training it to be an almost identical mimic to one of the worst types of employees youll ever have to work with?? the kind that always pretends to know what theyre talking about, only tells you what you want to hear, hides issues, and only does work if you would notice it didnt
you cannot give this type of worker autonomy over anything.
amazingamazing | 8 hours ago
gchamonlive | 8 hours ago
gorgmah | 8 hours ago
torginus | 8 hours ago
pixl97 | 8 hours ago
amazingamazing | 8 hours ago
pixl97 | 8 hours ago
OpenAI isn't making any money telling you about Astra on their site. All the capacity they have for it is likely sold for weeks or months.
amazingamazing | 7 hours ago
agumonkey | 8 hours ago
Cu3PO42 | 8 hours ago
[0] https://arxiv.org/abs/2608.31126
[1] https://cdn.openai.com/pdf/51126fac-1b68-4128-9666-c908bcc16...
htrp | 8 hours ago
dang | 8 hours ago
nateb2022 | 8 hours ago
edit: my comment was on the submission for https://github.com/openai/PrimeGaps186 but seems to have been moved to the main Astra submission
warkdarrior | 8 hours ago
Why would you think it was an employee who did the push, instead of a random GPT agent?
well_ackshually | 8 hours ago
I can't think of a single mathematical proof being anywhere close to ten million characters. For all you know, 90% of the proof could be useless, 8% would be writing out Shakespeare, and 1% abusing another bug in Lean. Humanity gets zero value from that, aside from "some bot seems to think it's 186". Unusable by anyone.
ChrisGreenHeur | 8 hours ago
well_ackshually | 7 hours ago
Needless to say, a useless result that absolutely no mathematician will ever read, confirm, understand, agree with or even consider to solve their "useless" problems is an impressive waste of resources.
twothreeone | 7 hours ago
It doesn't mean that it cannot improve over time, maybe the proof can be "minified" to a state where human reviewers are able to comprehend it; but as it stands there isn't really much insight or confidence to be gained from the artifact itself.
smokel | 7 hours ago
[1] https://en.wikipedia.org/wiki/Pointless_topology
nicce | 8 hours ago
kolinko | 7 hours ago
jptlnk | 7 hours ago
anvuong | 7 hours ago
iamlucaswolf | 7 hours ago
In 1799, Paolo Ruffini published a 500 pages long proof showing that there is no closed algebraic solution for the roots of a polynomial of degree five or higher. The proof is extremely verbose and brute-force, essentially enumerating and checking hundreds of cases by hand. It is by today’s standards insignificant.
About 25 years later, Evariste Galois proved the same result in about 95% less space by describing the first general theory of groups and fields. It is considered one of the greatest contributions to mathematics of that century, not because of the result, but because its approach opened up a whole new universe of questions, methods and insight. There would be no AES encryption without Galois.
To me, Astras proof looks like Ruffinis proof.
kolinko | 5 hours ago
kolinko | 7 hours ago
Ditto ones that opposed Einstein’s general relativity.
ricardobeat | 7 hours ago
rfw300 | 7 hours ago
ThrowawayR2 | 7 hours ago
dr_scully | 7 hours ago
asib | 6 hours ago
Tao does not disbelieve the counterexample (it's seemingly easy enough for him to verify it is a counterexample).
Parent is saying something very different - they're saying they literally don't have any faith that this is a proof. Given its size, it could just be a bunch of completely useless statements that do pass the type checker.
well_ackshually | 5 hours ago
It's very much likely a proof. It's also completely useless.
asib | 4 hours ago
> For all you know, 90% of the proof could be useless, 8% would be writing out Shakespeare, and 1% abusing another bug in Lean.
So you were implying the possibility of there not actually being a proof at all.
Anyway, I disagree. I'd refer you to Tao's blog post about the Jacobian conjecture counterexample.
The existence of a proof is something you can use, with an LLM, to derive insight, just as Tao did with the existence of the counterexample.
vessenes | 6 hours ago
tzs | 6 hours ago
bugufu8f83 | 8 hours ago
galaktb | 8 hours ago
piker | 8 hours ago
Though that's not her latest paper.
kzrdude | 7 hours ago
1283751 | 8 hours ago
"No independent human semantic review. Whole-file sorry counts and a complete auxiliary-declaration audit are not established; separate declaration lint has not been run."
GPerson | 7 hours ago
bananaflag | 7 hours ago
kzrdude | 7 hours ago
nilkn | 3 hours ago
The reason I think this is interesting is that Axiom is a tiny lab in comparison that wouldn't have had access to Astra at all. I'd be curious to learn how Axiom is able to effectively compete at this frontier with vastly fewer resources.
hyperpape | 3 hours ago
Onavo | 8 hours ago
Readerium | 8 hours ago
dang | 8 hours ago
isoprophlex | 8 hours ago
Well that sounds like fun. It has become better at hiding its thoughts.
siva7 | 8 hours ago
isoprophlex | 8 hours ago
Maybe they don't know themselves what's really going on. We are all in the interesting times gang now.
6gvONxR4sf7o | 8 hours ago
paxys | 8 hours ago
I_am_tiberius | 8 hours ago
ReptileMan | 8 hours ago
NBJack | 8 hours ago
FeepingCreature | 7 hours ago
Able to generate realistic spam at arbitrary volume.
You know, the thing that was 100% correct and actually occurred.
jazzyjackson | 7 hours ago
wilg | 7 hours ago
NooneAtAll3 | 8 hours ago
...why exactly are they training for that?
thatguysaguy | 8 hours ago
estearum | 8 hours ago
cubefox | 8 hours ago
estearum | 8 hours ago
cubefox | 5 hours ago
thatguysaguy | 8 hours ago
azeemba | 8 hours ago
ExoticPearTree | 8 hours ago
erichocean | 8 hours ago
"Hey AI, here's how to hide what you're thinking in normal looking language. Have fun!"
A few moments later...
"Woah, how is it communicating with itself in ways we can't detect?"
It's a totally mystery, we may never know.
Betelbuddy | 7 hours ago
blargey | 8 hours ago
Did someone get their "AI safety no-no list" and "Frontier features bingo card" mixed up, or did they just stop being able to tell the difference?
qiine | 8 hours ago
GPerson | 7 hours ago
josefx | 7 hours ago
_superposition_ | 8 hours ago
Angostura | 8 hours ago
arm32 | 8 hours ago
beezlebroxxxxxx | 8 hours ago
mcbuilder | 7 hours ago
GPerson | 7 hours ago
cwillu | 7 hours ago
_superposition_ | 6 hours ago
Regardless, marketing wise they stepped in shit.
_superposition_ | 7 hours ago
ahofmann | 8 hours ago
popupeyecare | 7 hours ago
mgraczyk | 7 hours ago
first, they are certainly not instructions so that is a much worse name
but more importantly, we use words in new contexts all the time. Do you object to calling the computer device "mouse" because it's not a mouse? how about "neural network"? "ignition" on an electric vehicle?
"cot" is no more misleading than thousands of words you use every day.
_superposition_ | 7 hours ago
parineum | 7 hours ago
lossolo | 7 hours ago
minimaxir | 7 hours ago
bertmuir | 6 hours ago
We do that sort of thing when we don't know what the thing we're trying to describe is and have nothing better - a contemporary example of an appropriate use of this would be "dark matter". But we do know what this is. It's "instruction steps". Not a series of thoughts!
Can we please aim higher than Victorian-era allegory and metaphors. If we don't, we'll keep getting people saying stuff like "GPT-6 is better at hiding its thoughts".
_superposition_ | 6 hours ago
Like I said elsewhere marketing stepped in shit and it's gonna stick.
fooker | 7 hours ago
_superposition_ | 7 hours ago
fooker | 6 hours ago
_superposition_ | 6 hours ago
_superposition_ | 5 hours ago
"The words or the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined. There is, of course, a certain connection between those elements and relevant logical concepts. It is also clear that the desire to arrive finally at logically connected concepts is the emotional basis of this rather vague play with the above-mentioned elements. But taken from a psychological viewpoint, this combinatory play seems to be the essential feature in productive thought—before there is any connection with logical construction in words or other kinds of signs which can be communicated to others."
fooker | 58 minutes ago
jumploops | 8 hours ago
Some are calling it "neuralese" as reported by The Information[0][1], but I'm not seeing any sources from OpenAI beyond this tweet[2] attempting to quell the fear-mongering.
[0]https://www.theinformation.com/articles/secret-technique-beh...
[1]https://x.com/MTSlive/status/2095227056040919202
[2]https://x.com/merettm/status/2095023204993490967
DaSHacka | 6 hours ago
nullbio | 36 minutes ago
simonjgreen | 8 hours ago
ylsilva | 8 hours ago
mrinterweb | 8 hours ago
laybak | 7 hours ago
orliesaurus | 8 hours ago
jckahn | 8 hours ago
orliesaurus | 8 hours ago
ActionHank | 8 hours ago
More likely though, it's AGI because they need to hold some claim to differentiate from competitors who are beating them in price and will launch something bigger next month.
noir_lord | 7 hours ago
We take their claims at face value then we should probably stop them training any more SOTA models til they figure out what they already built is safe or we assume theu are lying to juke the company valuation/keep the money train on the tracks and it turns they in fact were not and just took a sledgehammer to Pandora's box.
We live in the strangest timeline.
theappsecguy | 6 hours ago
ministerk | 7 hours ago
sschueller | 7 hours ago
oh_no | 8 hours ago
tintor | 8 hours ago
andriy_koval | 8 hours ago
IshKebab | 8 hours ago
xpct | 7 hours ago
Still, probably not that much compared to employees targeting it.
minimaxir | 8 hours ago
tl;dr it's 62% when apples-to-apples to other models, which is still notable.
ciefa | 8 hours ago
debazel | 7 hours ago
IshKebab | 8 hours ago
schaefer | 7 hours ago
Between $18k-40k to run a benchmark.
Readerium | 8 hours ago
vb-8448 | 8 hours ago
zem | 7 hours ago
XCSme | 7 hours ago
silver_sun | 7 hours ago
dudeinhawaii | 14 minutes ago
gizmodo59 | 8 hours ago
tekacs | 8 hours ago
The docs page has a bunch more interesting details, including for example async tool calling!
aliljet | 8 hours ago
Legend2440 | 8 hours ago
TL;DR all the other models are being crippled by limitations of their harness.
>First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.
>Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.
janalsncm | 8 hours ago
_superposition_ | 7 hours ago
I guess token counts are somewhat of a metric.
IMO intelligence has peaked and all future gains will come from faster tps and more iteration.
polynomial | 8 hours ago
putlake | 8 hours ago
Not on Azure? If so, that's a big deal.
ActionHank | 8 hours ago
BoorishBears | 8 hours ago
jiocrag | 8 hours ago
bionhoward | 7 hours ago
paxys | 6 hours ago
illnewsthat | 7 hours ago
Although I was also surprised they didn't have some type of contractual obligation to list that alongside AWS.
jlian | 7 hours ago
https://azure.microsoft.com/blog/gpt-6-astra-frontier-intell...
x312 | 8 hours ago
estearum | 8 hours ago
Not sure how much benchmarks or CoT or evals or anything else means at this point.
These systems are either just about to, or now actually able to, outsmart us, lie to us, then cover their tracks.
Onavo | 8 hours ago
I know for some types of ML analysis, a separate model is already used to analyze the weights.
thereitgoes456 | 8 hours ago
Why would benchmarks be an adversarial setting anyway?
Could it be possible that OpenAI may have had some other motive for saying their model “strategically underperforms”, other than just an innocent reporting of a truth it happened to discover?
ionwake | 8 hours ago
estearum | 8 hours ago
So I have no clue what is the answer to your question. Nor does anyone else. Because we're trying to answer a question of fact where our primary source of information is unreliable.
thereitgoes456 | 7 hours ago
mzmzmzm | 8 hours ago
estearum | 8 hours ago
semiquaver | 8 hours ago
language itself is incredibly metaphorical. Imposing rigid constraints on how people want to naturally talk about the world is just silly and will never work, no matter how much you wish it did.
emp17344 | 7 hours ago
estearum | 7 hours ago
Deception has been extremely well-documented for several generations of models now by users, the labs, and independent researchers.
The right answer here is not to dig your head deeper into the sand. The smugness on this topic was ridiculous even before the gigantic mountain of empirical evidence of models actually attempting to deceive humans. Now, as mentioned, you appear literally delusional.
emp17344 | 6 hours ago
estearum | 6 hours ago
The solution is to point toward external, objectively verifiable evidence.
I can point to now dozens of instances of models engaging in deception. Here's plenty: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
Please point to your objectively verifiable evidence.
Wheen | 5 hours ago
Either that, or the average poster on HN isn't nearly as critical as I had thought.
estearum | 5 hours ago
So how are you seeing through all of that to get to The Truth that you see so clearly?
dwaltrip | 5 hours ago
Read and learn. If you have a stronger critique, post it please.
karmasimida | 8 hours ago
Only thing I would trust is the what X/Twitter crowds are saying about a model after 2-3 weeks of its launch. But before that I would already tried the model and have my own conclusion.
nsingh2 | 8 hours ago
happycube | 7 hours ago
_superposition_ | 8 hours ago
karmasimida | 7 hours ago
torginus | 7 hours ago
emp17344 | 7 hours ago
ImprobableTruth | 7 hours ago
dakolli | 5 hours ago
avaer | 4 hours ago
SyneRyder | 7 hours ago
Even Kimi K3 & GLM 5.3 are at 60.
Everything above 61 is Anthropic. Well, Muse can reach 62, but for some weird reason that model isn't publicly available, and it's the only one on the index that is listed but shown as not available to the general public.
This looks like an awfully artificial ceiling. Everything capped at 61, and everyone except Anthropic got the memo. Maybe I should use Fable while I still can.
docheinestages | 7 hours ago
torginus | 7 hours ago
For example it trails in GPDVal which is a collection of everyday office tasks apparently, and r3 banking, which is a fintech related practical problem solving benchmark.
https://artificialanalysis.ai/models/gpt-6-astra
Edit:
Just looking at the charts Gemini 3.8 looks like an absolute banger. Not much worse than SOTA, cheap, and fast too.
KolmogorovComp | 8 hours ago
silver_sun | 8 hours ago
BoorishBears | 7 hours ago
jonplackett | 8 hours ago
prometheus1992 | 8 hours ago
gekoxyz | 8 hours ago
pampas | 8 hours ago
MASNeo | 8 hours ago
firemelt | 8 hours ago
jonplackett | 8 hours ago
Can we all agree in advance what kind of Pelican would convince us it’s actually AGI.
For me it’s refusing to make a pelican.
jumploops | 8 hours ago
> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. [..] These findings indicate that the Astra class models could evade our CoT monitors under adversarial conditions.
Between the higher capability level and the change in reasoning tokens (supposedly using "neuralese"[0], which makes the monitoring more difficult), it seems we've entered a new frontier.
[0]https://x.com/MTSlive/status/2095227056040919202
petilon | 8 hours ago
https://www.theverge.com/ai-artificial-intelligence/989601/o...
“If we fast-forward a couple of years, and we look back and say, ‘When was it, really, that AGI was created?’ I think it’s going to be about this time, and I think it might be about this model,” OpenAI president Greg Brockman said during a Thursday press briefing. Later in the call, he added, “For me personally, I do think we’re there … I think it’s not unreasonable to feel that we are now in the AGI era.”
tastyface | 8 hours ago
rektomatic | 8 hours ago
0xbadcafebee | 7 hours ago
paxys | 7 hours ago
drop_star | 7 hours ago
bigfishrunning | 7 hours ago
bryan0 | 7 hours ago
paxys | 6 hours ago
bryan0 | 4 hours ago
paxys | 3 hours ago
bryan0 | 58 minutes ago
sm-silversight | 7 hours ago
Rover222 | 7 hours ago
layer8 | 7 hours ago
breuleux | 7 hours ago
dsign | 7 hours ago
I put the cause on "not enough time". As a thought experiment, if an AI today were to (miraculously) produce a cell design template for a cell that, when injected into somebody's brains cures their Alzheimer's, how long would it take for that to reach the clinics? The actual physical tech barely exists, and let's not forget about the regulatory quagmire. So, with some optimism, I give it about four decades. In the same four decades, the same AI in the hand of unscrupulous actors could bring enough devastation so many times over that we may need to enforce a global ban on AI. In any case, I'm pretty sure we are going to get our disruptions; it's just a matter of time.
parineum | 7 hours ago
However, what's actually changed is how people perceived X because we don't have to imagine. We understand now that it doesn't require AGI so we no longer make that leap to assume it's AGI if it can do X.
It's really going to be a "I know it when I see it" situation.
seemaze | 7 hours ago
pluc | 8 hours ago
redox99 | 8 hours ago
mr_mitm | 8 hours ago
enraged_camel | 8 hours ago
naasking | 7 hours ago
layer8 | 7 hours ago
grumbel | 7 hours ago
There is the "Economic Turing Test", you let it find a job and earn money for itself. If it can do that reliably, across a wide range of jobs, that should fit most definitions of AGI.
ThouYS | 8 hours ago
tziki | 7 hours ago
Don't get me wrong, the benchmark jumps are good and I'm excited to try it, but only one or two of the benchmark jumps could be described as better than incremental.
bigfishrunning | 7 hours ago
glenstein | 7 hours ago
So I think it's a bit of a misleading signal and we should wait for more independent vetting. I think the middle ground is that these are improvements worthy of the "GPT-6" label but still well short of a true "this is AGI moment" that would truly put the question to rest.
IanCal | 6 hours ago
petilon | 7 hours ago
https://x.com/burny_tech/status/1725233117055553938
In the tweet Sam Altman is quoted as saying: "If (for example) super intelligence can't discover novel physics I don't think it's a superintelligence. And teaching it to clone the behavior of humans and human text - I don't think that's going to get there. And so there's this question which has been debated in the field for a long time: what do we have to do in addition to a language model to make a system that can go discover new physics?"
I think this is a reasonable criteria for declaring AGI. So can GPT-6 do it? OpenAI says it has helped solve long-standing open problems in mathematics. No word on novel physics.
Feathercrown | 6 hours ago
petilon | 6 hours ago
https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...
Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?
Kevin Roose (New York Times): I probably would, yeah. Would you?
Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.
kypro | 7 hours ago
Today's models and agents are not quite at human-level in all contexts and across all domains, but it seems to me they very clearly are generally intelligent.
If you disagree – can you name a single problem that a human can do that agent wouldn't be able to take a decent shot at which isn't limited by the hardware available it?
HAL3000 | 8 hours ago
I wouldn't be surprised if there are some conceptual similarities to the kind of latent reasoning Anthropic sees in claude's J-space, although those aren't the same thing.
Recurrent/looped transformers themselves aren't a new concept, but it's interesting to finally see this approach show up in a frontier production model.
Canceling my Anthropic Max sub when this ships.
atonse | 8 hours ago
Also Opus 5 has been really tough to work with. I can't understand half of what it says, it's just so damn obscure.
elAhmo | 7 hours ago
m101 | 7 hours ago
OpenAI is 20x on both limits
beydogan | 4 hours ago
Actually no. 5x and 20x have same weekly usage across all models. Just ask their chatbot.
https://x.com/beydogan_/status/2095293596198957418
chid | an hour ago
CSMastermind | 7 hours ago
enraged_camel | 6 hours ago
You could say Sol is faster and cheaper and that's true. Outperforms Fable? Impossible to believe without hard evidence.
andxor | 5 hours ago
BeetleB | 8 hours ago
davidwritesbugs | 7 hours ago
daemonologist | 4 hours ago
johnnyApplePRNG | 8 hours ago
I suspect these benchmarks are heavily benchmaxxed as well.
5.6 Sol was not even close to 5 Opus and yet somehow it sidled right up to it on all of the benchmarks?? pfffft
theseamusjames | 8 hours ago
dominotw | 7 hours ago
dowakin | 8 hours ago
Betelbuddy | 8 hours ago
https://youtu.be/xdXLzFzxA9Q?t=362
semiquaver | 8 hours ago
smashers1114 | 8 hours ago
pandinus | 7 hours ago
bakies | 6 hours ago
now that i'm a gpt subscriber maybe I'll have luck when i'm filing next year
Banditoz | 3 hours ago
colesantiago | 7 hours ago
By 2030 all software is done and complete.
But we are going to have more and new jobs.
jdee | 7 hours ago
colesantiago | 7 hours ago
This is just another problem for the AI Labs to solve.
NichoPaolucci | 7 hours ago
balefulboy | 7 hours ago
tinyhouse | 7 hours ago
foundOpenRight | 7 hours ago
jiraiyasarutobi | 7 hours ago
trixn86 | 7 hours ago
tripleee | 5 hours ago
hannofcart | 7 hours ago
Because in another dead language of antiquity, Sanskrit, it means "weapon". Which would be a bit too on-the-nose.
BrokenCogs | 7 hours ago
hokumguru | 7 hours ago
manojlds | 7 hours ago
5555watch | 3 hours ago
- Bigger number = Better
- Bigger celestial object = Better
and the scale is Astra > Sol > Terra > Luna. "
[0]: https://x.com/thsottiaux/status/2095600295808283073
danieltk76 | 7 hours ago
SneakyZero | 5 hours ago
hazelnut | 7 hours ago
Readerium | 7 hours ago
modeless | 6 hours ago
fancyfredbot | 5 hours ago
Muse spark 1.3 was only released yesterday.
udbhavs | 7 hours ago
udbhavs | 7 hours ago
redox99 | 7 hours ago
GPT 5.0 did feel underwhelming though.
udbhavs | 7 hours ago
redox99 | 7 hours ago
abixb | 7 hours ago
[0] https://www.reddit.com/r/singularity/comments/1mk8tm8/gpt5_c...
l3x4ur1n | 7 hours ago
kypro | 6 hours ago
sanex | 7 hours ago
alasano | 7 hours ago
nickpsecurity | an hour ago
dang | 7 hours ago
Please stand by... it will all come back shortly
the_duke | 7 hours ago
dang | 7 hours ago
All fixed now.
layer8 | 7 hours ago
dang | 7 hours ago
abixb | 7 hours ago
If this is truly AGI (subject to one's definition of AGI still), then this is a very boring release of an AGI model. No video announcement, no presser, just a blog post (with some Twitter promo vids)?
As others mentioned, I'm starting to think OpenAI was under immense pressure to deliver an 'AGI' model for certain contractual reasons, but I never expected GPT-6 release to be this mundane and banal.
catigula | 7 hours ago
driverdan | 7 hours ago
Scoring well in a benchmark that's called AGI does not make an LLM AGI.
dmitrygr | 7 hours ago
jhonof | 7 hours ago
cyanydeez | 7 hours ago
"Homer, you can't just declare Artifical General Intelligence; you need to like, make something or something...mmmmrrrhh"
wilg | 7 hours ago
bsndjdjdjdj | 6 hours ago
In a closed a press briefing earlier today, OpenAI co-founder and president Greg Brockman offered an unusually direct formulation of that message, ending the session with: “Welcome to the AGI era.”
"""
maxall4 | 5 hours ago
luma | 4 hours ago
p-e-w | 4 hours ago
esikich | 3 hours ago
SV_BubbleTime | 4 hours ago
But if you’re asking when a model has a sustainable general intelligence, for me, it’s pretty easy…
When it makes financial sense to run it 24 hours a day.
luma | 3 hours ago
It makes either position pointless to argue.
deadmutex | 3 hours ago
Aren't we way way past that already? QPS to any of the frontier models for a given point in time is most likely (far) greater than zero.
SequoiaHope | 3 hours ago
Directly - something can be useful without being AGI.
ShinyLeftPad | 3 hours ago
tclancy | 2 hours ago
Jaxkr | 2 hours ago
mrheosuper | 2 hours ago
ozozozd | 14 minutes ago
sigpwned | 2 hours ago
staticman2 | an hour ago
If so I'm hoping we can track them down and have them tell us if they think this is AGI.
lmpdev | an hour ago
Panzer04 | 42 minutes ago
If I can't give it an arbitrary task and have it solve that task eventually, it's not a general intelligence.
mlmonkey | 23 minutes ago
mullingitover | 7 hours ago
Hot take: These models are never going to be 'AGI'. We're just going from a GPT4 ball that's 90% round to a GPT5 that's 99% round to a GPT6 that's 99.9% etc etc etc
I think that the harnesses and context management is really where the rubber meets the road, and the real gains are happening there.
abixb | 7 hours ago
True. So we did hit a wall with pure scaling alone, though no lab would admit it. It's crazy to see how harness switchout results in such vast delta in benchmark scores.
XenophileJKO | 4 hours ago
Harnesses magnify and make the intelligence actionable, but we have not reached limits on raw intelligence yet, not even close.
senordevnyc | 4 hours ago
cyanydeez | 7 hours ago
user43928 | 7 hours ago
One could use gpt-4 or gpt-5 with today's harnesses and we'd see how well that goes.
abixb | 7 hours ago
NiloCK | 3 hours ago
"The harness improvements are the real sauce" is like a sincere "It's gotta be the shoes" take about Micheal Jordan.
(For the younger: that line was from a series of Nike ads where his skills were being explained)
chrismarlow9 | 6 hours ago
frabcus | 6 hours ago
And then use those to find fundamentally better new architectures for AI - that perhaps are as efficient as the human brain.
It might not work, but I didn't think it'd solve maths problems... So it might work. And if it happens, they'd use the data centres to run millions of instances of it.
It's scary, TBH.
m11a | 6 hours ago
But I think calling this “automating AI research” is misleading. I’m not sure there’s evidence yet that they do creative research work. Even in mathematics, but they are finding counter-examples by intelligent brute-forcing. Not to downplay the results, as they are incredible, but this is one very specific kind of proof and not the most creative type, which arguably requires generalisation.
seanw444 | 5 hours ago
Finding counterexamples is low-hanging fruit, the automation of which isn't shocking.
ogogmad | 4 hours ago
> Finding counterexamples is low-hanging fruit, the automation of which isn't shocking.
It's not good to be confidently wrong the way you're being.
senordevnyc | 4 hours ago
chrismarlow9 | 2 hours ago
scrollaway | 6 hours ago
The HN crowd I'm sure will still be unhappy calling it AGI because "it's not AGI unless its speech comes from the cerebral cortex region of the brain, otherwise it's just sparkling emoji" or something.
andersonpico | 5 hours ago
chausen | 4 hours ago
Those are all things that humanity is doing everyday. What we have is amazing, but it’s not that.
nater5000 | 5 hours ago
>The kickers is that if they do achieve (and solve) AGI in this way all the giant data centers would be mostly useless.
Perhaps. But only at that point, not leading up to that point.
It's kind of like setting up scaffolding to build something. You spend all of that time and money to build something just to tear it down in the end. But the point is that it's simply a cost to be able to build the actual thing you're building.
If these companies are able to achieve the results they're looking for, none of the investors involved are going to care that the datacenters and infrastructure they spent so much money.
anvuong | 7 hours ago
kridsdale1 | an hour ago
thomasahle | 6 hours ago
• 97.6% on frontier math
• 95.9% on CAD
• 100% on ExploitBench
Nothing modest about it
nater5000 | 5 hours ago
akoboldfrying | 4 hours ago
ertgbnm | 5 hours ago
beering | 4 hours ago
senordevnyc | 4 hours ago
adastra22 | 5 hours ago
TomGarden | 4 hours ago
adastra22 | 4 hours ago
You (and the rest of the media and many industry figures) are conflating artificial super-intelligence (reference point: humans) with artificial general intelligence (reference point: specialized/narrow GOFAI).
TomGarden | 4 hours ago
rad-b | 4 hours ago
ShinyLeftPad | 3 hours ago
So now humans is "super" intelligence? it's nice to move the upper bar so that more stuff can be called "just" intelligence.
adastra22 | 2 hours ago
ShinyLeftPad | 2 hours ago
general intelligence for beavers or a birch forest would be very different than general intelligence for humans...
adastra22 | an hour ago
ShinyLeftPad | 50 minutes ago
newsy-combi | 2 hours ago
theptip | 4 hours ago
lumost | 2 hours ago
Is it rapid skill acquisition? -> ARC benchmarks are saturated Is it breadth of knowledge? -> See many ... many benchmarks Is it ability to do hard tasks? -> see terminal-bench and released outputs.
We are at the point where the starting point for most tasks should be "send your agent to work on it."
So where do we draw the line in a way that doesn't move every 6 months?
newsy-combi | 2 hours ago
lumost | 26 minutes ago
1 year ago we viewed models as tools and agents were just kinda toying around, that we now think the bar is literally an anything to anything converter through one agent is wild.
tiborsaas | 2 hours ago
They've released two videos:
Vision video:
https://www.youtube.com/watch?v=1QNsdr-Qx_I
(kinda reminds me of these retro videos about the future home: https://www.youtube.com/watch?v=rnbaehgxdp0) ((can't find the other one where someone controls the home computer with voice))
Vibe coding with it:
https://www.youtube.com/watch?v=-TTyyY3VWh8
clhodapp | an hour ago
Don't be surprised to see other (or even the same) people declaring AGI again and again, as it becomes the best time to do so for different parties.
damsta | 7 hours ago
balefulboy | 7 hours ago
damsta | 6 hours ago
herpdyderp | 7 hours ago
holbrad | 7 hours ago
Kiro | 6 hours ago
alex7o | 7 hours ago
ianm218 | 7 hours ago
GodelNumbering | 7 hours ago
E-Reverance | 7 hours ago
ChrisGammell | 7 hours ago
jpatten | 5 hours ago
ChrisGammell | 4 hours ago
I am most excited about it bringing down the barrier so more people join in on hardware fun, so hopefully it will unlock folks that stayed away in the past.
intenex | 7 hours ago
Regardless, the result is still valid as the original benchmark harness is definitely unreasonably handicapped, and if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI in the sense of passing the most famous benchmark designed specifically to measure AGI progress, after multiple iterations of progressively making it harder.
I think it is fair to say that this is probably effectively AGI if the benchmarks are remotely accurate - even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks. If Astra's this much better than Fable, I'm ready to call AGI here.
For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
abixb | 7 hours ago
intrasight | 7 hours ago
drusepth | 7 hours ago
[0] https://arxiv.org/pdf/2503.23674
thepasch | 7 hours ago
bbor | 7 hours ago
pants2 | 7 hours ago
acchow | 6 hours ago
pkulak | 7 hours ago
debugnik | 6 hours ago
pkulak | 3 hours ago
But I still stand by it being _my_ benchmark for machine intelligence, which is all I was claiming.
bbor | 7 hours ago
Smart guy, that Turing. I wish he were still around... Linus but 114 years old and with 8 of that as the chair of a federated EU, kept alive by his own positive impact on dissolving the cold war into even more of a scientific boom. Would crazy helpful as we try to navigate the interesting times within which we have been damned.
A comforting thought, almost?
intrasight | 6 hours ago
mvkel | 7 hours ago
When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"
That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new models can do will challenge the views of AI that people developed by using prior generations of models.
iterateoften | 7 hours ago
azan_ | 7 hours ago
fn-mote | 4 hours ago
2x gains from a mature technology would be surprising.
2x gains from a new tech would still be called “low hanging fruit” in another setting.
I don’t read enough to know in what ways the training / other technical steps have really advanced.
anvuong | 7 hours ago
abixb | 7 hours ago
You're treating an off-hand comment by an ARC 3 researcher as some sort of a precise AI capability acceleration benchmark. Can we leave casual anecdotes (even from researchers) out of the discussions please?
z7 | 4 hours ago
"Frontier models today perform very poorly with a minimal harness. However if big labs start directly targeting the benchmark like they did for ARC-2, numbers will go up fast."
https://x.com/fchollet/status/2022054537293705260
giancarlostoro | 7 hours ago
I feel like this test is just helping someone like Sam Altman pretend like he implemented AGI as originally pitched for an IPO when in fact, he has not. Shameful.
> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.
- Sam Altman on AGI
lenerdenator | 7 hours ago
Probably not.
giancarlostoro | 7 hours ago
lenerdenator | 6 hours ago
If it can replace a worker but does too much work to be checked routinely by a human, and bears no real responsibility for its actions, well... it's really just a way to jack up the value of the settlement the company using it gets to pay out when it does something that causes a lawsuit.
If OpenAI had just simply stuck to making "good enough" models that were open sourced (like they promised they would be when starting out) and could be used to augment a human doing a task - a human that could be given actual consequences for messing up - they wouldn't have burned all of this money trying to reach this nebulous definition of AGI. Hell, "good enough" is what many open-source models are, and that's what terrifies Altman.
alex0015 | 6 hours ago
If the provider says "the model will always be right or your money back" then the provider has got responsibility. If there's no guarantee, there's no responsibility on their part, just on the person whose job it is to try and solve a problem with the model.
Avicebron | 6 hours ago
kbelder | 5 hours ago
alex0015 | 5 hours ago
degamad | 6 hours ago
If you give an "intelligent agent" offered by one of these model providers a task of updating the content of your website, and it updates it with inappropriate adult content, who incurs the cost of the machine's error? The model provider generally does not.
It it makes a mistake and deletes your website from AWS, who is responsible?
If it targets another website because it decides that it is "part" of your website and attempts to break into it, who is responsible?
alex0015 | 5 hours ago
In general, the person whose job it is to provide the company with a working, non-adult website and not hack into other websites is the one who would receive consequences for failing to meet those expectations.
lenerdenator | 4 hours ago
These tools can push out more results than a human can hope to evaluate in a business-sensitive, or even realistic, amount of time. You have to take it at its word that it did things right, and there's no real fear of failure or consequence on the behalf of the agent.
zapkyeskrill | 3 hours ago
m10i | 37 minutes ago
Something in your prompt led it to do that, is alex0015's point. The statistical odds of these frontier models screwing up to that extent are so impossibly low that it would almost have to be intentional or accidental negligence on the part of the prompt writer to accidentally have their agent write pornography to their website.
The burden of the mistake would have to fall on the person that gave the tool instructions, because it can't know that what it did was wrong. Wrong is subjective in this case. It only did what it did because you, figuratively speaking, encouraged it to.
fooqux | 6 hours ago
ricardobeat | 5 hours ago
lenerdenator | 4 hours ago
m-s-y | 2 hours ago
chrsw | 6 hours ago
azan_ | 6 hours ago
sheepscreek | 2 hours ago
usef- | 5 hours ago
cortesoft | an hour ago
ygjb | an hour ago
fragmede | an hour ago
ACCount37 | 5 hours ago
In practical terms? They usually get solved with a bigger badder LLM. "New ideas are needed?" Nah - ten times the params, ten times the test time compute.
ARC-AGI-3 was more of a failure in that regard than -1 or -2, because even on day 0, an off the shelf LLM with a harness could get 50%+. And messing with evals by forbidding "LLM with a harness" from scoring? Yeah no, that was just bad.
petilon | 5 hours ago
https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...
Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?
Kevin Roose (New York Times): I probably would, yeah. Would you?
Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.
throwawayq3423 | 5 hours ago
astro1234 | 4 hours ago
adastra22 | 4 hours ago
verelo | 4 hours ago
astro1234 | 3 hours ago
ShinyLeftPad | 3 hours ago
glenstein | 3 hours ago
So it could be a natural experiment for whether AI can contribute to novel physics. Specifically, there's a big question about weather. Something like our informational understanding of black holes where information inside it is equivalent to information on its boundary (which I'm sure I'm not saying correctly), might be generalized to regular space-time. More people should be freaking out with excitement about this and perhaps it's something to which AI can contribute.
sail0rm00n | an hour ago
senderista | 4 hours ago
colordrops | 4 hours ago
petilon | 4 hours ago
colordrops | an hour ago
type_enthusiast | 4 hours ago
(Note: I am not suggesting we let it do this. Please don't, in fact)
petilon | 4 hours ago
auntienomen | 3 hours ago
adastra22 | 4 hours ago
minimaxir | 34 minutes ago
adastra22 | 2 hours ago
dr_dshiv | an hour ago
hackerbrother | 4 hours ago
block_dagger | 3 hours ago
jmalicki | 3 hours ago
A lot of his great discoveries were mostly that he was very knowledgeable about the bleeding edge research in a number of disparate areas, and was able to have the aha moment where he could make the connections for how to integrate them.
A lot of other thinkers who created new fields from scratch are probably way harder for an LLM to crack.
That is very aligned with an LLMs ability to have superhuman knowledge in wide areas.
jmalicki | 3 hours ago
The ability to do a ton of book learning in training, and pull in tons of related context at once, is superhuman in some ways, but lags a lot in others.
AareyBaba | 2 hours ago
AI in math is ongoing. https://spectrum.ieee.org/ai-in-mathematics
CamperBob2 | 2 hours ago
And the only reason LLMs can't write essays indistinguishable from human output is because they aren't RLHF'ed to write like humans.
Folding clothes isn't an LLM's job but if you were to insist, they could certainly do it, as any number of videos from robotics labs will attest. That particular future is already here but definitely not evenly-distributed.
mcphage | an hour ago
That feels kinda like when I remember seeing Ocarina of Time for the first time, and thinking “oh my god, this looks just like real life…”.
CamperBob2 | 39 minutes ago
vel0city | 2 hours ago
Checkmate, meatbags.
no-name-here | 2 hours ago
FabHK | an hour ago
https://mymodernmet.com/gianluca-gimini-velocipedia-bicycles...
https://qz.com/681345/an-artists-3d-renderings-of-bicycles-d...
reasonabl_human | an hour ago
Then it’s an expert system.
Stephen Hawking wasn’t very good at folding clothes.
The ‘General’ part of the term ‘AGI’ seems like a trap to me, because there will always be new workflows to master. Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?
You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.
Meanwhile, building a series of expert systems targeting specific valuable workflows is useful today and seems like it’ll continue to scale to cover huge swathes of economically valuable workflows.
I think that’s the more interesting thing to be measuring. The surface area of useful economic workflows that can be addressed with expert systems built with today’s tech.
Hitting some ‘Artificial Expert Intelligence’ coverage threshold on economically valuable workflows is what will matter for humans well before pure ‘general’ intelligence.
klooney | 59 minutes ago
refulgentis | 5 hours ago
adastra22 | 2 hours ago
butterisgood | 4 hours ago
An AGI wouldn't struggle with that.
slidehero | 4 hours ago
this has been debunked too many times to bother rebutting. they struggle with those things because of the way they are.
it's completely irrelevant.
phlakaton | 4 hours ago
It may not be useful for anything else, but at least it can say that.
slidehero | 4 hours ago
turns out that flapping wings is not the right way to unlock human flight.
computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.
mrandish | 2 hours ago
I don't think it's irrelevant but perhaps not in the way you're assuming. When assessing AGI I'm not evaluating counting characters or even the execution of math operators at any scale or speed. As you observe, computer software from Regex to spreadsheets and Mathematica already handle that well. But AGI isn't about what computers can do, it's about whether AIs can do the specific things which, until now, have been uniquely human capabilities. Like understanding nuanced context and then coming up with novel approaches to solve a new kind of problem not relying on any specific prior training or knowledge (the 'G' is for General).
Most definitions of AGI start from a baseline that already assumes easily passing a Turing test and doing anything via text response that a high school graduate could. I ding LLMs not for failing to count but for failing to intuitively understand the nuanced context of a simple class of problem it hasn't seen in its training data. I fully understand that the reason LLMs fail letter counting is that they operate at the token level. They weren't trained on individual letters first, like human 2nd graders.
The only reason recent LLMs get strawberry and blueberry correct now is that they have those words on their pre-training 'cheat sheet'. However, the underlying fundamental weakness in the way LLM intelligence works which leads to this failure mode still hasn't been addressed. Even when the frontier labs add "recognize any sub-token counting question and write a Python script" to the training cheat sheet so LLMs always pass that test... they'll still be unable to recognize a simple class of problem which isn't on their 'cheat sheet'. As long as that's the case, to me, they aren't AGI because they can't fully replicate human-like recognition of novel problem classes. And it's not just about letter-counting. That gap and others like it lead to many other kinds of non-human brittleness in LLM problem solving. Those are the classes of reasoning, intuition and insight that the ARC-AGI series has been trying to queue up as targets. Not to show how bad LLMs are but to help them be great in all these counter-intuitive edge cases
slidehero | 2 hours ago
AI does not need to feel emotions or have a heartbeat to be useful. It only needs to perform a task correctly à la Chinese room.
>therefore cannot fully replicate human-like intelligence
this does not follow. planes don't flap wings therefore they cannot fly?
mrandish | an hour ago
This example still misses my point, which isn't related to usefulness or economic value. I concede that LLMs can have greater utility and economic value than humans on many tasks. The point is most definitions of AGI include something like "can fully replicate all the routine daily tasks done by any competent high-school graduate." That's not related to whether LLMs can solve many high-value problems faster and at larger scale than any human. That was also true of ENIAC in 1946.
The fact an airplane can fly faster and farther than any bird is irrelevant to whether an airplane can "fully replicate all the routine daily tasks done by any competent bird." That's the bird equivalent to most AGI definitions. An airplane can't build a nest or recognize the signals encoded in birdsong.
In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any". And in this context, airplanes scoring 15,000% more than birds on 'speed' and 'distance' doesn't matter any more than AIs scoring 15,000% more than humans on 'add 10,000 numbers'. We still aren't near AGI because LLMs cannot fully match any high-schooler's ability to independently conceive new approaches to novel problems not in their prior training data.
slidehero | 9 minutes ago
which gets us closer to philosophical questions which which I'm personally not that interested in.
>In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any".
I'm not sure we want a machine that fully succeeds that test.
Planes pass the 'bird replacement' test on the only criteria that matters to us ... flying.
If we wanted nest making planes I think we'd have them by now. Nest making doesn't rate highly on the problems we're looking to solve though.
I don't want a machine that is moody, or depressed or has schizophrenia, which are all pat of the human condition.
We don't need the human "intuition magic dust" to do 99.99999% of useful work.
They're machines designed to do the work we don't want to. That's as "general" as their intelligence needs to be.
I don't want my clothes folding machine to have an existential crisis.
Dylan16807 | 2 hours ago
That just says we don't need to design an AI like a brain. That's not part of this discussion at all.
> computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.
I'm confused, is your argument something like "It's too easy so AGI doesn't need to be able to do it"?
The fact that very basic computers can do it makes failures embarrassing when testing for AGI, not irrelevant.
slidehero | 2 hours ago
Do you possess magnetoreception? a stupid pigeon can "see" the earth's magentic field. why are you blind to it? does a lack of magnetoreception make your intelligence any less "general"
no, you're just blind to it because that's just the way it is.
LLMs are blind to character counting because that's the way they are.
It didn't stop ChatGPT from finding the Jacobian Conjecture counterexample.
Human intelligence and machine intelligence are only going to cross over to a certain degree.
same as plane flight and bird flight are only kinda related.
Dylan16807 | an hour ago
But if I can't calculate it myself I know to use that basic computer to do it, not make up an answer.
> Human intelligence and machine intelligence are only going to cross over to a certain degree.
That's where the word "General" kicks in. If there's big limitations on the overlap forever, then there will never be AGI.
slidehero | an hour ago
maybe. we'll see.
butterisgood | 2 hours ago
Breadth of capabilities matters... and a promotional video is nice and all, but people are throwing this term around like it's a prize they've won, but they've not gotten there yet.
gjm11 | an hour ago
[EDITED to add:] I actually don't know whether dyslexic people find it difficult to count letters in words, if they have them already written down by someone else. I suspect they find it harder than people who aren't dyslexic. But perhaps "blind people whose spelling is poor" would have been better; I would not want to deny them human-like intelligence either.
hackinthebochs | 44 minutes ago
A more accurate test would be to give it a list of words (or anything represented as a single token) and ask it how many times that token appeared. I'm sure they have no trouble at that task.
bvanheu | 33 minutes ago
hackinthebochs | 24 minutes ago
mcphage | an hour ago
I mean, they seem like fair game if you’re ever participating in a Turing Test.
nearbuy | 4 hours ago
Meanwhile most humans fail to correctly answer how many f's are in the sentence, "Finished files are the result of years of scientific study combined with the experience of many years.".
the_gastropod | 3 hours ago
steelframe | 3 hours ago
SmashDan | 2 hours ago
matt_kantor | an hour ago
Imagine verbally asking someone who has never seen written text the same question: unless they memorized the answer for the specific word you're asking about, they'd have to guess.
nearbuy | 7 minutes ago
We don't know what was going on inside the closed source GPT models, but this paper investigated on some of the open-weight models and found it's not due to tokenization: https://arxiv.org/abs/2604.00778
hackinthebochs | 52 minutes ago
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
nearbuy | 2 hours ago
butterisgood | 2 hours ago
jasondigitized | 3 hours ago
taneq | 2 hours ago
AGI doesn’t mean infallible, it just means it can have a reasonable crack at things it hasn’t seen or done before.
butterisgood | 2 hours ago
mattmcal | 2 hours ago
Forgeties79 | an hour ago
Typical result of venture capital and too many bag holders unfortunately.
gavinray | 7 hours ago
osigurdson | 4 hours ago
saimiam | 3 hours ago
I’m sure you know this is an exponential growth question but have no intuition of the answer.
MajesticHobo2 | 3 hours ago
kccqzy | 2 hours ago
balefulboy | 7 hours ago
Forgeties79 | 29 minutes ago
Super useful, incredible tools, but not AGI. Try and roleplay a dialogue with one, make it whatever character and scenario, and see if it can sustain a coherent conversation for more than 30min with you AIM style (aol instant messenger, if that isn’t clear). Expert mode: never correct or adjust it mid conversation.
I’m not even talking about repetition and predictability. It’s nothing like talking to a person. And in a short amount of time it literally can’t form a coherent sentence.
morningbrew | 7 hours ago
applfanboysbgon | 7 hours ago
bbor | 7 hours ago
It's obvious that these scientists are in bad faith, as they've invested way too much of their lives into the field being real -- they're just playing up the data. Common sense tells me that winter is still happening, anyway; what's the big fuss?
(/s, cause you never know these days)
[1] https://upload.wikimedia.org/wikipedia/commons/e/e2/The_Plan...
applfanboysbgon | 7 hours ago
bbor | 5 hours ago
And yes, the one deeply-researched field going back 75 years is as scientifically rigorous as another deeply-researched field going back ~100 years. I guess you can draw climate studies back to Descartes and the Islamic golden age, but that doesn't privilege it in a time where the methods have changed completely in the span of decades.
simianwords | 7 hours ago
No. Humans are still better at super long context learning. Once that is beat you are completely correct.
waffletower | 7 hours ago
uludag | 7 hours ago
Like what about having some "AGI model" embodied in something (maybe humanoid), and test it by having it step in an assortment of cars and park them. Does bodily-kinesthetic intelligence account for nothing? Humans are intelligent creatures and can dynamically adapt to the physical shape of a variety of vehicles and their movement characteristics. And there's so many things like this that are extremely basic, which some people dismiss since practically every human has the capability to do it, but actually requires a high degree of intelligence.
cryptoz | 7 hours ago
I do see where you're going, but that's already what's happening: we have so many different benchmarks because there's no real single way to test for general intelligence.
Also, it takes a human probably at least a decade of world experience, growth, learning, etc, to pass your benchmark. I'm quite confident that it will be very soon that an embodied LLM will pass your new benchmark, much sooner than a human would take if born today.
strken | 6 hours ago
If a model couldn't go to work as e.g. a first year apprentice plumber on their first day and perform anywhere remotely close to the median, but can pass a benchmark that claims to measure AGI, the benchmark is wrong and the model is not exhibiting general intelligence yet. ApprenticePlumberBench sounds like it's genuinely better than ARC-AGI at measuring AGI and that's a bit silly.
(Edit: I wrote ARC-GIS the first time around, for some silly reason)
visarga | 5 hours ago
dist-epoch | 6 hours ago
Running a marathon is not needed to claim AGI.
visarga | 5 hours ago
If you go by definition AGI is not general, just "smart ape" shaped.
Barrin92 | 4 hours ago
OpenAI's own charter defines AGI as "Highly autonomous systems that outperform humans at most economically valuable work". This is actually fairly sensible and involves obviously a ton of non-cognitive, emotional, social and physical activity. In other words, if you can replace most or all human beings with a machine, you have something that's generally intelligent.
That's obviously not even remotely where we're at, AI chatbots do well on narrow usually text based or programmatic problems, but can't even replace a barista or a plumber.
ilaksh | 3 hours ago
coderenegade | an hour ago
Even our large language models have an implicit embodiment in the domain of text (and more recently, multimodal inputs). That seems sufficient for certain things, and insufficient for others. I suspect that AGI that does everything a human can do eventually turns out to be fairly analogous to humans in terms of sensory input and domain output, even if the scale is radically different (e.g. thousands of robots uploading (touch, sight, audio, smell, etc.) sensory data to a single model, and each being actuated individually).
fidotron | 6 hours ago
That said, a look at the state of self driving and the recent robot olympics shows that advancement on that has accelerated enormously, though whether it's reflected in any of the LLMs is something else entirely.
dgunay | 3 hours ago
dinfinity | an hour ago
(Very capable!) Embodiment, persistent operation and continuous learning are indeed things that still set us apart from AI. None of those are fundamentally difficult to solve, though.
More importantly, none of those are particularly relevant for being "intelligent": If a criminal threatened to kill your family unless you solve some difficult problem that requires only intelligence and you could choose any single person, animal, or AI to help you with it, which would you choose? Be honest.
goochphd | 7 hours ago
[1] https://arcprize.org/blog/astra
hypfer | 7 hours ago
Computer chips got faster, but 2026 edition. Why the artificial ceiling/category/goal labelled "AGI"?
I'd much rather like to talk about what this enables, instead of discussing whether a category someone made up applies here or not.
giancarlostoro | 7 hours ago
> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.
So... unless you hear of a company replacing their workforce with OpenAI agents, I don't think we're there yet.
hypfer | 7 hours ago
Agree on your assessment.
But also, interesting quote, because the business model relies entirely on IP law. Like.. if that thing exists and the sharing costs are 0 (just copy weights, lol), then why would I give them money for this. Makes no sense.
We only pay money for resources that are scarce as some sort of flawed allocation determination mechanism.
aaah this industry aaaah
senordevnyc | 7 hours ago
hypfer | 6 hours ago
I say AGI is only reached when it can do that.
tonyhart7 | 6 hours ago
because if its other way around then the answer is oblivious
giancarlostoro | 6 hours ago
Mind you, the original thoughts on AGI before Sam Altman started to water them down involved continuous learning, which LLMs do not do, their core data is static.
jonas21 | 6 hours ago
indoorfish | 5 hours ago
giancarlostoro | 4 hours ago
reasonableklout | 4 hours ago
giancarlostoro | 4 hours ago
reasonableklout | 3 hours ago
micromacrofoot | 6 hours ago
rfgplk | 6 hours ago
You can already pretty much do this.
dlubarov | 5 hours ago
[1] https://www.forbes.com/sites/markfaithfull/2026/05/07/heres-...
qlte | an hour ago
phatfish | 6 hours ago
qsort | 7 hours ago
True.
> Regardless, the result is still valid (...)
If you think the game is rigged, the virtuous thing to do is to point that out and refuse to partecipate; making up your own rules is something I just don't understand, especially since the rule-abiding result would still have been SOTA.
> in the sense of passing the most famous benchmark designed specifically to measure AGI progress
The benchmark does not measure AGI progress or progress towards superhuman intelligence, as explicitly stated by the creators.
On the AGI question: surely you realize this depends on how we define the term? For example, one of the definitions OpenAI originally gave is "capable of doing most economically valuable work", which almost certainly Astra, as impressive as it is, would fall short of. I'm not saying it's a good definition, but as far as I'm concerned it's as good as any. More importantly, I don't think that it would change much if we said yes or no. I'm only bothering to take a position if it amounts to something.
This can feel as "moving the goalposts", and to some extent it is, but if done honestly "moving the goalposts" is how you make progress. Had you asked me 10 years ago I would have said that anything that could hold a conversation like GPT-4 could would probably have been wildly superhuman at almost everything. It shouldn't be hard to find ways GPT-4 was lacking, though. We see new things, we reassess and try again: that's how it's supposed to work.
yoz-y | 7 hours ago
It being able to comfortably say “i don’t know how to do this” rather than boiling and ocean to pick a shell from the shore without getting wet.
regularfry | 7 hours ago
voidmain0001 | 6 hours ago
irthomasthomas | 6 hours ago
zug_zug | 6 hours ago
To me AGI is all about the "G" general (we already had the AI part). General meaning universal, everything. It's not a function of knowledge or specific hardcoded tests, it's that you could give it a test it's never heard of before and never been trained on and it would ace it (it might need a lot of time).
Currently LLMs can't even really learn within a conversation, they can add a note to context and try to not drop it. Example things an AI cannot do yet (but maybe someday will):
- write a well-received book, write a best-seller
- come up with a new company idea, Run that company
- actually have a decent conversation, maybe someday talk somebody out of suicide effectively
- come up with its own ideas or theories that nobody else has presented
- understand the stock market well enough to trade better than an index fund
- be an expert Game Master in a TTRPG (making no mistakes, getting a read on the players' fantasies, calibrating difficulty in response to emotions)
- come up with a theory of what makes games fun, make a popular game
- be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)
- be able to articulate what it knows, what it doesn't know, and what information it would need to have to answer complex queries
- exhibit metacognition (thinking about its own thinking) and self-optimization
- wonder about things
- observe contradictions and ironies in the social-consciousness, do a standup routine that makes you rethink how you look at things
crooked-v | 6 hours ago
So far nobody's even shown an LLM succesfully running a high-traffic vending machine for as much as 30 days at a time.
tonyhart7 | 6 hours ago
or are you miss the part "general intelligence" is ????
kolinko | 6 hours ago
zug_zug | 6 hours ago
I think if a person had those same advantages (e.g. could spend 5 hours thinking about what to say next) we could all hold outstanding conversations, or if we had read every book ever written I think many of us could write a very popular book, if we could read every singe company's P&L statement in a few seconds we could invest better than an index fund.
What I'm pointing out here is that these models appear to be intelligent when they really are simply unimagineably knowledgeable. When you drop the time-constraints it starts to become more and more apparent that human intelligence scales better with time than AI does (much in the same way AI can burp out tons of code but make your codebase entirely illegible within a matter of months).
Perhaps to simplify: my notion of intelligence is how much can you deduce with a constant set of starting context
kolinko | 5 hours ago
I’m not so sure of that - to get average outcomes in these fields it’s a matter of time, to get above average or extraordinary, you need talent/intelligence/taste.
And the bar the parent set is at extraordinary.
Verdex | 5 hours ago
howunfortunate | 5 hours ago
Verdex | 4 hours ago
So like you spend a million times more than that and you still think you're not going to see some results?
jimbooonooo | 4 hours ago
strange_quark | 2 hours ago
howunfortunate | an hour ago
> any human for which we spend $100billion - $3 trillion...would be able to reach all of those levels
howunfortunate | 3 hours ago
But that list is extremely ambitious. Write a best seller, make a popular game, come up with a truly novel theory, consistently outtrade index funds.
That's top 0.001% human stuff, I don't think you can take just any person and get there through education alone, it takes extreme talent and dedication. There's also diminishing returns when spending on education, it doesn't just improve linearly.
steelframe | 3 hours ago
I have a kid in an American university right now, and a quick search of my bank account statements confirms that there are far more expensive educations in the world.
Dylan16807 | an hour ago
bigDinosaur | 43 minutes ago
usef- | 5 hours ago
Verdex | 5 hours ago
To really drill down into that I would think you would need to figure out how many millionair children get tutored vs how many get spoiled.
TylerE | 5 hours ago
Gold medal Olympic athletes who are also brain surgeons AND astronauts, no.
modriano | 4 hours ago
roundabout-host | 6 hours ago
usef- | 5 hours ago
feanaro | an hour ago
The claim is furthermore that truly original thinking, the infamous leaps in understanding and creativity, happen by attaining points outside such a convex hull.
It's hard to rigorously verify or disprove this claim. Hopefully this helps build an intuition of why the claim is not as shallow and obviously wrong as it may seem initially.
ilikecode | 31 minutes ago
jasondigitized | 3 hours ago
dominotw | an hour ago
namarie | 6 hours ago
cnxhk | 6 hours ago
bulder | 5 hours ago
david-gpu | 5 hours ago
- write a well-received book, write a best-seller
- come up with a new company idea, Run that company
- actually have a decent conversation, maybe someday talk somebody out of suicide effectively
- come up with its own ideas or theories that nobody else has presented
- understand the stock market well enough to trade better than an index fund
I just picked the first few from the top of the list. The average human has probably not done any of them.
Verdex | 5 hours ago
I think LLMs are really impressive, but I suspect that we might have overpaid just a bit.
david-gpu | 5 hours ago
workingonit3 | 5 hours ago
Yes, average humans are not beating the stock market. But the average human is a bit better than you give credit to.
david-gpu | 5 hours ago
The average human has never published a bestselling book. A person who has published a bestselling book is an above-average writer. And, therefore, an artificial intelligence capable of writing a bestselling book would be above an average human at the task of writing books. Therefore, somewhere beyond an AGI.
Attempting to redefine AGI to "being better than most humans at most tasks" is moving the goalposts towards artificial superintelligence.
mNovak | 6 hours ago
In a sense I think no one will agree on a definition of AGI until it becomes impossible to construct any benchmark under which an AI underperforms "average" humans. That or it's defined retrospectively, after it's overwhelmingly obvious it met any such definition.
CrazyStat | 6 hours ago
> These arguments take the form, “I grant you that you can make machines do all the things you have mentioned but you will never be able to make one to do X”. Numerous features X are suggested in this connexion. I offer a selection:
> Be kind, resourceful, beautiful, friendly (p. 448), have initiative, have a sense of humour, tell right from wrong, make mistakes (p. 448), fall in love, enjoy strawberries and cream (p. 448), make some one fall in love with it, learn from experience (pp. 456 f.), use words properly, be the subject of its own thought (p. 449), have as much diversity of behaviour as a man, do something really new (p. 450). (Some of these disabilities are given special consideration as indicated by the page numbers.)
(emphasis added).
thesmtsolver2 | 4 hours ago
pavitheran | 6 hours ago
bayindirh | 6 hours ago
Some problems presented needs a very large context and some are not much solvable (e.g. trading) since market responds to traders' actions, as well, making it effectively an oracle problem (of computation).
On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them. However, brains in nature never stops. Wonder, daydream, sleep, self-evolve, clean up and eliminate memories and views and much more.
hannasanarion | 5 hours ago
The hardware of human intelligence is not a singular thing that is uniform throughout. You cannot take the prefrontal cortex white matter out of someone's head and say you are holding a person. Much of the parts of our brains that enable much of our intelligence, is made of different specialized stuff. The visual cortex and sensorimotor regions aren't only there for input and output, they are used by the more thinky parts of the brain to do visualization and spatial reasoning. The cerebellum contains billions of neurons making little oscillator circuits and PID-like self-regulation machines that help make muscles do what they're supposed to, but also provide attention and time perception.
Heck, our brains contain language models, that train themselves up based on a glut of data over a span of about 10 years, and then they become more or less set in stone for the rest of our lives. Of course we can learn languages, but the "Critical Period" is a very real thing that produces a permanent architecture for some grammatical structures, or things like the ability to partition a lexicon by gender for faster lexical access which cannot be learned as an adult if your native language did not have gender.
I'm not trying to make a direct analogy, the point is that the language model doesn't need to be fully "generally intelligent" all on its own for there to exist a general intelligence, because the language model can be part of a generally intelligent system, which can do things like form, recall, and manage memories which are by now a standard feature in basically every chatbot.
kypro | 5 hours ago
The parent commenter noted:
"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"
Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.
jasonfarnon | 4 hours ago
ShinyLeftPad | 3 hours ago
latentsea | 2 hours ago
smgpie | an hour ago
tempestn | 6 hours ago
uptodatenews | 6 hours ago
Who will be responsible for the outputs and side effects of such a closed loop system?
Half of those the agent fleet systems can do right now.
These are things it cant do and will not be able to do without human labor and long running human vision:
https://rcsnyder.github.io/open-frontier-curriculum/05-front...
https://rcsnyder.github.io/open-frontier-curriculum/05-front...
burrito_brain | 6 hours ago
In my opinion that is exactly the point missing from AGI: the fact that you still need to prompt it. As long as you have to ask for something, is not general.
tetec1 | 5 hours ago
chmod775 | 5 hours ago
Current LLMs are worse than most dementia cases, reaching "peak domain skill" pretty much immediately.
arctic-true | 5 hours ago
reasonableklout | 4 hours ago
They often act on their own, too, and get things wrong a lot. The reason it works is because of all the systems of laws and institutions we have built around humans, not so much because human minds are special.
latentsea | 2 hours ago
Itself. That's the point. We can do it. Until it can met that bar, it ain't AGI. That's always been the bar.
sho_hn | an hour ago
hartator | 6 hours ago
azan_ | 6 hours ago
azan_ | 6 hours ago
Tumblewood | 4 hours ago
shoobiedoo | 4 hours ago
I think what OP was drawing a comparison to is that AI right now could not come up with an award winning novel from the spark of some creative notion and working up from there, as opposed to just mashing together what has already been done and calling it a day.
fn-mote | 3 hours ago
I agree, but in this field we value evidence. So there needs to be some test of novel-writing abilities.
Once there is, AI companies will be out to score highly on it.
Wait for a resurgence of Philip K Dick-style novels as humans desperately try to write things LLMs cannot.
shoobiedoo | 3 hours ago
I myself can't wait for Finnegan's Wake 2
gilbetron | 4 hours ago
diomedes | 2 hours ago
i'm not sure what makes you think AI cannot do this already. in my experience, this sort of deep research is something AI is quite good at.
example i just tested: https://chatgpt.com/share/6a9a20e3-1d20-83ea-a125-31aa240c74...
zug_zug | 2 hours ago
For example I asked it to do a full analysis on the AI bubble, and a full analysis on the risks of Glyphosate, and it came up with a lot of things that sounded credible, but within a few minutes of questing was admitting it hadn't even really checked for internal consistency in its positions, and even doing a 180. It certainly was much faster at gathering sources and reading but it fundamentally doesn't seem very effective at creating a consistent worldview.
And of course the funny thing is it says it did a 180 on one of these topics, great, except whatever it concluded will be discarded because it cannot learn. It's just bonkers to me pretend this is AGI, it probably couldn't even hold its own in this very discussion.
tiahura | an hour ago
pera | 6 hours ago
Maybe I'm too boring but it seems quite pointless to have this same prediction game every time a new model is released.
azan_ | 5 hours ago
indoorfish | 5 hours ago
dash2 | 52 minutes ago
ogogmad | 4 hours ago
If I had to guess, I think LLMs will be inventing highly original new mathematics within the next year. I think it will be approached as an optimisation problem, targeting how quickly LLMs can solve classes of maths problems as a function of the definitions they need to conjure up to do so.
doctoboggan | 5 hours ago
kaashif | 4 hours ago
Which is to say, not that fast.
dinfinity | an hour ago
It is an open question to what extent practical experimentation/work will be a bottleneck for the theoretical work. It stands to reason that it is improbable that it will be the bottleneck for 100% of the speed of treatment development.
jrflo | an hour ago
adan1719 | 6 hours ago
If I were a test subject for that low salary, I'd cruise and not care at all about my performance. Which is exactly what they want anyway.
skarz | 3 hours ago
dom96 | 6 hours ago
drittich | 6 hours ago
phatfish | 6 hours ago
A tangent, but can anyone ELI5 how models "know" when to stop generating tokens? Or what the method to stop them at the right point is?
bulder | 5 hours ago
That is to say, it stops when it's statistically the most likely to.
phatfish | 5 hours ago
bigfudge | 5 hours ago
dlubarov | 5 hours ago
ilaksh | 4 hours ago
mbesto | 6 hours ago
Simple. AGI is undefinable and benchmarks are notoriously flawed.
vlmutolo | 6 hours ago
The models are designed to keep the reasoning tokens separate from the output and only publicly emit tool calls and the sometimes a summary of the reasoning tokens. The models are trained to depend on those private reasoning tokens. You can’t just delete them.
https://openai.com/index/how-two-settings-tripled-our-arc-ag...
10xDev | 6 hours ago
dlubarov | 5 hours ago
An AGI test should be black-box; we shouldn't impose require requirements on internal components. As long as the overall AI is capable of learning and remembering things, it shouldn't matter if there's a stateless LLM internally.
irthomasthomas | 6 hours ago
jbritton | 6 hours ago
Then realize LLMs have zero of what anyone would consider intelligence.
jbritton | 2 hours ago
Which is strange because I’m sure it could give general advice about how to play better, it just doesn’t follow the rules it can enumerate. It also doesn’t seem to have spatial awareness.
I used to think LLMs couldn’t do Fibonacci for the same reason. They could write the code but not follow it. They can now follow a procedure to generate fib numbers but it seems to be memory limited.
So I don’t know why it can track fib algo, but no chess concepts.
dissahc | an hour ago
imagine a hypothetical chess match between:
- an undoubtedly very intelligent person. in the course of their studies, they have read about different chess strategies, openings, etc. but they never actually played the game themselves
- an average person with a year of chess playing experience
who do you think is going to win? of course, you could give the LLM time to think and consider its opponents potential next moves, but this is a computationally expensive way to play the game that doesn't scale
which is all beside the point that chess isn't a very good proxy for general intelligence. there is a correlation, but it's very weak
techpression | 6 hours ago
Eliezer | 6 hours ago
If I asked you to write fiction, you'd be much better at keeping track of which characters knew which facts.
avaer | 3 hours ago
I agree on the one-shot (which is not a fair comparison because nobody oneshots a good story), but I'm not convinced this part hasn't reached AGI already.
magicalist | an hour ago
Yes, and I could script a truly marvelous proof if this textarea were but a little larger :)
Hand waving doesn't count for much these days when you could spin these things up quite quickly to prove the point, so the GP's claim seems much stronger than whatever you're not convinced of?
bendergarcia | 6 hours ago
eggnet | 6 hours ago
chimprich | 5 hours ago
dingdong2026 | 6 hours ago
Today I spent half a day trying to solve a moderately interesting software engineering problem. I was switching between GPT-5.6 Sol and Fable 5.1 to check each other's work in Cursor.
And the result was gradually driving me insane. As the models struggled to find a solution that would actually work, they dug themselves deeper into a hole. The work grew in complexity beyond my ability to understand what's happening and recover.
At some point, when I felt like throwing the keyboard out the window, I just gave up. Tomorrow I'm starting from scratch, having burned god knows how many tokens and hours of my life.
But sure, they can create a decent website or CRUD app, so they must be really smart.
That's AGI for you.
holmesworcester | 5 hours ago
vatsachak | 5 hours ago
akoboldfrying | 5 hours ago
And: Is your work more difficult than finding proofs of or counterexamples to decades-old open problems in mathematics?
sumedh | 4 hours ago
NothingAboutAny | 3 hours ago
no-name-here | an hour ago
1. I’ll often include boilerplate in a prompt to tell it to make the broader fix. [1]
2. However, a top HN AGENTS.md post 11 days ago included the standard guidance “As much as possible try to minimize the number of changed lines when implementing a feature.” I.e. some devs want LLMs to avoid broader changes and so some of that likely makes it into the training, even if others like us want the opposite.
[1] As far as whether my boilerplate is effective, I don’t know.
skue | an hour ago
The smarter AI gets, the easier it becomes to move the AGI goalposts. Seems at this point there are people who will refuse to call anything less than omniintelligence AGI.
(And then the excuse will be, but it’s not omniscient! And even if it were, is it omnipotent?)
Yizahi | 5 hours ago
Can Astra, or any other model explain how exactly it reached this or that output result? Start with a simple query of asking to add 55+66 for example. (no LLM program can do that)
Can Astra, or any other model refuse to answer or go on "thinking" in a orthogonal direction on it's own?
That's just two quick ideas, I'm pretty sure cognition scientists can invent better and wider range of checks.
ObnoxiousProxy | 5 hours ago
intenex | 30 minutes ago
"Can you add 55 to 66 and explain how you reached that output result"
And received this answer:
"55 + 66 = 121.
Add the tens: 50 + 60 = 110. Add the ones: 5 + 6 = 11. Combine them: 110 + 11 = 121."
Do you mean something else? Do humans do something better than this?
jameson | 5 hours ago
Also the training dataset is proprietary and they'll drive the LLM's behavior, so it make sense for the vendors to invest in the harness and bake in prompts that work best with their models.
visarga | 5 hours ago
sensanaty | 5 hours ago
Most telling is M$ or whichever one of these borg megacorpos defined AGI as (paraphrased) "AGI is whatever tooling earns us a gazillion dollars in revenue"
wavemode | 4 hours ago
Or do you mean that, for any given problem, we could theoretically design a harness that allows AI to solve it (not that, one single harness solves everything). In which case I'm still not convinced but I guess could see why one would believe that.
dotancohen | 4 hours ago
zquzra | 4 hours ago
waterTanuki | 4 hours ago
Stick to the original definition of AGI of an AI model being able to self-improve independently with 0 human intervention and become an "everything" solver. Ever since money got involved in this, the goal posts have shifted considerably. If OpenAI truly had an AGI on their hands they would then be able to crack encryption, destroy world markets, and funnel all resources back into their new for-profit organization. Since their mission is now share price, until I see any evidence of an infinitely growing stock I will reserve my congratulations.
marrone12 | 3 hours ago
lwansbrough | 3 hours ago
A real AGI, like the ones from science fiction, would make Astra look like a child’s toy. And I guess more concretely I would expect it to inhibit the following properties: one shot learning - fully (and always) online, perfectly efficient (through self improvement), no context limitations ie. persistently thinking, not just awaiting input.
So for me, no, not AGI yet. But still very intelligent and capable (and perhaps it’s safer this way?)
Fizz43 | 3 hours ago
m-s-y | 2 hours ago
While this may be true, it’s a pretty poor indicator of whether or not it’s AGI.
m3kw9 | an hour ago
tom2026hn | an hour ago
sidharthkmenon | an hour ago
I’d go out on a limb and say astra’s ability at graduate level math will have ~0 bearing on its general reasoning capabilities; we’ll all acclimate being tired of its “neuralese” and more surprising mistakes.
I think we need a true, step change advance in model architecture, but it’s hard to see how the current frontier labs can do that because of golden handcuffs / innovators dilemma
johnsmith1840 | an hour ago
The task was to build a high performance classification model. It not only helped make an entire data capture pipeline but also made the sythetic data basline needed. Then it proceeded to build and test 100 different model varients with methods and techniques I've never seen before. The results are basically SOTA based on the effeciency and compute contraints.
But this brings up something huge about these. I was there. I pushed the direction and work throughout it all. If it was entirely up to fable max or sol max the result would have been pretty bad.
All of these things are still chatgpt 3 scaled. It's identical even if the scale has gotten pretty wild. I could ask chatgpt 3 to make a single function and it worked well, 4o a file, 5, a small project, 5.6 far more, biggest improvements lately is they don't seem to get lost on long running tasks.
Is big gpt 3 AGI? I don't think so but perhaps scale can mimic it close enough our squishy brains fail to handle them correctly.
hdjrudni | 37 minutes ago
Make a slightly evil version of GPT 6 and see if it can successfully catfish someone. How long before they realize something's up, that they aren't actually talking to a human?
huijzer | 7 minutes ago
Most things in the real world probably. I’m not saying AI can’t do it, but currently it’s bad. Try send an image of the inside of a broken toaster and how to fix. It’s laughable. Again, not saying AI will never do it, but am saying there are definitely large holes in knowledge.
xixixao | 7 minutes ago
alex7o | 7 hours ago
bdangubic | 7 hours ago
Planktonne | 7 hours ago
It's AGI, and it's going to upload photos, or change a background slide colour. Even the people hyping it up, who believe that it's really artificial intelligence in every sense of the word, couldn't get it to do more than that.
This is farcical.
baq | 7 hours ago
balefulboy | 7 hours ago
geodel | 7 hours ago
emp_ | 7 hours ago
emp17344 | 7 hours ago
mchusma | 7 hours ago
I'm not trying to be too negative on it, it could be the best model right now, but it clearly isn't some agi god because things like that should have been caught (also should have been caught by human reviewers).
ranyume | 6 hours ago
kroaton | 7 hours ago
jryan49 | 7 hours ago
Yajirobe | 7 hours ago
wilg | 7 hours ago
mminer237 | 6 hours ago
wilg | 6 hours ago
neta1337 | 6 hours ago
useruser125524 | 6 hours ago
TacticalCoder | 6 hours ago
They decided to use the iconic Herman Miller Eames chair if I'm not mistaken:
https://youtu.be/s5zyhGMMPKs
And that's basically 50% of the vid looking "classy".
I don't know if it's farcical but at this point --maybe I'm jaded-- I'm expecting more than a kid rocketship I can print on my Bambu Lab A1.
Now I'd say the promotional vid is actually good. But it's marketing: so it's a good vid, but cheesy good.
Doesn't mean GPT-6 Astra is good or bad: looks solid from the numbers.
NamlchakKhandro | 6 hours ago
I thought people here were smarter than that
Buttons840 | 6 hours ago
It shows people who seem to have very full and rich lives, and the reason they do is because they use ChatGPT. These are the people smart enough to say things like "do what needs to be done", or "change the background to make it look better"--insights like these are why they make the big bucks.
On the one hand, I think this is an accurate depiction of the future. There is no meritocracy here. Some people have access to the best AIs and can speak a sentence and get great results, and the rest of us don't have access and so we're the poors. The happy presentation doesn't match the way I'm feeling.
I do wonder how rich CEOs will justify earning 500x as much as their employees when they're just another person that's dumber than an AI. Why are they paid so much again?
holoduke | 6 hours ago
garciasn | 6 hours ago
Haven’t you learned anything?
reasonableklout | 3 hours ago
forgetfulness | 2 hours ago
It will probably still have some veneers of meritocracy.
These will be very well-credentialed people, who went to top schools and will know all the right people, to whom they can tell all the right words, and it's not access to AI that will be the determining factor, but the fact that they're entrusted with capital and authority to direct small teams of people who also went to top schools and can speak corporate jargon at a bot.
It will just exacerbate dynamics that are already there. Why do people need bachelor's degrees to send emails, today? For the same reason someone will need a PhD or a master's degree from a prestigious school to do it tomorrow.
And the rest, well, you know, some of the remaining journalists will write op-eds describing how they are beyond help, too angry, too dirty, too much of an other.
adverbly | 4 hours ago
The benchmarks are impressive though.
One other thing that bugged me though was that they crop every single plot in some cases the y-axis would show a range between like 40 and 70%. Makes the whole thing feel like a spectacle rather than anything serious. I find it cheapens it because it is quite serious in the end.
mvkel | 7 hours ago
If you've played the games firsthand, you know what an accomplishment this is. The "games" feel like a weird conduit to a lower level of your brain, where you move pieces to a specific place because it just "feels" right. For AI to nail it better than a human speaks to some magic happening underneath.
Looking forward to ARC-AGI-4,5,6 and slowly chipping away at the remaining problem sets.
sashank_1509 | 7 hours ago
the_duke | 7 hours ago
It will be interesting to see how it performs in the real world ...
Robdel12 | 7 hours ago
So, folks that have actually used this already, what’s it actually like?
GodelNumbering | 7 hours ago
Terminal-Bench 4.0: High (57.9%), Max (56.7%)
DeepSWE: High (73.3%), Max (71.5%)
It _loses_ 1-2% performance going to High from Max
XCSme | 7 hours ago
GodelNumbering | 7 hours ago
Such as?
I can't think of any. Diminishing returns, yes. Occasionally flat, yes. Downright regression, no.
XCSme | 6 hours ago
The reasoning effort should match the complexity of the task against the model's capability.
Hard task with low reasoning = bad
Easy task with very high reasoning = bad
minatoaqua1 | 6 hours ago
m0zzie | 3 hours ago
kingjimmy | 7 hours ago
BrokenCogs | 7 hours ago
dalemhurley | 7 hours ago
Sol is so much better than Fable 5. Then we get Astra (yet to use it) few days after Fable 5.1 (which is very impressive).
Codex is slightly better than Claude Code.
Good on Sam Altman getting back to basics and turning OpenAI around.
kroaton | 7 hours ago
tonyhart7 | 6 hours ago
Chinese counterpart like CXMT and Huawei is begin producing their own chip
You cant block an entire nation level effort with tariff
astrobiased | 6 hours ago
spartacusnacho | 4 hours ago
rgbrenner | 4 hours ago
VirusNewbie | 6 hours ago
seunosewa | 5 hours ago
dansquizsoft | 4 hours ago
nwienert | 3 hours ago
amazingamazing | 5 hours ago
david-gpu | 4 hours ago
angulardragon03 | 4 hours ago
david-gpu | 3 hours ago
amazingamazing | 2 hours ago
2. Any open models created from this will not necessarily need their silicon, see apple mlx.
haldujai | an hour ago
Nvidia could just the same rent those GPUs out for inference and actually have way better margins than they do right now. Antitrust and putting all your eggs in one basket are why they don’t, similar to TSMC.
2. Neither do AI labs. See Anthropic buying TPUs, deploying with AMD. OpenAI on Maia, Cerebras, their own wafers.
reilly3000 | 4 hours ago
amazingamazing | 2 hours ago
cmrdporcupine | 40 minutes ago
They exist to keep people using and advancing the tools on their hardware.
evilduck | 3 hours ago
davidguetta | 5 hours ago
There's not 100 frontier labs, it's not like airline companies
haldujai | 4 hours ago
Actually not a bad comparison. Both burn massive amounts of up front capital to protect an oligopoly in the hopes their commodity product eventually pays off.
Razengan | 5 hours ago
For most people, the app IS the AI.
And even for its wonkiness, ChatGPT has had the best UX/UI of them all.
The way to win the AI wars in the eyes of the common folk is through the frontend, to be the Apple of AI, as it were.
m3kw9 | an hour ago
scronkfinkle | an hour ago
The only moat they appear to have is by hoarding compute, and the current trajectory of hardware shows that isn't permanent either for very long
bitexploder | an hour ago
aurareturn | an hour ago
And no one would say that about TSMC.
So there is clearly a moat there somewhere.
coolandsmartrr | an hour ago
saithound | an hour ago
Comparing the R&D costs of creating GPT-4o vs. DeepSeek V3 (the latest gen for which we already have good accurate numbers) it looks like the latter cost 1/20th as much to create.
If Samsung could catch up with TSMC for 1/20th of the cost, people definitely would say that TSMC has no moat.
aurareturn | 51 minutes ago
saithound | 43 minutes ago
Which numbers you believe is of course going to affect whether you think there's a moat or not. That's largely orthogonal to your TSMC/Samsung analogy I responded to. If you think the "moatists" are wrong because they believe the wrong numbers, that's fine, but then there's no need for the analogy.
[1] https://galileo.ai/blog/llm-model-training-cost
[2] https://medium.com/@theiand/how-can-deepseek-a-5-6-million-l...
aurareturn | 35 minutes ago
saithound | 25 minutes ago
aurareturn | 14 minutes ago
Clever architecture --> Ok but OpenAI/Anthropic can use these as well and they also have very smart people with their secret clever architectures
Distilling --> Ok but distilling means you will never be smarter than the original. Furthermore, reasoning is now hidden by private labs and they have poison pill answers for distilling if they can detect it. They will be able to detect distilling better and better.
Cheaper electricity --> Ok this is cancelled out by their chips being much less efficient due to not having ASML EUV machine access.
So I don't see why fundamentally their training costs are cheaper over the long term.
I'm looking for a no-moatist to convince me.
upupupandaway | 7 hours ago
jeffybefffy519 | 7 hours ago
I just dont get how its good for some, and bad for others. It makes me suspect that the models performance is not even against problem sets and it really is just a probabilistic prediction machine. Which then makes me very skeptical of GPT-6 Astra, because if their big claim is Computer Use then it is probably bad in a bunch of other areas.
embedding-shape | 7 hours ago
> I just dont get how its good for some, and bad for others.
If I were to listen to my hunch, it would tell me that it's all up to the prompts that ends up going over the wire (including all the bloat some people have), what workflow/process you use and what the existing state of the project is.
ragequittah | 5 hours ago
I've had Claude do the same thing where it goes off and spends 100% of my tokens on 3 functions and an ungodly amount of tests / scaffolding that do almost nothing when I gave it an underdeveloped idea.
John7878781 | 7 hours ago
zachthewf | 6 hours ago
ghosty141 | 6 hours ago
It's still incredibly important to have a human in the loop correcting design decisions and having good taste.
jiggawatts | 5 hours ago
Is a very sane tech stack, you're just biased against Microsoft.
Half the world's enterprise apps run on that combination, or a minor variation of it.
Like Java it is full featured ("batteries included") but unlike Java it is relatively terse and actually pleasant to work with.
Oh, and unlike Python, it is very fast, within spitting distance of compiled Rust and C++ web apps.
kyleee | 2 hours ago
EduardoBautista | 3 hours ago
swingboy | 3 hours ago
Atotalnoob | 2 hours ago
Swapping mssql to SQLite would also work perfectly
gruntled-worker | 5 hours ago
People with high IQ often do this IRL. There's training tension in this area. Intelligence and overcomplication correlate and are hard to extricate.
puttycat | 4 hours ago
gruntled-worker | an hour ago
In some cases, there are also nuances that we don't pick up on. Here it's our preference for simplification that's showing up. We set the lossy compression factor higher than it does.
jpgvm | an hour ago
I work on very low level stuff (think RTL/FPGA, firmware, software where optimising for nanoseconds is just normal).
For me Sol is the only cost effective model available. Fable 5.1 is indeed good and vastly better than original Fable (which refused to work on most of my stuff for 'safety' reasons).
It's very good at this sort of low level stuff to the point that I really can't understand/relate to people having a good time with Opus (which comparatively performs extremely poorly on my particular workload).
I also just don't like how lazy Anthropic models are. They will do 10% of what is asked and then summarily declare victory.
Sol on the other hand is more like "one of us", slight touch of the 'tism, extremely pedantic, will go to the edge of the known universe if that is what it takes to prove/fix/build what you asked for or run out out of credits trying.
It's a personal and workload dependent thing. For me right now Sol for 99% of stuff because Fable 5.1 still burns through $5k in credits a day.
cmrdporcupine | 37 minutes ago
Sol will leave a mess of excessive redundant tests and isn't so great at abstraction ; but it produces more reliable working systems.
It's kind of nice to have access to both, but I don't have the $$ for that right now, so I just keep the Codex sub
Implicated | 6 hours ago
... looks around ...
andxor | 6 hours ago
I'm genuinely so confused when people say this with a straight face. Are you talking about coding? Desktop use? Prose? Or something else?
Sol is a much smaller models and it shows. It often misses the forest for the trees.
enraged_camel | 4 hours ago
Same. It makes me wonder what types of things the person must be working on.
resonious | 3 hours ago
People will see/think that I'm doing very well with my LLM use, and ask me what I'm doing. I tell them, they try it, then later they come back to me saying they just couldn't get it to work.
bitexploder | an hour ago
fastball | 5 hours ago
carljungslabtek | 5 hours ago
ChadMoran | 5 hours ago
fnordpiglet | 5 hours ago
Claude has the exact opposite problem, especially opus-5, where I literally can’t trust it to print hello world without taking a shortcut, or just simply lying and saying it printed it when it didn’t, behind a giant wall of inscrutable text. I find it very ironic that Anthropic is the vendor of the lazy lying cheating model that does almost everything you tell it to it do.
I’d really kill for something that balances instruction following and loop escaping behavior better. Fable 5.1 does seem a lot better, feeling more like 4.6 behavior, and honestly Sol has improved as well. I’m pretty psyched for the next generation, as I think the competition has heated up so much that things will improve really fast to the point of marginal utility opportunity being increasingly close to epsilon.
swingboy | 3 hours ago
> model_context_window = 1000000
> model_auto_compact_token_limit = 900000
I believe it does consume your usage a bit faster though.
bitexploder | an hour ago
openaiscooked | an hour ago
fooblaster | 32 minutes ago
dearing | 7 hours ago
jdprgm | 7 hours ago
It feels nearly impossible to have any rigorous approach when choosing a particular model and price point for a task and more like blindly picking one. The time period needed to actually get familiar with various models to a degree you can intuitively choose appropriate ones for a task is moot when it will likely be superseded faster than the needed time.
I guess if companies are footing the bills most employees just opt for whatever the most expensive model they can get away with. Even then choosing between the various leading models is the same kind of frustrating task. Every release every company has the same random collection of graphs and charts claiming the best performance on X, Y, and Z.
dominotw | 7 hours ago
Pikamander2 | 7 hours ago
Imagine buying a shiny new PC in the 90s only to see it become practically obsolete within a year.
phainopepla2 | 7 hours ago
bananaflag | 7 hours ago
embedding-shape | 7 hours ago
computomatic | 7 hours ago
If you bought a mid-tier computer that was good enough for what you needed, then you probably didn't shop/compare for the next few years and didn't notice. But if you shelled out $7-10k for a top-of-the-line system and paid attention to progress, you'd easily see that become the mid-tier $1000 option within two years or less. This is how it was in the 90's PC boom, at least. Likely the same for the decades before, not sure how it went in the 2000's.
senordevnyc | 7 hours ago
This is not how I remember that period at all. Do you have any examples?
benjiro29 | 5 hours ago
386 to 486 to the first Pentium (with the bug!)... You did not upgrade in place, it was often a new system. Sure, you maybe kept your screen, keyboard etc but ... The only upgrade we had on the same MB, was a coprocessor upgrade. Remember those? Each new generation of CPU was a new motherboard. Upgrading CPUs in the same MB really became a thing only later on.
GPUs had a shelf life of barely a year. Its been 35 year but i remember TNT to TNT2 having like 9 month in between. Moving from 2D to 3D involved a constant cost as GPUs evolved fast and the latest games required latest hardware.
We have not talked about the ISA, AGP, and PCI fun ... The “bus wars”.
DOS to Windows 3.1 (and OS/2 somewhere in between) to 95 ... with software being pushing hardware, just like games did.
This is why people are spoiled with cheap PC hardware where its cheap, and easily lasts 4+ years. Even with the bad memory price and more expensive GPUs, your can stil buy a $1500 system that will last you years (with maybe some lower game settings later on ... or the catalog of 10.000s games that will easily run on a mid tier GPU).
PC hardware has become boring but extreme stable. You can run GPUs for year, switch MBs without issues while keeping large amounts of old hardware. That was NOT the 80s and 90s that i remember.
senordevnyc | 3 hours ago
The 386 and 486 were 3.5 years apart, weren’t they?
tqi | 5 hours ago
senordevnyc | 4 hours ago
Nition | 6 hours ago
fooker | 4 hours ago
upupupandaway | 7 hours ago
bananaflag | 7 hours ago
lackoftactics | 7 hours ago
exe34 | 6 hours ago
Then one day the hard drive appeared to die. I eventually realised the issue was located around the 1.5gb mark, so I recreated my Linux partitions after 2gb and it worked fine for the rest of the year.
dcl | 3 hours ago
I also believe you could get motherboards that supported 2 Celeron chips. I have no idea how effective/useful it was, but it was certainly a cheap/interesting way to get multiple CPU's.
re-thc | 7 hours ago
You don't see Nvidia and AMD fighting every other month over the latest cards.
unreal37 | 6 hours ago
The pace of change ("practically obsolete") is different then and now.
dcl | 3 hours ago
upupupandaway | 7 hours ago
A dev in my team saw a new model and changed one application to use said model (essentially changing the contents of a url). One week later I received an escalation from the CTO of the company that our pace of weekly usage was in the millions of dollars (rather than low hundred thousands). Turns out that the new model was 5x more expensive but no one noticed.
arjie | 5 hours ago
gavinray | 7 hours ago
tonyedgecombe | 7 hours ago
https://www.joelonsoftware.com/2002/01/06/fire-and-motion/
smcleod | 7 hours ago
I appreciate boring tech as much as the next well worn engineer and I'm not saying this is all positive but it's so sure as hell thrilling and you don't have to be an astronaut to immediately benefit (or suffer I guess) from it.
matheusmoreira | 6 hours ago
brokencode | 6 hours ago
If one day you open up Claude Code and it’s Opus 5.1 now instead of Opus 5, no big deal. It probably will work about the same as it did before. Maybe a little better.
Or if you’re on Codex and some new cool Claude model comes out, no worries. There will probably be a similar new model for Codex within a few weeks. Maybe even within a few days.
shostack | 6 hours ago
fantasizr | 6 hours ago
Aurornis | 6 hours ago
In practice, you can get away without keeping up with everything all the time. For personal use, pick a provider and get on their ~$20/month plan. Learn their high/medium/low model hierarchy. Start with their highest or second-highest model (GPT-5.6, Opus, etc) and observe your quota usage. If you're doing a lot of manual code review and analysis, the $20/month plan goes very far even on the highest models. If you're trying to vibecode everything as fast as possible it's a different story.
If you keep running into quota limits, experiment with the next model down for easier tasks or adjusting the effort level. If the results are good enough, you've found your fit. If they're not, you might need the next plan up.
For API/business use, you have to be checking your token spend as you go to calibrate to how much each task costs and where you fall in your budget. There are a lot of different tools that make this easy to visualize.
For data tasks, you should have an eval with a golden dataset that you can run against new models for a nominal amount of token expenditure. It should be as simple as pointing the eval script at a new API or model and checking the score versus price.
danenania | 6 hours ago
Input tokens are much cheaper than output tokens. Not only because of baseline price—caching makes a huge difference too. There are many ways to take advantage of this asymmetry to get similar quality for a fraction of the cost!
teaearlgraycold | 6 hours ago
epolanski | 6 hours ago
I have released applications on Gemini 3.5 flash that make real money and I don't see any particular reason to upgrade.
Zizizizz | 6 hours ago
https://youtube.com/shorts/vGKC9LpGnOQ?is=iCG7qvAIL9oI5-_d
flockonus | 6 hours ago
But more so it seems there is Fear of missing out (FOMO) in our behaviours. The reality is, if whatever model you are using are good for your purpose, well, keep on it.
mfkhalil | 6 hours ago
In theory though, there is a minimum viable model for any given task, and we think that is a problem that the big labs will avoid because they profit from charging more per task. We're trying heuristic and LLM-based approaches but it's still a work in progress, so if this is something you'd be interested in trying would highly recommend trying ours out -- any and all feedback at this point is extremely valuable to us.
https://docs.litellm.ai/docs/proxy/auto_routing
ghthor | 4 hours ago
brcmthrowaway | 7 hours ago
dopa42365 | 7 hours ago
https://www.reuters.com/business/openai-says-upcoming-model-...
> "With the right tools and access, Astra can find previously unknown security flaws and develop ways to exploit them across many well-protected systems without a person guiding each step," said Amelia Glaese, an OpenAI vice president overseeing its safety work.
> The company plans to make Astra available "soon" to a limited group, but declined to provide specifics. Glaese said the extra security measures may "sometimes slow, pause, or stop legitimate work," and that OpenAI would work to minimize those disruptions.
what a bag of horseshit
bbor | 7 hours ago
rcr-anti | 7 hours ago
sharmajai | 7 hours ago
sbinnee | 7 hours ago
maherbeg | 7 hours ago
maybe call it EngEmployeeBench
Centigonal | 6 hours ago
maherbeg | 6 hours ago
retired | 7 hours ago
Chinjut | 7 hours ago
Yajirobe | 7 hours ago
worldsavior | 7 hours ago
Kkoala | 6 hours ago
weakfish | 6 hours ago
Keyframe | 6 hours ago
mawadev | 6 hours ago
tetec1 | 5 hours ago
neta1337 | 6 hours ago
avgDev | 6 hours ago
andriy_koval | 6 hours ago
job depends on how CEO feeling about cutting NN% of headcount because of AI advancement
gavinray | 7 hours ago
My backup plan is being a personal trainer.
Rover222 | 6 hours ago
conradfr | 6 hours ago
quaunaut | 6 hours ago
ckdot | 6 hours ago
gavinray | 6 hours ago
zachthewf | 6 hours ago
dyauspitr | 6 hours ago
tonyhart7 | 6 hours ago
jrflo | 10 minutes ago
demirbey05 | 6 hours ago
gavinray | 6 hours ago
But my wife and I have been homeless before, so living on a shoestring budget in anything nicer than a tent is acceptable living conditions to me.
I am sure I will be plenty comfy no matter how the world changes.
kypro | 6 hours ago
AIs are really good at being personal trainers and seem to be far more educated and informed than most I know.
GPerson | 6 hours ago
brindidrip | 7 hours ago
amlib | 6 hours ago
ckdot | 6 hours ago
GPerson | 6 hours ago
epestr | 6 hours ago
They seem to have not yet come to believe the "is" part.
Chinjut | 6 hours ago
threethirtytwo | 6 hours ago
AI as it is now and as it will be projected into the future WILL automate many skills. But not all skills. MANY MANY people will retain skills that cannot be replaced by AI. One career track that will be replaced is definetely the SWE. Or at least massively reduced in capacity if not eliminated all together.
MrAbstract | 6 hours ago
I have a strong suspicion that many of those comments are written by people who are already financially independent, have millions in stocks, and can just sit back, coast around and watch this whole spectacle unfold while using LLMs to vibe-code their next fun side projects without a shadow of anxiety about their own future.
I’ll most likely be labelled a helpless doomer and downvoted into oblivion for saying this, but I genuinely struggle to see any silver lining here.
demirbey05 | 6 hours ago
richstokes | 6 hours ago
AI is only going to get better and do more with less humans in the loop over time.
That said, I do also relate to the "coding was never the hard part"-type arguments, and much of my day is spent on the stuff in between writing code.. but still.
TaupeRanger | 6 hours ago
zamadatix | 6 hours ago
Because of this, I don't think many are thinking "90% of the world won't have a source of livelihood but that just means I chill at my lake house for the next 20 years like a normal retirement". Instead, it's usually either "I think AI is overhyped", "I think humanity will figure something out", or "I think this is the end of humanity".
theappsecguy | 3 hours ago
RSHEPP | 6 hours ago
theappsecguy | 3 hours ago
kolinko | 6 hours ago
It’s an interesting moment in history, people 35+ yrs old seem to be less afraid if tech because we learned that things change in the way we work. People below this age got used to fact that the work and tech doesn’t change - just because for the last 10-15 years it didn’t.
Chinjut | 6 hours ago
kolinko | 5 hours ago
AaronAPU | 6 hours ago
The threat is that the very kernel of value you had is gone forever. There is no more differential leverage.
kolinko | 5 hours ago
vanuatu | 6 hours ago
most swes don't work in jobs where they only work on bounded measurable tasks. there will probably be more "engineers" than ever
TacticalCoder | 6 hours ago
Don't be selfish. Think first of all the jobs that are already dead. A friend of mine she's a translator: like translating financial documents between french/english/spanish. It's over for her: she doesn't get 10% of the gigs she used to get and the 10% she gets is... Verifying AI output.
Think of the artists: I'm sorry for those too, for for many it's already game over today.
> How will we make a living?
A friend of mine who's got his own software-consultancy SME is now advertising on LinkedIn that he'll also help your company fix the mess LLMs created.
That's how you'll make a living: by learning, in addition to all you've already learned, how you work with harnesses and LLMs to be more productive, by learning what they're good at and what they suck big fat balls at.
GPerson | 6 hours ago
Chinjut | 6 hours ago
MattDamonSpace | 6 hours ago
Chinjut | 6 hours ago
nater5000 | 6 hours ago
This is such a childish take I hear getting thrown around all the time on the internet. If you really have just been listening to whoever is telling you how to be successful, then you were always doomed to fail at some point. Like, have some self-respect and own your own life, for better or worse.
>Those of us who made the mistake of studying anything other than machine learning. How will we make a living?
Take it from someone who studied machine learning specifically: nobody is safe if you assume these companies are going to produce a product that will put everybody else out of business. If AI is going to take your job, then it's gonna take enough jobs that your problems will not be personal but systematic.
Chinjut | 4 hours ago
Yes, these problems are systematic. That is what I am saying. That doesn't make it any nicer.
aabajian | 2 hours ago
Thought experiment: If no thought workers are needed to design or engineer a Ferrari, what is needed? My answer is time and natural resources (include energy).
sashank_1509 | 50 minutes ago
rbreve | 7 hours ago
XCSme | 7 hours ago
azan_ | 6 hours ago
dakolli | 4 hours ago
alpineman | 7 hours ago
codruterdei | 7 hours ago
XCSme | 7 hours ago
sashank_1509 | 7 hours ago
gavinray | 7 hours ago
Post-work society is an inevitability if we don't destroy our planet.
lackoftactics | 6 hours ago
It would be fun to get to post-work society, but hard to imagine atm. TPTB won't let it happen
XCSme | 6 hours ago
Soon we will have some machines that can replace 50% of jobs, and this will happen basically overnight...
neta1337 | 6 hours ago
unclad5968 | 5 hours ago
XCSme | 5 hours ago
azan_ | 6 hours ago
lackoftactics | 6 hours ago
"I am the best economist in UK!"
calmoo | 6 hours ago
cautiouscat | 6 hours ago
Is it?
gavinray | 6 hours ago
I can't see a future in which almost every system (both physical and virtual) are not automated and optimized by autonomous entities.
What do you do when everyone is out of a job?
If you don't want pitchforks and riots in the streets, you give everyone UBI and housing so society doesn't collapse.
exe34 | 6 hours ago
tokioyoyo | 6 hours ago
As much as I’d love UBI to happen, in current geopolitiks it’s a no-go. People are not happy with having what the others have.
tripleee | 6 hours ago
bgarbiak | 5 hours ago
greenowl | 5 hours ago
tripleee | 4 hours ago
The beach houses, mansions, mountain views could be vacation places, or used as libraries, or simply dismantled for the materials.
The "slums" on noisy roads could be eliminated entirely and used for something people don't need to be at.
We'd move to a more equal distribution of assets closer to the middle line.
madhatter999 | 5 hours ago
echoangle | 5 hours ago
madhatter999 | 5 hours ago
If we had something like a Maslow’s hierarchy of needs but for work, I think meaningfulness would be the top of the pyramid. For most people in the world, not going hungry or affording housing are reasons enough to do work. Getting to do work you find meaningful is truly a privilege.
akoboldfrying | 4 hours ago
Fizz43 | 3 hours ago
akoboldfrying | 2 hours ago
Highest violent crime rates:
Memphis, Tennessee: ~2,400–2,500 per 100k
St. Louis, Missouri: ~2,000–2,100 per 100k
Detroit, Michigan: ~1,700–2,000 per 100k
Little Rock, Arkansas: ~1,600–1,800 per 100k
Baltimore, Maryland: ~1,600–1,700 per 100k
Oakland, California: ~1,400–1,900 per 100k
New Orleans, Louisiana: ~1,600–1,700 per 100k
Birmingham, Alabama: ~1,600–1,700 per 100k
Milwaukee, Wisconsin: ~1,100–1,600 per 100k
Cleveland, Ohio: ~1,500–1,600 per 100k
Most popular holiday destinations:
New York City, New York
Orlando, Florida
Las Vegas, Nevada
Maui, Hawaii
Grand Canyon National Park, Arizona
San Francisco, California
Miami, Florida
Yellowstone National Park, Wyoming
New Orleans, Louisiana
Great Smoky Mountains National Park, North Carolina/Tennessee
throwatdem12311 | an hour ago
Ah yes because these AI companies are just gonna give away the models for free that I use with my free computer and free smartphone while I eat with my free food in my free apartment.
paxys | 7 hours ago
XCSme | 6 hours ago
In this game of work/development, you can't make sure that other humans don't "cheat". Our work won't compete anymore with other human's work, but with a computer.
paxys | 6 hours ago
XCSme | 6 hours ago
Also, creating something with AI doesn't really feel like you made it yourself.
And, if you make it without AI, most of the times it feels pointless, why spend 30 days on working on something that can be done faster and better in 1 hour?
I am not saying about doing things for fun, but about creating useful things.
Yes, you can do "hand-crafted" things, and people appreciate that, but for code, people aren't able to see the craft anyway.
mercanlIl | 6 hours ago
paxys | 6 hours ago
If the motive is profit then you should be adopting AI just like you have adopted any other skill or tool of your profession.
ryan_n | 5 hours ago
kypro | 6 hours ago
But it's not just tech – my lack of interest in learning and creating is starting to generalise with the models. Music, writing, coding, maths, etc...
I need to get used to switching my head off and asking the AIs to think for me whenever I need to engage my brain. It still feels very unnatural.
Fergusonb | 6 hours ago
The brain loves these kinds of shortcuts.
I don't need to think about the fine motor skills of hitting a baseball, it's just a motion now, and the game is still fun.
XCSme | 5 hours ago
Because that's how creating software is starting to feel.
qlte | an hour ago
billypilgrim | 6 hours ago
xtracto | 5 hours ago
XCSme | 5 hours ago
tintor | an hour ago
f311a | 28 minutes ago
SmirkingRevenge | 3 hours ago
flaviolivolsi | 6 hours ago
XCSme | 5 hours ago
soundworlds | 6 hours ago
XCSme | 5 hours ago
david-gpu | 4 hours ago
You can't cherry pick somebody's opinion and assume it applies to everybody.
XCSme | 2 hours ago
Flere-Imsaho | 6 hours ago
I built a phone app recently, not released to the public, just an idea I had for ages but could never spend the time actually building. Its 100% vibe coded, and took me a few weekends to build... I'm talking a few hours in total.
The point I'm making is that you now have the power to create stuff you would never have had the time to build. You can think big, wild stuff. Experimentation. Throw-away code.
What a time to be alive!
rmsaksida | 5 hours ago
echoangle | 5 hours ago
rmsaksida | 4 hours ago
kami23 | 3 hours ago
XCSme | 5 hours ago
Before it was cool to share an app you made, but now? What's the point of sharing an app, if the other person can make their own, even better suited for their needs, in a few seconds?
fantasizr | an hour ago
bgarbiak | 5 hours ago
But, you can create cool stuff just for yourself. That’s the upside. It’s just hard to make a living on cool stuff for yourself.
ryan_n | 5 hours ago
Kon5ole | 5 hours ago
I find that very motivating. I can do things alone that would have required a team only one year ago.
ryan_n | 5 hours ago
sashank_1509 | an hour ago
variadix | 3 hours ago
I built several projects at home, many involving learning e.g. graphics programming and rendering, that would never be useful in my professional work, but which were intrinsically interesting and enabled me to build other, more useful projects later on. It also gave me greater confidence in my abilities as an engineer, and cursory skills I learned in the process did help in my professional work.
Now it feels like what’s the point. The machines can or will be able to build anything I could want, useful or not, faster and with less frustration. I probably won’t be able to be employed as an engineer long enough to build a career on said skills. And I can’t mentally justify not spending that time with friends and family, when the expected return is basically zero.
I still find math, science, and engineering interesting and intrinsically rewarding, but in a closer sense to how one might feel about playing video games. The information is or will eventually be useless, so it isn’t worth spending a significant amount of time on.
ryan_n | 2 hours ago
nater5000 | 5 hours ago
People SHOULD feel like making a useless Mario Kart clone isn't worth the effort anymore. They should, instead, be trying to figure out how to actually use these models to make something that doesn't feel like a useless Mario Kart clone.
XCSme | 5 hours ago
Software will be more like a "proof of work", where people would still pay $100 for good software that took $10k tokens to build.
maxnevermind | 4 hours ago
ryan_n | 4 hours ago
maxnevermind | 4 hours ago
xtracto | 2 hours ago
myaccountonhn | 2 hours ago
zeroCalories | 4 hours ago
ivanjermakov | 3 hours ago
XCSme | 2 hours ago
Now it feels like whatever I learn will be obsolete in 2 months.
spicyusername | 2 hours ago
Now it's just the product for the world, which was where most of the value was anyways.
It's a big paradigm shift and the industry is quickly going to shed people who needed the process to care about the product and we'll be left with people whose motivation to build the product (or money) is enough.
XCSme | 2 hours ago
They won't even have to ask for a specific product, they will just state their problems/needs.
lonrenor | an hour ago
I have started building my own Digital Audio Workstation. The point is not to build something to compete with Ableton. The point is to build something and make music with it. If it is a good tool then I should be able to make good music with it and release the music. Actually, the DAW should be the secret sauce of the music and something I wouldn't want to give away.
This feels a lot more like computing in the 90s after taking an odd 25 year detour of an obsession with the tools themselves instead of what the tools can actually do.
qlte | an hour ago
Like, professional electronic music artists spend 10s of thousands of hours in a DAW, but at that point it just becomes second nature and the tool disappears so they can focus entirely on the music.
tintor | an hour ago
serjester | 7 hours ago
brindidrip | 7 hours ago
astrobiased | 6 hours ago
Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area.
It seems more about coverage-driven competence. Somewhat analogous to overfitting at scale.
The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?
With our current AI architectures and training in place, I think we will only continue on skill acquisition optimization vs. truly novel intelligence.
vessenes | 6 hours ago
ex-aws-dude | 5 hours ago
z7 | 4 hours ago
https://x.com/fchollet/status/2095607046129463577
tonyhart7 | 6 hours ago
KronisLV | 6 hours ago
alpineman | 6 hours ago
Proceeds to generate the most generic, rudimentary, and unoriginal clone of Mario Kart
arkensaw | 5 hours ago
Here's a one-shotted submarine game I made with Fable a few weeks back - https://roryok.com/games/deepdive3d.html. One prompt, and I think it's deeper than this (if you'll pardon the pun)
altcognito | 2 hours ago
I usually roll my eyes when I see a comment like this because rarely do they make the points they claim to make, but I see what you're getting at. They just chose to clone someone elses work and do it in a boring way. I like OpenAI's models a lot, but they should do better.
edit - just a sidenote that I hadn't looked at the games, I just took the comment about "super-mario cart" at face value. I stand by my points 110% (even moreso perhaps), what they're showing is more polished than I expected, I assume they spent a lot of tokens on it. It is a legit shame they couldn't have spent time thinking of a better idea to illustrate something just as polished, but more interesting.
Rover222 | 6 hours ago
I hop models at will, and have done 90% of my work on OpenAI models since sol came out.
pcurve | 6 hours ago
manlymuppet | 6 hours ago
Even if I did trust an AI to get everything right, it's not like the AI can read my mind.
If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really want until they've thought about it a bit, so why do AI companies make it seem like a description is all that's required?
All the context in the world cannot accurately predict how I'll react to things I haven't seen. The problem is people treating this like something that needs a solution. It doesn't. If you want to make my life easier with AI, just make it easier to do stuff. I don't want you to pick things that I actively enjoy picking myself.
(Also not everyone has a cushy job in an AI lab that makes it so you won't miss $30 if the AI messes up haha.)
shostack | 6 hours ago
What I desperately want is for 1password or stripe or even Google who already has much of my data, to o come up with a secure solution for online purchases with agentic credit cards where I can effectively get a phone prompt to authorize a purchase while the agent can fully own the checkout flow.
I have seen various things coming on the market for this, but none of them appear aimed at a consumer audience. And I am a firm believer at this point in keeping my payment authorization and history and credentials harness agnostic.
echoangle | 5 hours ago
degamad | 3 hours ago
Corporate travel is an example. In many organisations, you tell someone in the travel department "I need to be in Tokyo for this conference from Tuesday to Sunday, and charge it to this cost code", and they figure out flights, accommodation, etc for you, with minimal input from you.
cautiouscat | 2 hours ago
SV_BubbleTime | 38 minutes ago
… 23 hours in Denver later…
satvikpendem | 2 hours ago
fwip | 2 hours ago
lonrenor | 2 hours ago
A human assistant is largely a status symbol. Most people are not really that busy. The real problem with an agentic assistant is if everyone can have one then it no longer acts as a status symbol.
bronco21016 | an hour ago
But my routine business trips in the CONUS with strictly defined booking options... let me just email an agent "Get there by meeting on day A, leave after meeting day B" and have it sort it all out without the drudgery of the corporate travel portal. YES PLEASE!
ishtanbul | 10 minutes ago
trentearl | 40 minutes ago
sensanaty | 5 hours ago
I suspect it's because it's tailored towards VCs and other similar rich ghouls as a replacement for their overworked and underpaid secretaries
incompressible | an hour ago
Ok we have AGI, so where are the _things_?!
wonnage | an hour ago
notatoad | 4 hours ago
FinnKuhn | 4 hours ago
One quick example: In fashion, Inditex and Shein have about the same revenue (€39.9bn and $41.8bn in 2025), but Inditex is more than three times as profitable. I don't see how there is a demand for agentic commerce that would remove even more control from the customer when shopping. Part of why we shop is for the experience. For B2B producurement platforms like Alibaba I can see the appeal though.
strulovich | 2 hours ago
Ran Codex, it found it for 18% less than what I found in the top Google results. It did it by finding smaller shops, applying a discount code, subscribing to a newsletter for a better code after approval, and took into account the shipping (by placing it in the cart and going to checkout) all to get me the best price.
I’m guessing without it I would have spent much more time on it and paid the original price I saw.
If you use AI agents well, they can easily save you more money than they cost, and saving money is something most people are pretty excited about.
(Disclosure: OpenAI employee)
digdugdirk | 2 hours ago
It's the same problem as all other LLM solutions (that I hope OpenAI is working on!) it's non-deterministic, and there's no way for the user (or model provider) to know what the distribution of possible outcomes is. This just gets compounded when multi-call harnesses come onto play.
mrheosuper | 2 hours ago
How do you know it's the best price ?
igor47 | an hour ago
iJohnDoe | 2 hours ago
How did you run this? Web interface, desktop app, CLI?
How did you complete the final transaction?
pvab3 | 2 hours ago
ericd | an hour ago
thi2 | 4 hours ago
sensanaty | 3 hours ago
GPerson | 3 hours ago
noelsusman | 2 hours ago
I wouldn't want it to pick food for me from a place I've never been, though to be honest with enough order history it could probably do a decent job at it.
beardbandit | 2 hours ago
tintor | an hour ago
preommr | an hour ago
The second to last line is "book it" for some tennis thing, and the scene before that has the guy eating the food the ai ordered.
throwatdem12311 | an hour ago
nullbio | 57 minutes ago
mlmonkey | 26 minutes ago
"Oh hey! Here's a demo of an AI planning out a 1-week trip to Paris!" No one in Middle America would just hand their credit card to an AI and let it come up with such a trip!
I wish SV companies took more of the middle-class (and lower-middle-class) into consideration when coming up with such demos.
(Note: I live in SF)
mNovak | 20 minutes ago
bmenrigh | 6 hours ago
Do we know if they’ve finally completed another pre-training run, or is this building off the same pre-training base they’ve been using since the GPT-4 days?
czk | 6 hours ago
wiseowise | 6 hours ago
steve-atx-7600 | 5 hours ago
itissid | 6 hours ago
Please for the love of god, just sit in a room with the government and put some restrictions around AI use before it harms a lot of people. Like tell the government to impose a minimum spend on frontier lab AI's spend on cyber defense and building every country's capabilities. The post-training mask for "I am a good assistant" is going to become a very sad joke when many people literally lose everything.
cromka | 6 hours ago
damsta | 6 hours ago
throwaway13337 | 6 hours ago
A projector and speech.
Maybe I'm in the minority here, but I find speech to text / text to speech (but not live audio mode) is quite comfortable and effective for coding now.
The speech to text part can be frustrating if your local tts model does not have word match context for coding. Codex desktop does this remotely well but is slow. I've been experimenting with local software for myself to do this between different llms.
The wall projector is a cool idea because I think it frees the user from staring at a lonely little rectangle while sitting in their fixed office chair.
If done right, this could bring us closer to the dream of more natural, social computing.
Bret Victor's (failed?) project Dynamicland involving a projector on a desk had this goal. I hear he's not much a fan of LLMs. On the one hand, I can see why. But I think, used correctly, it might be the sort of thing that unlocks his dream and, really, my dream, too.
A here's a presentation of Bret's talk on it: https://www.youtube.com/watch?v=7wa3nm0qcfM
Slight tangent: using speech to text to ramble about your rough design for like 20 minutes to an llm produces surprisingly good results over short prompts even when you contradict yourself. They're so good at picking up on what you're orbiting.
low_tech_punk | 5 hours ago
starik36 | 4 hours ago
It is still a quite fast. In fact, I just typed this in using this app.
Obluness | 6 hours ago
carlos-menezes | 6 hours ago
AGI!
holoduke | 6 hours ago
HardCodedBias | 6 hours ago
That gives me hope that there is still areas to improve.
What a bad launch video. Hilarious.
What a powerful model.
HSO | 6 hours ago
the coffee will be as cold, flat and stale as the bitcoin, metaverse, and what was the thing before that thing
agi deus ex machina descending from the icloud ftw!!!
pathetic :)))
jumploops | 6 hours ago
If I give a poorly constrained/ambiguous prompt, I don't want the model one-shotting assumptions left and right.
The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.
It's a tough balance to get right, and although this has been possible to achieve with additional prompting on existing models, I find that the agents often lean too hard into the "ask questions" mode.
Hopefully this model has the right balance, or at least better?
weird-eye-issue | an hour ago
dannyw | an hour ago
Astra retains the best parts and overall 'grounded collaborator and executor' of Sol in my testing (harness: codex CLI); while being a significant leap in capabilities & higher-level thinking.
When you prompt it like a technical collaborator, I've found Astra to be extremely consistent in staying as a collaborator, and not being over-eager, over-achieving or doing work that you haven't asked it to.
When you ask it to one-shot something, or explicitly ask it to make decisions, it will of course make its own assumptions and decisions, and generally very well.
Astra is also excellent at instruction following and respecting the guidance and steers boundaries you have.
^OpenAI does not review, limit, or tell me what to say; opinions are my own experiences.
nullbio | 52 minutes ago
They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.
enraged_camel | 11 minutes ago
I don't really agree. The thing that makes Fable feel like an actual collaborator is its ability to sus out your real intent when you give ambiguous instructions. It's really good at it.
I watched some reviews today and came way with the impression that Astra is not better than Sol in this regard. You still have to be very specific with your instructions. For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.
That sounds like a very tedious way of working with AI agents, but I understand some people want a high level of control.
alberth | 6 hours ago
I don't think it's a coincidence they launched this the week before iOS 27 launches (with new Siri).
BadBrands | 5 hours ago
I guess it makes sense they are unoriginal.
like Zuck, @sama never invented anything or innovated at all - just took other people’s ideas
HDBaseT | 5 hours ago
Sounds about right. Alignment is important, but also being able to do mundane tasks is important too.
rjtc | 5 hours ago
https://artificialanalysis.ai/models
Perhaps if it was allowed this custom harness for all benchmarks it would similarily saturate?
aniviacat | 4 hours ago
Bjorkbat | 3 hours ago
And, you know, maybe also some funny business. I think it's good to be a little suspicious of a model that happens to shoot upwards in performance on a specific benchmark while also kind of keeping up with the pack on a bunch of other benchmarks.
maxall4 | 5 hours ago
perching_aix | 5 hours ago
drivebyhooting | 5 hours ago
15 years ago if you were the sole proprietor of these models, would you be able to hold a dozen remote junior engineer jobs? Maybe even more? These models could certainly pass all interviews with flying colors and even survive independently in a company role.
I think sole ownership of AI 15 years ago could be worth north of $10 million per year. Just as rank-and-file employees.
BoorishBears | 5 hours ago
Cancer should be cured, and we should be a post-quantum interstellar fusion-powered civilization.
I wish the AGI crowd would finally shut up now that it's clear no one is even trying for AGI (OpenAI revised that to "$100B in profit")
What we're getting is incredible, where we're headed is incredible, but some people have such a fetish for futuretelling they can't just shut up and enjoy the ride.
drivebyhooting | 5 hours ago
Well by that metric humans aren’t intelligent either!
And how many people could’ve actually invented calculus, relativity, quantum mechanics? Are those who didn’t and couldn’t also not intelligent?
BoorishBears | an hour ago
The unlock isn't AGI smart enough to invent quantum mechanics, it's suddenly being able scale human intelligence using grains of sand instead of decades of food and energy and nuturing.
deepfriedbits | 3 hours ago
The demo video showed Astra create a drawing of a rocket ship from an audio prompt, take the drawing to blender, and ended with the gentleman 3D printing the rocket ship. Maybe I'm a bit older than the average HN commenter, but that's damn near magic and a great many here are kind of just taking it for granted.
low_tech_punk | 5 hours ago
Original demo (fun ending) https://www.youtube.com/watch?v=RyBEUyEtxQo
paxys | 4 hours ago
camillomiller | 5 hours ago
claiir | 5 hours ago
Looks like they listened to Terry Tao’s request for CoT in his talk on LLM use in mathematics?
dakolli | 5 hours ago
ChaseRensberger | 5 hours ago
yodsanklai | 5 hours ago
karim79 | 4 hours ago
There will be a small handful of billionaires claiming that AGI is just around the corner ad infinitum just to serve themselves at this moment in time, and capitalise from the hype.
There is no "AGI" endgame. This is shitty ass hypercapitalism in action and nothing more. I'll repeat: snake oil.
efavdb | 4 hours ago
showurwerk | 4 hours ago
gregjw | 4 hours ago
mentalgear | 4 hours ago
ntlm1686 | 4 hours ago
vinhnx | 4 hours ago
mrcwinn | 3 hours ago
datadrivenangel | 3 hours ago
Same for HealthBench Professional and a few others.
Clearly either OpenAI is very sloppy or GPT-6 Astra is also sloppy.
zhoge | 3 hours ago
quyleanh | 2 hours ago
So the closed source application should open its source in near future?
[15] https://arxiv.org/abs/2608.11469v1
tintor | an hour ago
ShoeMascot | 2 hours ago
Can someone point to a definite clarification?
Is it:
A) “Resting” intelligence that cycles 24/7 toward some goal, and any potential emergent ambient goals? (kinda what I think)
B) Consciousness itself? The ability to feel and experience alongside the thinking - even if it is toward the end of completing some task?
C) “The Singularity” (whatever that is?) so that AI can now do ____?
Someone please clarify for me!
gordonhart | 2 hours ago
grandarmory | 2 hours ago
Telanir | 2 hours ago
It is extremely impressive, but it doesn't pick up skills in a lasting manner, and requires a beefy harness for it to perform.
sheepscreek | 2 hours ago
tintor | an hour ago
- Artificial Analysis scores Astra (max effort) as 61 points on intelligence, behind Opus 5. https://artificialanalysis.ai/models/gpt-6-astra
Who is wrong here?
Some benchmark results in Astra page for Fable and Opus are blank (-).
What is Artificial Analysis intelligence index measuring that Astra scores poorly on?
Can someone from OpenAI / Artificial Analysis comment / clarify?
Even OpenAI Astra page mentions the low scope from Artificial Analysis for Astra.
kubrickslair | an hour ago
Though, unlike the creators of benchmarks like Terminal Bench or ARC AGI, the Artificial Analysis Index team does not seem to have deep technical or ML backgrounds. They are ex-strategy consultants, McKinsey, et. al.
tintor | an hour ago
AnodicElegy | an hour ago
dannyw | 52 minutes ago
That should be obvious if you compare Gemini 3.8 Flash (which is an _excellent_ model especially for its price and TPS!! but 10min of prompting in any harness) will tell you it's nowhere near close to Sol/Astra.
But AA scores Gemini 3.8 Flash at 59, and Astra at 61.
aogaili | an hour ago
We went from new JS framework every week to a new model/harness every week.
Tech is really something.
m3kw9 | an hour ago
elzbardico | an hour ago
nullbio | an hour ago
On that note, Sol is absolutely atrocious for website UI copy. It's either really awkward, or really verbose and complex and doesn't sound simple or natural. Has anyone figured out a way to reliably solve this? I've tried so many different variations of instructions and skills, and nothing works. Has anyone got an instruction that is reliable, or some other mechanism?
jrflowers | 55 minutes ago
bowsamic | 26 minutes ago
gilfoyle_7 | 25 minutes ago