I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.
I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.
Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).
At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.
They are cracking software and math, not finance, law or medicine or whatever it is you do.
Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.
Are you able to give hints to some of those use cases? Recent Opus and Fable seem to have made big leaps in art and music, and I don't think all of that is the longer context window since Sonnet 3.x. But those are both things that can be created with code and mathematical thinking as well. I definitely wouldn't want to use recent models for writing though (not that I ever wanted them to write for me anyway).
I also use LLMs for a difficult non-coding task (historical HTR), and my experience is that Gemini Argon is much better than its competitors on that.
Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.
Still, those are much better failure modes than hallucinating, which powerful models are still prone to.
That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.
Uhm, I'm not a programmer but why isn't this a good look? I would want my engineers to fix problems that they notice in passing. (Edit: the "unnecessary work" angle is what irked me in the article)
Good SWEs get a feel for the minimum amount of changes to fix something. Fewer changes usually minimizes the chance for new problems to pop up.
Junior SWEs usually try to do too much.
This. There should be 2 PRs. The core fix, the the unrelated fixes. Because I guarantee sometimes those unrelated fixes will cause regressions so they have to be scrutinized too.
I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.
It depends on how much you're willing to wait. I used Argon a few times and while it did some interesting things, it was so slow that I couldn't fit it in my workflow. GPT Sol was great and 5-10x faster.
I feel that right now Hydrogen is the only viable model Google has in cost/speed-benefit. It's cheap and makes sense for general use in the $20 month plan with Youtube and storage, but not as a prime coding model.
it depends but IMO Carbon is best at the moment with code reviewing and documentation writing. Opus is still my go-to model for code generation but at moment Carbon does better job for those auxiliary jobs.
Why not Fable? I've mostly been using GPT side models and have recently been dipping my toes into Claude models. I went big with Fable 5.2 as it was "the best" the offered. Apart from token usage, is there any particular reason to use Opus?
Using them together is faster and cheaper and not quite as good as just Carbon (if I read that right). So that's a pretty good sweet spot for most use cases.
It’s hella slow for sure and agy harness makes this worse by telling it to read a ton of files before making any changes!! But once it starts, its solutions are very good and it usually continues until the task is fully completed! The output was so good at times, I was honestly surprised by how thought-through and properly implemented everything was. It implemented edge cases I would have missed and made decisions that I would call sensible. For example adding a —-flag that absolutely improved the product.
Code review still catches medium priority issues but that’s the same with all of the models.
I’m now putting Carbon on bigger tasks that I don’t care how long they take. As I am writing this I’m having dinner while Carbon slowly chops away at a task queue on my server.
All in all, Carbon is great, Hydrogen is great, Google is killing it!
(agy is still bad though, but you can tweak the “read a million files first” behavior with your system prompt)
My experience is similar: Carbon found better ways to solve open-ended problems than Fable. In these cases, where I don’t have a solid design ahead of time, it’s OK to wait.
Definitely agree with this, I regularly find that carbon writes less overengineered code and considers more of my existing architecture without additional documentation or prompting. Maybe that's what it's doing with all of its "wasted" turns?
That said, it's incredibly slow. I haven't played much with the orchestrator that agy added but if it's Carbon occasionally delegating to Hydrogen, that might be a little backwards. I feel like what I actually want is Hydrogen delegating more difficult tasks to Carbon instead.
I'd guess that it's because Google eats its own dogfood, and if they didn't train the models to be overly meticulous, the existing codebase would implode under a tsunami of slop.
It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:
1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.
2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.
3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.
It’s unlikely to be deliberate tbh, it’s more likely the data, algorithms and the training environments aren’t quite as good as Anthropic, Open AI is a lot closer to Google than people want to admit.
Since when? Google recently disobeyed some legal requirements because "haha, couldn't catch me / too late now bitch". (This is when they cut down a forest for a datacenter)
Internet data is only good for some parts of model training. That is why there is now a huge cottage industry that creates custom datasets, RL envs, etc.
Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.
Google's frontend web servers routinely time out connections from agy (the antigravity cli) when using Carbon, wasting tons of work. Like any effort level more than "low" is completely unusable.
It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.
One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.
Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.
Yup I use carbon hours daily with 10+ minutes between requests at the extreme without timing out. That timeout has to be the harness. I also haven't seen API errors since release day.
I never got any google model to work using their infra - 429, billing errors and timeouts. As someone wrote here a few months ago - it's an exercise in managing frustration. So I had to resort to openrouter.
Lots of Google tooling is the same for me, simply unusable whenever I try even simple stuff.
I think no one tests this as a "non-Google employee" or something, they all have specific testing backends or something, and once deployed to the public, there is no one responsible for the experience of non-Googlers and no one testing it, that's why most of the Google stuff has such poor UX because of hundreds of small cuts all over the experience.
Google never had the mandate to align design standards across internal and external tooling. BigQuery and some of the other analytics tools are great though, I'd be surprised if staff didn't use the same internally - maybe someone here can confirm?
So in short, as is usually claimed about Google products from bazel to tensorflow to Kubernetes: they may work better inside Google than outside of it.
4. they are so behind that they can not create effective datasets like other frontier labs do. So instead they do all the tricks the Chinese labs do: using RL with the single purpose of benchmaxxing the AA score.
Look how all Chinese models, or haiku/luna class US models behave, you can see the same pattern: they think too much and only work well with a subset of tasks
Which is weird, because Google is in the best position for getting training data: they have YouTube (which I guess is why Google's models are good at multimedia), as well as the Google crawler. Which effectively websites cannot opt out of if they want to be Google searchable. So Google's bot probably is the _only_ bot (scraper) that hasn't been blocked from a large portion of the internet.
Not only that, they pioneered some of the key technologies... Who isn't still on transformer models? Also funny when you consider Meta is in a weird similar boat, introduced key things, so far behind the frontier.
To go along with that they're extremely slow (massive orgs) and don't feel their existence is at stake.
If GPT slips meaningfully behind, OpenAI is dead. The same goes for Anthropic. Both would die extremely quickly.
If you're within Google that is not the sense you have about the current context, nor within Meta or ByteDance or Amazon or Microsoft. They certainly view all of this as very important, they don't think they'll die within a year or two if they don't stay on top of the frontier. Meta has run from behind endlessly, as has Google, and their businesses are doing just fine (for now). If OpenAI were in that position, there would be no reason for them to exist and they would be dead by now.
Microsoft likely doubts their existence is at risk just because they can't field a frontier model. For now they don't see Office / Windows being killed off as a short-term risk (plus they have Azure and other products).
AWS is extraordinarily difficult to replicate, it's more akin to trying to compete with dozens of steel mills (industrial infrastructure) than a mere piece of software. And the rest of Amazon's cash spigot is advertising on its retail platform, which is facing close to zero threat from the frontier race, so Amazon can take its time and endlessly screw-up on AI (and so it has).
A frontier LLM can't manufacture a smartphone or laptop for you, and certainly can't manufacture 100 million of them with a zillion dollar manufacturing contract with TSMC et al., Apple isn't particularly worried. They want intelligence on their platform, they're not overly concerned about needing to own a frontier model. They can afford to be patient and get it right.
Capital is more powerful than AI. The capital owns the GPUs the AI requires, the capital owns the datacenters, the capital owns the grid, the capital owns the generation, the capital owns the factories, the capital owns the machines that produce semiconductors, the capital owns the patents. Apple, Microsoft, Google, Amazon, Nvidia, Meta, etc control the capital, financially they're far more powerful than most mega banks these days. And if Apple needed $100 billion for a project, it's one human phone call to Berkshire Hathaway. For now AI is feeble standing next to capital.
AI is the little bitch of capital, to put it bluntly. AI is on its knees begging capital for more compute: please sir, may I expand?
I think raw base text training data may be decreasingly important vs good RL environments/gyms, and maybe less clear that Google wins by default there?
Seeing how every new benchmark that has come out has Carbon doing well, I'd have expected the benchmaxxing allegations to die out. But they've been ok incredibly persistent. It's jagged like all frontier models. You could argue about it's warts, but benchmaxxing is unfounded. Did it benchmaxx every public and private dataset? Is that what RL is perhaps?
Google has a huge flow of user data; they were the first to figure out about the Transformer, yet they’re lagging behind in the race. This is very strange, although their flash models are quite good, they’re expensive compared to the Chinese ones.
1) "Gemini 4 Hydrogen is adapted from Gemini 4 Carbon." From https://deepmind.google/models/model-cards/gemini-4-hydrogen... . Also, the Agy team is pushing is that Carbon is a great planner, and then use hydrogen as the worker. They want people to use it.
2) lots of people inside Google use carbon. If they didn't care about it, it wouldn't have been pushed on all of Google.
3) Deepmind is competing with cloud and others. they are the only Frontier Lab that actually has to compete with other business units within an org for compute. GCP and others want that compute as well.
1. Small, efficient models are often derived from "the best" models. That doesn't mean "the best models" are efficient too.
2. GP said they don't care about cost, not they don't care about the model
3. Competing for existing compute + being able to assign the projects own capex investment for new compute provides more compute than needing to assign all compute via new capex investment. But it's hard to say if that's actually the type of case or not unless Google wants to tell everyone publicly - so I'd take that point with a grain of salt in either way.
I find this behavior to be common across most of the big models -- Sol 6.1 does it a ton, adding lots of super defensive programming cruft for things that A) by design can never happen and B) should fail loud anyway, despite prompting. It gets a little better after steering tweaks but still common.
A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.
It really looks like something is broken more than just being inefficient.
Theo seems to dislike anything that isn't Anthropic Claude. Suspicious, to say the least. On CursorBench, a good measure of real-world coding tasks, Gemini 4 Carbon comes out top: https://cursor.com/evals
Every time I see a Theo tweet he's judging something on the most absurd metrics. Today for example, he was saying Astra 6.2 is bad because when he gave it a single line prompt telling it to create a new level for a game it used a name for the level that he'd seen before. Judging a model on how often it reuses a name is one of the most ridiculous benchmarks I've ever seen.
As far as I can tell, all his judgements are based on similarly absurd metrics, I don't know why people treat his preferences as valuable.
No, they have not. But they don't seem to have the latest ones. I think that's because OpenAI announced that they will stop providing their models to Cursor.
How recent is that video? Google just reduced time to first token (https://x.com/_mohansolo/status/2122821676249625014): "Time to first token (TTFT) for Carbon is now 1.5x faster on Antigravity. Will follow soon with Argon too.
Lots of work went into this so you all could have a faster experience. More to come "
Then that's "test-time scaling" again and Google then showed that they can reach frontier level benchmark results using it. It's pretty cool as a technology actually (the demonstrated capability), even if the model is not so useful when it's slow.
Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?
Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.
i'm having all sort of problems with the gemini carbon lag.
using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.
Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED).
Gemini couldn’t finish this response
I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.
I only got around the issue by strictly setting an effort budget or a stopping rule.
Carbon is extremely good for particular research questions I'm doing, and the only hope at the moment is that they don't eventually nerf it down, especially if topics like these about the speed might raise concerns.
In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.
So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!
Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.
Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.
Yeah, Carbon is good when the problem needs some imagination, when a better solution is worth waiting for. Fable 5.2 moved towards faster or more efficient responses. So it’s more practical for most tasks, but tends to be less creative.
Might be the only one in the thread but ever since Gemini 4 Argon. I’ve become a huge fan of the Google models. I do use Carbon most of the time since it’s become available but notice that i also am falling back more and more on Argon again without much difference. For security purposes argon has far fewer triggers.
I only use the Google models as orchestrator for how they talk and analyse. I let it delegate everything to Astra / Fable / Opus models
I tell t to send it to Claude -p opus 5.5 and or Astra. I made the model mappings explicit in GEMINI.md. I don’t ever leave agy anymore and anytime I need to I dislike how Astra and Claude always beat around the bush
I do like Argon, but usage continues to be an issue for me. The economics of everything pales in comparison to the value you get with DeepSeek. I'll end up using my limited opaque usage based limit with Antigravity to have Argon review something or maybe write a plan, but the bulk of the work has to be done with cheaper models.
I use it similarly but have the subscriptions to do other worker models. But yea I tell it only to delegate except when the change is something tiny. But most of it is done n via delegation.
Similarly $20 on opus in Claude code seemed to last me 10x longer than argon in antigravity. And for some reason there doesn’t seem to be an opus equivalent in antigravity - but in Gemini I can select “pro” which doesn’t exist in antigravity (flash or argon only). Antigravity $20 plan also gives a comical amount of Anthropic credits… it can literally only answer one question for me before hitting limits.
I've also become a fan. For most of my current projects I'm using Agy with the Carbon orchestrator, but switch to Hydrogen when I'm bantering back & forth. It's convenient, with things like Swarm, to have an orchestrator that can automate the selection of models for sub-agents. I've used quite a few harnesses -- both for local and cloud dev -- and although Claude Code has some clear feature benefits over Antigravity (for desktop/sandbox control & interaction), I'm generally very happy.
really glad to see _this_ kind of analysis of these models. These type of analysis is much more practically useful than some context-less (especially with regard to time) number of "how good it is".
I just got a 6 month max subscription via the antigravity science program. I've tried to start using it and found both the harness to be terrible and the model to be very slow.
It might do good work but I'm having a hard time figuring out if it is.
It's been jarring & confusing to go from a year and a half of Claude Code max plan, and equivalent of Codex, to Google agy & their models
My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy
While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models
My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem
> Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.
I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.
We have only 146 battles [1] with Carbon, so I kicked off an analysis with skepticism we'd have anything to add here. I was wrong. Carbon is that slow.
Biggest differences:
- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.
- Output tokens 16.4k for Carbon vs 6.8k for opponents.
- Tool calls: 21 for Carbon vs 11 for opponents.
All three of these are statistically significant.
The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).
Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.
- Both finished
83 carbon wins, 52 losses.
- Early votes
2 carbon wins, 40 losses!
Basically, Carbon is good when you're willing to wait.
I noticed a similar issue going from Opus 4.6 to 5.5. I also however upgraded my harness during that time, and I found out that it was injecting something like “Be helpful, read in between the lines and proactively go beyond the user’s request” at each turn. I immediately patched that out and saw decent improvement in that area. TFA mentions they tried different harnesses though, so yeah it’s prob baked in to whatever RL setup they’re using to train the original model.
This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good
This. Yeah I think if one knows what they are doing, e.g. a seasoned developer. They can get a lot of mileage out of an LLM that is good at instruction following. which is why i like codex models. Astra may not be as good as Opus but the workhorse really is 6.2 sol. The best workhorse when it comes to the pareto frontier of `cost vs accuracy` or `inference time vs accuracy`.
I am pretty sure caching is busted, atleast on github copilot, I tried doing it via pi harness however cannot seem to get any cached inputs post 5 minutes.
Carbon has been slow, but excellent at deep dive analysis of some of my projects, finding optimizations other models missed AND explaining them in a clear and concise manner, even as the conversation develops and really becomes four or five subthreads on the opportunities that it brought up. Maybe that massive context window helps.
Carbon is slow, but I've also realized that it works really well. I don't need to get involved or keep checking the results over and over again. I think it's pretty close to Opus 5.5 in terms of work quality.
I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).
The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)
> Closing loopholes the ticket had already ruled out. Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.
This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.
It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.
Using any of the latest models within Antigravity is incredibly slow. I use Claude Desktop for sending off coding agents into the cloud and I get about 10x more done with it than I can do with Antigravity + Google models. The only advantage to the Google models over Claude so far that I've noticed is they are way better at writing copy.
Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.
I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.
user_7832 | 13 hours ago
I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.
Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).
incognition | 13 hours ago
p_l | 10 hours ago
netrap | 3 hours ago
p_l | 2 hours ago
auyez | 12 hours ago
londons_explore | 10 hours ago
The existence of AI able to design a smarter AI I would not say is a given.
wellcolormesurp | 12 hours ago
Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.
oblio | 8 hours ago
Based on...? Wishful thinking?
SyneRyder | 11 hours ago
knollimar | 7 hours ago
I haven’t used a lot of Carbon though.
vintermann | 10 hours ago
Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.
Still, those are much better failure modes than hallucinating, which powerful models are still prone to.
SubiculumCode | 13 hours ago
ahsillyme | 13 hours ago
horacemorace | 12 hours ago
bot403 | 10 hours ago
shwetanshu21 | 10 hours ago
Most of the times it has to do with refactoring downstream services or dependencies or just unexpected behaviour that shows up.
horacemorace | 13 hours ago
weatherlite | 10 hours ago
glimshe | 10 hours ago
I feel that right now Hydrogen is the only viable model Google has in cost/speed-benefit. It's cheap and makes sense for general use in the $20 month plan with Youtube and storage, but not as a prime coding model.
antupis | 8 hours ago
eknkc | 8 hours ago
antupis | 7 hours ago
bagacrap | 3 hours ago
thisisauserid | 3 hours ago
Gemini gives you a lot of options.
artdigital | 7 hours ago
It’s hella slow for sure and agy harness makes this worse by telling it to read a ton of files before making any changes!! But once it starts, its solutions are very good and it usually continues until the task is fully completed! The output was so good at times, I was honestly surprised by how thought-through and properly implemented everything was. It implemented edge cases I would have missed and made decisions that I would call sensible. For example adding a —-flag that absolutely improved the product.
Code review still catches medium priority issues but that’s the same with all of the models.
I’m now putting Carbon on bigger tasks that I don’t care how long they take. As I am writing this I’m having dinner while Carbon slowly chops away at a task queue on my server.
All in all, Carbon is great, Hydrogen is great, Google is killing it!
(agy is still bad though, but you can tweak the “read a million files first” behavior with your system prompt)
joshuanapoli | 6 hours ago
nautilus12 | 5 hours ago
dhruvmittal | 50 minutes ago
That said, it's incredibly slow. I haven't played much with the orchestrator that agy added but if it's Carbon occasionally delegating to Hydrogen, that might be a little backwards. I feel like what I actually want is Hydrogen delegating more difficult tasks to Carbon instead.
jrflo | 13 hours ago
murderfs | 11 hours ago
tristanj | 13 hours ago
1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.
2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.
3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.
fsh | 12 hours ago
dpe82 | 12 hours ago
andy_ppp | 11 hours ago
exe34 | 8 hours ago
andy_ppp | 8 hours ago
exe34 | 7 hours ago
LoganDark | 8 hours ago
andy_ppp | 8 hours ago
LoganDark | 7 hours ago
tibbar | 6 hours ago
Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.
andy_ppp | 5 hours ago
htrp | 2 hours ago
PaulHoule | 5 hours ago
teknologist | 12 hours ago
zarzavat | 12 hours ago
kyrra | 5 hours ago
aberoham | 10 hours ago
It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.
One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.
Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.
p_l | 10 hours ago
elij | 2 hours ago
kingleopold | 8 hours ago
yard2010 | 7 hours ago
embedding-shape | 7 hours ago
I think no one tests this as a "non-Google employee" or something, they all have specific testing backends or something, and once deployed to the public, there is no one responsible for the experience of non-Googlers and no one testing it, that's why most of the Google stuff has such poor UX because of hundreds of small cuts all over the experience.
antihipocrat | 6 hours ago
kccqzy | 3 hours ago
kyrra | 5 hours ago
htrp | 3 hours ago
vintermann | 10 hours ago
npn | 6 hours ago
Look how all Chinese models, or haiku/luna class US models behave, you can see the same pattern: they think too much and only work well with a subset of tasks
Doohickey-d | 6 hours ago
giancarlostoro | 5 hours ago
aswegs8 | 5 hours ago
adventured | 4 hours ago
If GPT slips meaningfully behind, OpenAI is dead. The same goes for Anthropic. Both would die extremely quickly.
If you're within Google that is not the sense you have about the current context, nor within Meta or ByteDance or Amazon or Microsoft. They certainly view all of this as very important, they don't think they'll die within a year or two if they don't stay on top of the frontier. Meta has run from behind endlessly, as has Google, and their businesses are doing just fine (for now). If OpenAI were in that position, there would be no reason for them to exist and they would be dead by now.
Microsoft likely doubts their existence is at risk just because they can't field a frontier model. For now they don't see Office / Windows being killed off as a short-term risk (plus they have Azure and other products).
AWS is extraordinarily difficult to replicate, it's more akin to trying to compete with dozens of steel mills (industrial infrastructure) than a mere piece of software. And the rest of Amazon's cash spigot is advertising on its retail platform, which is facing close to zero threat from the frontier race, so Amazon can take its time and endlessly screw-up on AI (and so it has).
A frontier LLM can't manufacture a smartphone or laptop for you, and certainly can't manufacture 100 million of them with a zillion dollar manufacturing contract with TSMC et al., Apple isn't particularly worried. They want intelligence on their platform, they're not overly concerned about needing to own a frontier model. They can afford to be patient and get it right.
Capital is more powerful than AI. The capital owns the GPUs the AI requires, the capital owns the datacenters, the capital owns the grid, the capital owns the generation, the capital owns the factories, the capital owns the machines that produce semiconductors, the capital owns the patents. Apple, Microsoft, Google, Amazon, Nvidia, Meta, etc control the capital, financially they're far more powerful than most mega banks these days. And if Apple needed $100 billion for a project, it's one human phone call to Berkshire Hathaway. For now AI is feeble standing next to capital.
AI is the little bitch of capital, to put it bluntly. AI is on its knees begging capital for more compute: please sir, may I expand?
pfannkuchen | 4 hours ago
efromvt | 5 hours ago
htrp | 4 hours ago
icepush | 2 hours ago
owkman | 4 hours ago
unbrice | an hour ago
moffkalast | 6 hours ago
Danywalery | 6 hours ago
kyrra | 4 hours ago
2) lots of people inside Google use carbon. If they didn't care about it, it wouldn't have been pushed on all of Google.
3) Deepmind is competing with cloud and others. they are the only Frontier Lab that actually has to compete with other business units within an org for compute. GCP and others want that compute as well.
(Googler, opinions are my own)
zamadatix | 4 hours ago
2. GP said they don't care about cost, not they don't care about the model
3. Competing for existing compute + being able to assign the projects own capex investment for new compute provides more compute than needing to assign all compute via new capex investment. But it's hard to say if that's actually the type of case or not unless Google wants to tell everyone publicly - so I'd take that point with a grain of salt in either way.
serioussecurity | 4 hours ago
lp92 | 13 hours ago
That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.
With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.
gazebo2 | an hour ago
Lerc | 12 hours ago
It really looks like something is broken more than just being inefficient.
teknologist | 12 hours ago
LiamPowell | 11 hours ago
As far as I can tell, all his judgements are based on similarly absurd metrics, I don't know why people treat his preferences as valuable.
knollimar | 7 hours ago
ywvcbk | 6 hours ago
teknologist | 3 hours ago
https://openai.com/index/our-decision-on-cursor-following-it...
spprashant | 4 hours ago
96khz | 12 hours ago
Lerc | 11 hours ago
xnx | 5 hours ago
Lots of work went into this so you all could have a faster experience. More to come "
dario_amodei | 12 hours ago
NoahZuniga | 12 hours ago
joedwin | 11 hours ago
lukax | 11 hours ago
kzrdude | 11 hours ago
nandanadileep29 | 11 hours ago
kristjansson | 10 hours ago
logicchains | 10 hours ago
cybice | 10 hours ago
ahmadyan | 10 hours ago
mengqijin | 10 hours ago
But with simple tasks, Carbon is way too slow, and much higher cost.
timtam | 9 hours ago
I only got around the issue by strictly setting an effort budget or a stopping rule.
resters | 7 hours ago
5555watch | 6 hours ago
In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.
So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!
Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.
Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.
joshuanapoli | 6 hours ago
hetspookjee | 6 hours ago
moffkalast | 6 hours ago
xnx | 5 hours ago
chubot | 4 hours ago
WarmWash | 4 hours ago
hetspookjee | 2 hours ago
htrp | 2 hours ago
wronglebowski | 2 hours ago
hetspookjee | 2 hours ago
soared | 2 hours ago
eitally | 43 minutes ago
oytis | 6 hours ago
timcobb | 5 hours ago
dada216 | 5 hours ago
It's pretty far beyond other models (Opus 5.5 is at 128k).
gdevenyi | 5 hours ago
It might do good work but I'm having a hard time figuring out if it is.
lmeyerov | 5 hours ago
My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy
While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models
My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem
WeZZard | 4 hours ago
bagacrap | 3 hours ago
> Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.
I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.
erji77 | 3 hours ago
seizethecheese | 2 hours ago
Biggest differences:
- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.
- Output tokens 16.4k for Carbon vs 6.8k for opponents.
- Tool calls: 21 for Carbon vs 11 for opponents.
All three of these are statistically significant.
The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).
Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.
- Both finished 83 carbon wins, 52 losses.
- Early votes 2 carbon wins, 40 losses!
Basically, Carbon is good when you're willing to wait.
[1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel
jedbrooke | 2 hours ago
This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good
ghm2199 | 2 hours ago
niltecedu | 2 hours ago
totalhack | an hour ago
fwee | an hour ago
I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).
The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)
Anyway, great article!
Otterly99 | an hour ago
This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.
It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.
jaren | an hour ago
bugzbenny | 56 minutes ago
Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.
I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.
0xbadcafebee | 41 minutes ago