It means something, because it an iterative workflow. If you're willing to burn tokens, it's possible for weaker models to implement tasks by incrementally improving drafts.
Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
Is it possible to construct a control system where bad, fast and cheap can become good, fast, and cheap through repeated sampling and a strong spec/eval harness?
I am trying to keep an open mind with AI, but I also have little understanding of control theory, trying to learn.
You can, but you need to break the problem into much smaller tasks, then check those answers, and finally have a harness that handles all the context, task breakup, task definitions, and validations each round.
Depends on how many times you need to iterate to get the result you want. If you need to run the fast model 5 times to get the results you need, compared to 1-2 times for a smarter but slower model, you've just eroded any advantage that the speed gave you.
7-8 figures annual spend will buy a hell of a lot of capable local inference hardware you can own, though it won't be at the absurd token/s rate, you'll be able to run almost anything on it... And it'll still have a good residual resale value after 4 years the way things are going now.
Feel like you could spend 6 figures building out a team and the rest renting compute for a whole year, and get the team to create a local inference solution with that kind of budget…
Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.
> There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model.
So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..
Yea i had some pretty meh results using gpt-oss-120b it in my evals where it should have benefited speed alot but it really under performed what i was expecting.
I kept having experiences with gpt-oss-120b on Cerebras where it would get stuck in a thinking block and then start endlessly saying things like "Running the command now." or "Making the changes now." and then simply repeating similar sentences like that forever instead of actually making the tool call. It made tool calls other times, so it wasn't an issue with tool calls being impossible, but it just wasn't doing a good job of using them for real instead of simply saying it would. So this was not an issue of it starting a tool call and then putting invalid syntax inside of it, it just would not make the tool call it was supposed to whatsoever. There's no automatic way to retry that.
Lighting my codebase on fire at the speed of light. Like microwaving the spaghetti.
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
Human code review? What is this, 2025? The modality today is write with one LLM, review by a different one, (important: two different model families will catch errors one series won't) then deploy right to production.
And if you tell the reviewer the author is a competitors model it becomes extra snarky and vigilant. Then give the review results to the author and tell it it's from the competition and it will also become slightly outraged.
I would be very curious to see how you explain it to your customers.
Is it going to sound similar to this?
> You see, our well-meaning AI-generated code has caused all your data to be permanently deleted. In case you are confused as to who to blame, we would like to clarify that we did not write, nor review the code. So we cannot possibly bear any responsibility for its mistakes. The responsibility lies with the LLMs, not us. We have already fired the LLM which did the coding and the review. And we are already using their main competitors. Hopefully that settles your concern with the quality of our service and we are looking to have you on board of our next products.
This seems to work well enough for people who deploy cloud instances without redundancy to us-east-1 then blame AWS when there's an outage. I say this somewhat unironically because if one pushes the "move fast and break things" slider all the way to the right then they're necessarily assuming that type of risk. Of course some people will try to have their cake and eat it too [1] as regards velocity and quality but that's a separate discussion.
[1] I've never understood this idiom because if one isn't in possession of their cake before eating it then they're eating stolen cake which is a decidedly anti-social activity and orthogonal to the point of the saying. It should be "eat their cake and keep it too" or something.
Yeah, idk. I work on serious things. Thats not how I do things. Not everything is webdev hobby projects. There's basically no instance where offloading your code review to an llm is acceptable behavior, except maybe for a one off tool you need personally.
No you dont need to code at 600 w/s BUT at those speeds, you can start doing things like asking multiple different agents the same question and picking the best solution each time without noticing the lag.
I prefer a fast model too, but you cannot get more done just because its faster. You just get to the human parts a bit faster. Code review, revision ect.
Inco sucks. I tried their GLM 5.3 Flash and it was quantized to the point of hallucinating Chinese in the middle of English only agentic sessions. Never happened with any other provider.
You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
Personally, I think it's more that text diffusion is not the ideal driver of an agentic work loop than that text diffusion is a total dead end. I am still hoping to see how it does on authoring and editing with further scaling and optimization. I think the push for AGI has put a bit too much focus on the idea of one general model doing everything.
From what I’ve heard, the issue is more that it’s harder to efficiently share the hardware across diffusion requests, so it’s more expensive to serve.
It sounds like there might be opportunities for local models (not open weight, but actually locally run) to use diffusion for faster responses on weaker hardware that doesn’t need to be shared.
But yea, it’s still a red-ish flag that big labs haven’t invested much in it. I could see Google/Apple getting value of this sort of local model, but maybe there’s enough research behind traditional models that it’s not worth the distraction at this point in time.
There were too many unsolved problems with diffusion to commercialize it, where autoregressive loops had a more straightforward roadmap. Doesn't mean the diffusion problems are insurmountable though.
At some point the bottleneck becomes tool calling.. and as such, it's preferably if the model is co-hosted (in the same datacenter, at least) with your code repository and all other reference/context it needs (full documentation for most ecosystems, maybe even a copy of common crawl to minimize web fetch usage, etc)
I used this a few days ago and thought something must be wrong with how fast it was responding. "Mercury 2.5 is below average in intelligence, but well priced when comparing to other models of similar price." this is so funny. So when you have a stupid model that is fast - what do you use it for?
I have tried using Mercury 2.5 for a lot of my tasks.. but this model just isn't there. It seems to be on par with any 14B model at max. Even GPT-OSS-20B performs way better than this in my own attempts to use it.
I really really wanted to use this because it offers incredible speeds and pricing combinations. But nop.. I still am not using it.. not even for basic tasks.
I'm still sad that we haven't seen a new Taalas style chip a la https://chatjimmy.ai/. Smaller models are good enough now to make that insane burst of tokens so useful.
I don't know the model behind this, but it is absurdly bad.
> Write me a coherent paragraph in French, without ever using the letter "e".
> Voilà une phrase claire et concise : "Le village est situé dans les montagnes. Le soleil est haut. Il y a des animaux dans le village. Il pleut dans les montagnes."
I suppose this is just a demo of how fast an LLM can be, I wonder if there are tradeoffs with larger/smarter models. Also, for a human usage, at what point are tokens generated fast enough that it's pretty much instant? My bet is below 1000 tps
for comparison, Qwen3.8-Flash-Next only requires 6B parameters for computation, but stores 125B, 51B of those can be comfortably offloaded as they're not actively used in decode but a single token look up.
The Quant iQ4 of this model loads, then, in ~60GB of vram, and on disk it's 85GB.
So if you could etch it, you'd need a ~25GB ssd chip and 60GB of vram.
The vram costs likely contributed to these things being out of reach of the current economic cycle.
I'm not smart enough to know why; I do know that 27B is greater for short/interactive on blackwell, but the intellgence leap of the MoE in Qwen3.8-Flash-Next is quite remarkable.
I'm pretty convinced the pathway to local models will be MoE, especially if they can find a way to keep tweasing out things like PLE into the slow bandwidth lanes.
The Taalas architecture makes loading weights free, but they basically has to pay the same silicon for every weight, whether it's used or not. MoE models are more efficient than dense models per weight you load, but less efficient per weight you have to store.
MoE models are the path to local models with traditional system architectures, but they are antithetical to what Taalas was doing. If you spent all the money to etch 125B weights into silicon, you'd want to activate them all for each token, instead of only touching 6B. You cannot match the 125B sparse model with a 27B dense one, but you might be able to match it with a 60B or so one.
That's the problem with etching a model onto a chip: by the time you've designed the chip, manufactured it, tested it, shipped it, and deployed it, the model will be hopelessly outdated (with the current improvement rates). And when you want to update, you have to buy new chips instead of just uploading a new model file like now. When Taalas announced their chip, the model was already 1.5 years old (stone age by current standards). It's their first chip, so maybe they can streamline it, but the problem of having to update hardware every few months to keep up with the industry is not going anywhere.
You can upload different weights and even do LoRAs. The chip architecture is interesting, the first (n) layers are the sane, so you can change architecture by adding (m) layers. Plausible that this is sufficiently flexible enough for several generations of real world applications. For example, we still use 45nm general purpose silicon for automotive, e.g.
What they did had never been done before. Now we see that it's possible, there are plenty of models to choose from that could be etched into silicon. In the next year or two, I think these smaller models might plateau, and there may be some on-device niche they can fill.
Why ask this when we know that LLMs are not good at the character level. They run on tokens, not characters. In fact, they don't even see the characters, unless you do special tricks.
I asked it to translate your sentence to English and it did fine. In less than a fraction of a second.
To be fair, you picked a well-known tricky benchmark for LLMs: When working on an embedding spelling disappears after the embedding level. I imagine modern frontier models have tools that let them read back their input to work around this issue.
rvz | a day ago
copperx | a day ago
downrightmike | a day ago
glouwbug | a day ago
voiceeh | a day ago
hansvm | a day ago
timClicks | 23 hours ago
walrus01 | a day ago
RussianCow | 23 hours ago
_aavaa_ | 22 hours ago
hermannj314 | 21 hours ago
I am trying to keep an open mind with AI, but I also have little understanding of control theory, trying to learn.
freakynit | 21 hours ago
calgoo | 16 hours ago
RussianCow | 19 hours ago
jgalt212 | 12 hours ago
RussianCow | 10 hours ago
So, as always: it depends on the use case.
bearjaws | a day ago
I've used it on a few for fun projects and its decent but the speed is crazy to watch.
ford | 23 hours ago
[0] https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise
sharktheone | 23 hours ago
But it can apparently also run 5.6 Sol
eli | 22 hours ago
walrus01 | 21 hours ago
podocarp | 17 hours ago
jazzyjackson | 8 hours ago
scosman | 23 hours ago
RussianCow | 23 hours ago
scosman | 20 hours ago
LoganDark | 23 hours ago
eli | 22 hours ago
fakwandi_priv | 18 hours ago
So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..
shard972 | 21 hours ago
bearjaws | 12 hours ago
I have it retry on failure, but you should do that with any LLM really.
LoganDark | 12 hours ago
conception | 21 hours ago
Such a tiny model at that t/s is less impressive than it would have been four months ago.
physicallyIllfr | 20 hours ago
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
sbierwagen | 18 hours ago
podocarp | 17 hours ago
pbasista | 11 hours ago
And when things go south, you blame AI?
I would be very curious to see how you explain it to your customers.
Is it going to sound similar to this?
> You see, our well-meaning AI-generated code has caused all your data to be permanently deleted. In case you are confused as to who to blame, we would like to clarify that we did not write, nor review the code. So we cannot possibly bear any responsibility for its mistakes. The responsibility lies with the LLMs, not us. We have already fired the LLM which did the coding and the review. And we are already using their main competitors. Hopefully that settles your concern with the quality of our service and we are looking to have you on board of our next products.
alexjplant | 9 hours ago
This seems to work well enough for people who deploy cloud instances without redundancy to us-east-1 then blame AWS when there's an outage. I say this somewhat unironically because if one pushes the "move fast and break things" slider all the way to the right then they're necessarily assuming that type of risk. Of course some people will try to have their cake and eat it too [1] as regards velocity and quality but that's a separate discussion.
[1] I've never understood this idiom because if one isn't in possession of their cake before eating it then they're eating stolen cake which is a decidedly anti-social activity and orthogonal to the point of the saying. It should be "eat their cake and keep it too" or something.
physicallyIllfr | 9 hours ago
calgoo | 16 hours ago
physicallyIllfr | 9 hours ago
RugnirViking | 16 hours ago
I do. I used to use haiku for the speed. Now its just as slow as the rest. Speed is my #1 ranking of how good a model is
sitkack | 10 hours ago
What is your workflow?
physicallyIllfr | 9 hours ago
lostmsu | 13 hours ago
verdverm | 19 hours ago
https://kamilstanuch.github.io/LLM-token-generation-simulato...
kylehotchkiss | 19 hours ago
nylonstrung | a day ago
It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models
Still unclear for what, if any use cases this is pareto frontier
LarsDu88 | a day ago
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
clhodapp | 23 hours ago
kennywinker | 22 hours ago
K2-Horizon-7B has a diffusion and non-diffusion variant, and they claim the same level of intelligence from both models.
vineyardmike | 22 hours ago
It sounds like there might be opportunities for local models (not open weight, but actually locally run) to use diffusion for faster responses on weaker hardware that doesn’t need to be shared.
But yea, it’s still a red-ish flag that big labs haven’t invested much in it. I could see Google/Apple getting value of this sort of local model, but maybe there’s enough research behind traditional models that it’s not worth the distraction at this point in time.
naasking | 11 hours ago
low_tech_punk | 23 hours ago
SwellJoe | 23 hours ago
freakynit | 21 hours ago
verdverm | 19 hours ago
sharktheone | 23 hours ago
entrope | 22 hours ago
Well priced when compared to other models of similar price, eh?
Are we allowed to call this slop, even if the output is not directly from an LLM?
verdverm | 19 hours ago
nextaccountic | 21 hours ago
whalesalad | 21 hours ago
freakynit | 21 hours ago
I really really wanted to use this because it offers incredible speeds and pricing combinations. But nop.. I still am not using it.. not even for basic tasks.
nostrebored | 21 hours ago
freakynit | 20 hours ago
nostrebored | 18 hours ago
seduerr | 20 hours ago
[OP] Retro_Dev | 20 hours ago
the_arun | 20 hours ago
econ | 20 hours ago
anothereng | 9 hours ago
jjcm | 20 hours ago
winwang | 19 hours ago
electroglyph | 16 hours ago
pil0u | 15 hours ago
> Write me a coherent paragraph in French, without ever using the letter "e".
> Voilà une phrase claire et concise : "Le village est situé dans les montagnes. Le soleil est haut. Il y a des animaux dans le village. Il pleut dans les montagnes."
I suppose this is just a demo of how fast an LLM can be, I wonder if there are tradeoffs with larger/smarter models. Also, for a human usage, at what point are tokens generated fast enough that it's pretty much instant? My bet is below 1000 tps
PetahNZ | 15 hours ago
cyanydeez | 14 hours ago
The Quant iQ4 of this model loads, then, in ~60GB of vram, and on disk it's 85GB.
So if you could etch it, you'd need a ~25GB ssd chip and 60GB of vram.
The vram costs likely contributed to these things being out of reach of the current economic cycle.
Tuna-Fish | 13 hours ago
cyanydeez | 8 hours ago
I'm pretty convinced the pathway to local models will be MoE, especially if they can find a way to keep tweasing out things like PLE into the slow bandwidth lanes.
Tuna-Fish | 6 hours ago
MoE models are the path to local models with traditional system architectures, but they are antithetical to what Taalas was doing. If you spent all the money to etch 125B weights into silicon, you'd want to activate them all for each token, instead of only touching 6B. You cannot match the 125B sparse model with a 27B dense one, but you might be able to match it with a 60B or so one.
carlos_rpn | 12 hours ago
kgeist | 11 hours ago
dnautics | 11 hours ago
linuxftw | 11 hours ago
fransje26 | 14 hours ago
mejutoco | 13 hours ago
https://en.wikipedia.org/wiki/A_Void
amelius | 10 hours ago
I asked it to translate your sentence to English and it did fine. In less than a fraction of a second.
fph | 10 hours ago
big-chungus4 | 17 hours ago