Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
I'm trying to organize about 400k photos. What kind of questions are you asking it and what are you doing with that info? Applying tags/labels? Sorting into albums?
Basically just tags, but the questions branch based on the answers. Photo vs Graphic, Portrait vs Landscape, Person/People/No one, etc.
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
Not reliably - look at microsoft's guidance project that did some of this.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
> It makes an unbounded number of future predictions
nit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.
The thing I had to wrap my head around is: For a "classic" classifier, the classes are fixed and the data distribution is p(x,y). The calibration is checked over this single distribution. Jev makes the tasks part of the input, so the data distribution is p(task,x,y) and calibration is averaged over that joint distribution. What follows is that while for classic classifiers, calibration brakes down if your x was out-of-distribution, it can now also break if your task is out of distribution. The big question now is: Can subtle differences in phrasing your task already lead to this? And the only way to verify calibration is having ground truth data to validate on.
Also: Calibration over p(task, x, y) means calibrated on average across tasks. Even an in-distribution task can be over- or underconfident while the average looks fine.
> And the only way to verify calibration is having ground truth data to validate on
I’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
A model can be consistently over- or under-confident for similar inputs. The only way to check if it's actually calibrated is to compare against some known ground truth. I don't see any way around that.
In my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.
Wouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels.
Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution
Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
Unless I am misunderstanding you then I don't think this is correct. Chain of thought was first proposed in 2022. Assuming you mean reasoning models then the correct comparison is GPT-o1, which came out in September 2024, 2 months prior to R1 lite preview. In the next 2 months, prior to the release of the full R1 model, both qwen and gemini released reasoning models. Dario explicitly notes in early 2025 that most major labs had been working on reasoning models in 2024, and explicitly credits OAI for pulling things forward with their release.
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
I wasn't suggesting they invented it. I was suggesting that they popularized it, made a huge amount of noise about it, and it seemed like after they did, there was a lot more attention on it.
I think if you were paying attention at the time, CoT and ReAct loops were already "well known". Lots of GPT wrappers at the time were doing it, I remember doing CoT to help measure the calories for a recipe. The problem is tokens then were quite expensive and doing experimentation on it was pretty expensive for an individual. I once accidentally spent $20 trying to categorize a Who's Hiring post from here lol.
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
From what I’ve gathered, it doesn’t work at all like a normal classifier. As in, for a classifier you’d train it for a specific task to reduce the cross entropy. So, in a hand-wavy way “the model knows the truth”. For Jev-type models it’s more like an LLM, in the sense that the “truth” is in the context you provide. But instead of producing a completion it fills in the schema. But it’s always - “given the context, the schema needs to be filled in like this”
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
OpenAI was already working on a Decisions API, and Jev launched theirs quickly enough to have 1 week in the spotlight before OpenAI put theirs in beta (which works great btw).
When trying it out, OpenAI's Decisions was ~2x slower, worse accuracy and more expensive. Unless used with image-related tasks, Jev is clearly superior.
That's how the original Transformer architecture (LaMDA) should have been released back in the day, but Google thought it would cannibalize their search revenue.
Dont under estimate the power of branding/positioning/marketing. For more than a year I have been using a RL'd Qwen fine tune optimised for just taking a decision and outputting the decision number, initially I didnt have much training data so after a few manual examples I asked deepseek to make them for me and had codex train it.
I compared it to jev when it first came out and it was mostly neck and neck for my use case.
I can't have been the only one in fact, I'm sure there must have been other people who probably even open sourced, but I couldn't find anything. I think the question going forward will be how does one keep abreast of interesting stuff happening, now that its de-coupled from direct human labour so de-bottlenecked on production capacity - just bound by how residual time/attention one has (Terence Tao was talking about this exact predicament in the latest tyler cowen podcast).
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
> Which makes me wonder: how do these models calculate probabilities?
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for most uses of decision models.
In some sense everything an LLM does is by unprincipled methods. But it works! We judge it empirically post hoc by its performance rather than relying on tuning.
Hell, human thinking is unprincipled in basically the exact same way.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
That also makes me wonder: what other things are achieved by now and are up for grabs, and that will seem obvious in retrospect, but we don't notice it yet?
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
Startup / Founder focus vs Large Organizations with no vision.
With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma4 26A4. That makes me optimistic for this release, despite the high cost.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
What got me into Claude Code started with Google Antigravity. Somewhere along the way Antigravity took a dive. The rollout was great, then it started feeling like other G-products, the dreaded tail of lackluster developer energy after a peak.
We have Antigravity at work. It's become a lot better, especially since the Gemini 3.8 Flash and 4 Argon versions. The harnesses themselves (GUI and CLI) are updated quite regularly.
I don't really miss Claude when I'm using it for coding tasks. Also, it doesn't speak Claudish, which is a plus.
That's always been the enterprise tech way. Using the product that comes with the platform the company is already signed up to pay for is easier than requesting a payment to a brand new service.
They don't need to be the best, they just need to not force a customer to go elsewhere.
I suppose the key question is come 5 years time is enterprise going to just become happy to use openrouter, or will they lose competitiveness/ market share to those who do.
Fyi artificialanalysis currently puts drex highest on price vs quality
I'm using them to label training data for model development. You can do something like label a bunch of records with Jev, then train the model, then check results with opus and a hold-out set.
Yes, literally built my own version of /goal in Claude/codex today for pi using jev. My agent was stopping so jev is called each turn to see if it’s done or not - similar to how it works in Claude, according to Claude…
I've found that they work great as gates ("does this require a human review?") or filters ("is this suspicious?") and occasionally as judges (based on this context, is the expected pain score for this user above 7.5/10?).
Interesting how do you know if it does a good job at these tasks? It seems that thinking in LLM's has been one of the things that has been so important to making sure it's a good response.
When the use case allows room for an acceptable amount of incorrect decisions because there's a manual flow those cases can be routed to still anyways.
It doesn't. Decision models are not more accurate than small models with no thinking. If your classification task is accuracy sensitive, you still need to stick with one of the big frontier models.
It can be used a fast/cheap router, if a response is above a specified probability then send it to LLM based workflow branch, otherwise nothing more to do
I work in manufacturing and would like to use LLMs for scenarios where there's obviously still rules for what should happen next, but there's just so many variables that I can't just code it into a decision tree easily.
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
2a) decision-tree for *who* to send it to for reworking things
3) scrap
As I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
Task classification - given a set of classes, return the most likely class for each task
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
We're about at the point where the AI is now pretty insanely cheap. Passing 100K tokens of context then asking hundreds of questions of 100 tokens each would cost about as much as a single Google search API call.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
When I reviewed our internal AI workflows i was shocked at how often a decision API was a good fit or could streamline the process.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
I still have no idea why this is useful. Models are atrocious without reasoning, for classification and otherwise, and this doesn't even just eschew reasoning afaik, but parts of the models as well for lower latency and cost. I genuinely do not see the point.
What differentiates this from training a classifier model? That it comes pretrained with general knowledge?
I think that is exactly the value prop, you don't need to train a classifier. You get something that works well enough to validate your product/feature/whatever before investing in training.
Opaque classifier as a service, for when you want to discriminate people by race and social status assumed from name, but want some other entity to be responsible for plausible deniability of the process.
Hey Jev, what mortgage rate multiplier should a person named James Smith get?
Audio, Video and File input? That is very appealing and I can imagine a few very unique ways to leverage this if it works well across all modalities.
For what it's worth, ran my quick and dirty evals with it and got in many usage scenarios a 4x lower brier over Jev (2x - 6x depending on task), which is an immense improvement and the first time a model so significantly has outperformed Jev in my experience. p50 was also over 100ms lower though all this cost twice as much across the entire eval end-to-end.
Dare I say, this is a very impressive release by the Gemini team. While testing GPT-6 Luna lead to an honestly very pleasant interaction with OpenAI staff on here, it did perform very uncompetitively so seeing a great first effort in basic text classification performance with broad multimodality, that is great to see. Course, wouldn't be surprised if Google had such a more broadly trained classifier for years as part of internal usage for platform features which they just now commercialised, but hey, I'll take it.
I wouldn't even trust this is the high level of their pricing with the way they just casually doubled Gemini Omni pricing in 1.2 while trying to spin that it was somehow "cheaper"
This appears the first decision AI that actually handles audio, video, and PDF, and I'm surprised this isn't highlighted everywhere.
Marketing seems to have dropped the ball on that. I doubt anyone's picking it for the +2% higher points, but being able to answer based on a PDF or call is groundbreaking.
Jev is still faster and cheaper for many of the common text uses.
You could instruct them to behave like one but they’ll still eat a lot of tokens, probably more, even though you’ll save a small amount on the output. It’s a separate thing because of significant token savings.
[OP] iamdelirium | 18 hours ago
dude250711 | 17 hours ago
waximabbax | 18 hours ago
However, it can accept Audio and Video as input as well and for those specific cases it can be extremely useful.
jermaustin1 | 15 hours ago
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
8n4vidtmkvmk | 8 hours ago
jermaustin1 | 57 minutes ago
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
florkbork | 17 hours ago
It seems unlikely this will be around in 6 months at this rate.
Catloafdev | 16 hours ago
DaSHacka | 16 hours ago
rgreasons | 15 hours ago
p_l | 15 hours ago
Google is maintaining the AI models and APIs for much longer, even if many people don't know about them unless they use them to build things
kasperset | 16 hours ago
antonvs | 14 hours ago
Google kills consumer products because most consumers don't pay for them directly, so they often end up not making enough money.
With enterprise products, they have actual paying customers, so the situation is different.
panarky | 15 hours ago
I say plant a thousand flowers and cull the ones that don't bloom.
IncreasePosts | 16 hours ago
[OP] iamdelirium | 16 hours ago
apetresc | 15 hours ago
antonvs | 14 hours ago
IncreasePosts | 13 hours ago
eminence32 | 16 hours ago
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
NitpickLawyer | 16 hours ago
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
ak681443 | 16 hours ago
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
dist-epoch | 15 hours ago
londons_explore | 15 hours ago
mediaman | 15 hours ago
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
NitpickLawyer | 7 hours ago
JamesSwift | 16 hours ago
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
janalsncm | 15 hours ago
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
computably | 15 hours ago
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
JacobAsmuth | 14 hours ago
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
computably | 5 hours ago
nit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.
sdl | 5 hours ago
IanCal | 3 hours ago
I’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
coderenegade | 3 hours ago
coderenegade | 3 hours ago
orbis-tertius | 4 hours ago
reexpressionist | 2 hours ago
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
larodi | 13 hours ago
icepush | 6 hours ago
JoshTriplett | 11 hours ago
Chain of thought was that way. DeepSeek came out, and not long thereafter everyone had adopted chain of thought.
ethanj8011 | 7 hours ago
https://darioamodei.com/post/on-deepseek-and-export-controls
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
Ohentis | 7 hours ago
LASR | 6 hours ago
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
JoshTriplett | 6 hours ago
Karrot_Kream | 3 hours ago
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
jgilias | an hour ago
ashdksnndck | 15 hours ago
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
deepsquirrelnet | 14 hours ago
Decision models are really not much different than that, but (usually) scaled up to bigger causal models.
If you are looking to do Jev-like things with smaller models, check into NLI.
nrightnour | 15 hours ago
levmiseri | 14 hours ago
sroussey | 14 hours ago
jgalt212 | 14 hours ago
Because it's especially parsimonious with token use.
RomanKornev | 13 hours ago
fny | 10 hours ago
redmalang | 8 hours ago
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
pknerd | 6 hours ago
mh- | 6 hours ago
You're leaving out the UX (DX, developer experience) in this.
And the better zero-shot performance when you don't have a dataset yet (could argue this falls under DX).
Not to mention the faster time-to-decision and order-of-magnitude cheaper costs.
visarga | 6 hours ago
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for most uses of decision models.
howunfortunate | 6 hours ago
Hell, human thinking is unprincipled in basically the exact same way.
nlpnerd | 6 hours ago
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
phillennium | 4 hours ago
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
kaycebasques | 16 hours ago
pratzc07 | 16 hours ago
brandall10 | 15 hours ago
freeone3000 | 15 hours ago
blitzar | an hour ago
bugattis
ai-x | 15 hours ago
mikeocool | 14 hours ago
Getting a response in 250ms opens up a lot possibilities over 750ms.
clhodapp | 12 hours ago
BoorishBears | 7 hours ago
No I think Jev will do just fine. Not because Google can't even ship this day 1 on their enterprise platform (https://docs.google.com/forms/d/e/1FAIpQLSeupHxao3ajUvl9Dt7l...) but has time for moralizing a decision API.
It's not the spirit of a competitive company.
abhikul0 | 16 hours ago
https://developers.google.com/edge/mediapipe/solutions/decis...
ilaksh | 14 hours ago
vesterde | 16 hours ago
claiir | 15 hours ago
theanonymousone | 15 hours ago
kingstnap | 15 hours ago
https://youtu.be/cFx9Z3ZXca0?t=3340
[OP] iamdelirium | 15 hours ago
ssivark | 8 hours ago
londons_explore | 15 hours ago
theanonymousone | 15 hours ago
holografix | 15 hours ago
Standard practice by the big G.
hbarka | 15 hours ago
antonvs | 14 hours ago
I don't really miss Claude when I'm using it for coding tasks. Also, it doesn't speak Claudish, which is a plus.
SchemaLoad | 14 hours ago
They don't need to be the best, they just need to not force a customer to go elsewhere.
twelvechairs | 11 hours ago
I suppose the key question is come 5 years time is enterprise going to just become happy to use openrouter, or will they lose competitiveness/ market share to those who do.
Fyi artificialanalysis currently puts drex highest on price vs quality
https://artificialanalysis.ai/evaluations/artificial-analysi...
wasabi991011 | 15 hours ago
keyle | 14 hours ago
SchemaLoad | 14 hours ago
muzani | 9 hours ago
5555watch | 15 hours ago
npc99 | 15 hours ago
kodoman | 15 hours ago
caseydm | 15 hours ago
bushbaba | 13 hours ago
b--l | 11 hours ago
te_chris | 14 hours ago
conception | 13 hours ago
cromka | 13 hours ago
LaurensBER | 13 hours ago
kodoman | 13 hours ago
hmcdona1 | 12 hours ago
roncesvalles | 12 hours ago
oblio | 10 hours ago
Which classification tasks aren't?
jiggawatts | 9 hours ago
addaon | 6 hours ago
blitzar | an hour ago
sarjann | 2 hours ago
campers | 2 hours ago
bushbaba | 13 hours ago
Evals. is this hitting the criteria or not? (Test case passes or fails)
billylo | 13 hours ago
https://chromestatus.com/feature/5155080092385280 https://evergreen-labs.org/products/credibility-analyzer
Topfi | 11 hours ago
sky2224 | 10 hours ago
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
3) scrapAs I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
gopalv | 10 hours ago
We use it for our critique looping within our own harness.
But decision models can do things like write SQL queries, if you're willing to put a little bit of trickery upfront.
My team's writing a paper about it for ICDE 2027 as we speak (as a follow-up to the DICE demo/paper from VLDB 2014).
But my Linkedin post[1] has a quick summary of the idea, without explaining everything we do there.
[1] - https://lnkd.in/p/gAsWcrCd
Mengkudulangsat | 10 hours ago
oblio | 10 hours ago
Mengkudulangsat | 9 hours ago
skapadia | 10 hours ago
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
lprd | 9 hours ago
richardw | 9 hours ago
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
tirumaraiselvan | 9 hours ago
benhurmarcel | 3 hours ago
nico | 14 hours ago
It comes with a bunch of pre trained classifiers, and you can use the tooling to train your own
There are also integrations with workflow tools like n8n
2001zhaozhao | 13 hours ago
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
LaurensBER | 13 hours ago
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
zeandcode | 11 hours ago
low_tech_punk | 13 hours ago
Apparently the two teams aren't talking to each other.
bagacrap | 9 hours ago
perching_aix | 13 hours ago
What differentiates this from training a classifier model? That it comes pretrained with general knowledge?
foopod | 11 hours ago
tomaskafka | 12 hours ago
Hey Jev, what mortgage rate multiplier should a person named James Smith get?
BoorishBears | 11 hours ago
Topfi | 11 hours ago
For what it's worth, ran my quick and dirty evals with it and got in many usage scenarios a 4x lower brier over Jev (2x - 6x depending on task), which is an immense improvement and the first time a model so significantly has outperformed Jev in my experience. p50 was also over 100ms lower though all this cost twice as much across the entire eval end-to-end.
Dare I say, this is a very impressive release by the Gemini team. While testing GPT-6 Luna lead to an honestly very pleasant interaction with OpenAI staff on here, it did perform very uncompetitively so seeing a great first effort in basic text classification performance with broad multimodality, that is great to see. Course, wouldn't be surprised if Google had such a more broadly trained classifier for years as part of internal usage for platform features which they just now commercialised, but hey, I'll take it.
flockonus | 8 hours ago
Wish they provided more comprehensive inputs to the model's capabilities and comparisons.
cube00 | 7 hours ago
v18a | 7 hours ago
jimauthors | 6 hours ago
0815beck | 4 hours ago
blitzar | an hour ago
muzani | an hour ago
Marketing seems to have dropped the ball on that. I doubt anyone's picking it for the +2% higher points, but being able to answer based on a PDF or call is groundbreaking.
Jev is still faster and cheaper for many of the common text uses.
brap | an hour ago
Google's marketing consistently drops the ball on every opportunity so that's not surprising at all
amelius | an hour ago
Why does this need to be a separate thing?
mattmaroon | an hour ago
remus | an hour ago