Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing sessions to individual/multiple cards, occasionally waiting for what it thinks is best to become available.
I couldn't believe it at first, it shit out a damn fine multithreaded physics simulation fabric (integrated into a massive codebase, tests passing) in under an hour. Anything I could find online says they average like 80 but my logs average ~3x that.
>nobody is going to be doing 17 benchmark passes on a $50-100 task
Hopefully we'll see more of this as big companies try to optimize token usage where the cost of benchmarking is dwarfed by the potential savings across the org
Do you have instructions on how to run a custom harness against this? There are none on the linked page. I want to run Dirac (https://github.com/dirac-run/dirac).
Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
Afaik the reverse is also true. Some models are trained for specific harness setups so work better/more efficiently in that harness. If you give a model trained on a Read/Write/Bash tool setup a harness with only Bash where it needs to generate shell commands, I'd expect it to perform differently.
Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls
There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)
Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).
Claude Code allows you to change the config such that you use the harness with different models - you can actually ask a coding agent to configure it for you
Claude Code supports the base URL env var so you could tell it to talk with any LLM API endpoint that receives the Anthropic style request format, e.g. DeepSeek.
I use routatic/proxy (actually my own fork with various changes/fixes http://github.com/nijave/proxy/ -- fyi the fork is all vibes...)
Some gateways/proxies/hosts also offer multiple endpoints (z.ai has both Anthropic endpoint and Completions endpoint)
Edit: Here's the PoC I made for work although it assumes you're using Cloudflare Access (VPN) with a Cloudflare AI Gateway to hand wave the auth away (auth to the LLM API gets auto injected by Cloudflare Access derived from the client's VPN connection) https://gist.github.com/nijave/6f5e5793a992619808135ce85fa49...
Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.
It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.
I also feel Copilot is not getting the attention it should considering its user base and its importance in the enterprise context. There is a lot to be learned and I found https://news.ycombinator.com/item?id=49256057 (mitm copilot investigation) interesting. Playing with Copilot and asking it what was the biggest part of its standard payload I was surprised to learn that VSCode extensions plug in their MCP servers without me being much aware of it.
There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.
I am looking forward to your findings on home-field advantage. It's natural that Claude Code will work better with Anthropic models. If we know by how much, it will inform many of us.
Interesting to see harness benchmarks for coding agents. The same problem exists for conversational and data agents but I dont see anyone benchmarking them yet. Seems like manual-spot checks are the norm
I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.
Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems.
I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.
It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it
I'm not sure i agree with these. I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in '/goal' mode.
Prompt -
"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."
OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.
RadixIO | 12 hours ago
ADD-SP | 12 hours ago
monster_truck | 12 hours ago
Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing sessions to individual/multiple cards, occasionally waiting for what it thinks is best to become available.
I couldn't believe it at first, it shit out a damn fine multithreaded physics simulation fabric (integrated into a massive codebase, tests passing) in under an hour. Anything I could find online says they average like 80 but my logs average ~3x that.
try-working | 7 hours ago
nijave | 7 hours ago
nijave | 7 hours ago
Hopefully we'll see more of this as big companies try to optimize token usage where the cost of benchmarking is dwarfed by the potential savings across the org
GodelNumbering | 11 hours ago
fmxexpress | 11 hours ago
_matthew_ | 10 hours ago
molticrystal | 8 hours ago
Blog Post: https://runta.com/blog/introducing-frontierharness-eval/
The tasks and methodology: https://github.com/runta-dev/frontier-harness-eval
vidarh | 11 hours ago
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
nijave | 7 hours ago
Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls
There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)
Edward40 | 7 hours ago
jonstewart | 11 hours ago
tokencanopy | 11 hours ago
EFLKumo | 11 hours ago
verdverm | 10 hours ago
https://code.claude.com/docs/en/env-vars#variables
krat0sprakhar | 8 hours ago
nijave | 6 hours ago
Some gateways/proxies/hosts also offer multiple endpoints (z.ai has both Anthropic endpoint and Completions endpoint)
Edit: Here's the PoC I made for work although it assumes you're using Cloudflare Access (VPN) with a Cloudflare AI Gateway to hand wave the auth away (auth to the LLM API gets auto injected by Cloudflare Access derived from the client's VPN connection) https://gist.github.com/nijave/6f5e5793a992619808135ce85fa49...
Edward40 | 11 hours ago
EFLKumo | 11 hours ago
tamimio | 11 hours ago
But wow, opencode is that bad?!
kakugawa | 10 hours ago
1/ https://antigravity.google/product/antigravity-cli
verdverm | 10 hours ago
xnx | 10 hours ago
Edward40 | 9 hours ago
kaishin | 10 hours ago
TheJCDenton | 10 hours ago
fhn | 9 hours ago
netcyrax | 9 hours ago
grigio | 9 hours ago
nsingh2 | 9 hours ago
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
dfltr | 9 hours ago
It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
jdthedisciple | 7 hours ago
With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".
infecto | 7 hours ago
randomblock1 | 9 hours ago
infecto | 7 hours ago
Edward40 | 5 hours ago
knombertus | 9 hours ago
Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
heisenbit | 8 hours ago
yorwba | 9 hours ago
markbao | 9 hours ago
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!
Edward40 | 6 hours ago
billylo | 2 hours ago
travishcronin | 8 hours ago
joshheitzman | 7 hours ago
Edward40 | 7 hours ago
joshheitzman | 6 hours ago
I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.
nijave | 7 hours ago
Edward40 | 5 hours ago
fn-mote | 3 hours ago
benw214 | 2 hours ago
Prompt -
"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."
You can see the outputs here - https://alvins82.github.io/hangar-harness-model-tests/ https://github.com/alvins82/hangar-harness-model-tests/
OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.