Every local and reasonable model is bad at that. It’s OpenAI’s own benchmark, so take it with a grain of salt, but “GPT-6 Astra achieved 100.0% accuracy on contexts from 256K to 512K tokens and 96.3% accuracy from 512K to 1M tokens on OpenAI's 8-needle long-context benchmark”.
Depends on how unique the needles are. If they're fairly unique then even Qwen 3.5 27B @ Q5 with 128k context and Q4 KV cache has perfect recall of at least up to 5 needles, according to my tests.
But I'm sure if you make them less unique they'll start to struggle.
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
I spent very little $ having Claude just download and benchmark everything that would fit on my workstation. I’ve settled on qwen3.8 flash next nvfp4 (I have an Rtx 6000 Blackwell and 256gb ram). The same approach would work just as well for smaller setups. It took about a week, just working by itself and occasionally pinging me on the phone UI.
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning.
It’s fun, but it’s just so tough to see the value when the same research report via ChatGPT is done in 5 minutes and likely higher quality, for less than a dollar a day.
FWIW depending on what exactly you want to review something smaller like Gemma 4 E4B would be much faster. On my 7yo 3700X, llama.cpp runs Gemma 4 E4B UD-Q4_K_XL with a 131k context (context size doesn't seem to make much of a difference in speed) at ~10-15 tps (using MTP draft).
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
I kind of like the slow process, it forces me to be deliberate and not lean on the tooling as a crutch. I don't write that much, so having to wait a day is kind of like having your final work ready to review by a human editor.
Plus, using a bigger model makes me feel a little better that I am not taking a hit on quality when I do not even care about the turnaround time.
This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
When the XGX Spark is the equivalent of 1/10th of the pre tax annual salary of a top 5-8% earner, its far too expensive. Not sure if its cheaper comparatively in the US, but its a nightmare in Australia unless you are on an obscene salary. Throw in an approximate 30% tax rate and high cost of living - you've got to strongly wonder if it's worth it.
A DGX Spark is currently less than 10% of the median US household income (~80,000 USD/year), but that's still a big expense for the middle class. Until July or so, it was under 10% of the median personal income (~47,000 USD/year).
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
Yes! One method is KV cache compression to 4-bits - llama.cpp does have hadamard, but it's not very accurate - we were researching if 4-bit KV cache can be used with a sliding window of 1024 for example of 8-bit KV cache
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)
- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio.
IMO Byteshape is kind of the winner here if you're only looking at 24GB and down targets.
Continually compressing the model down with more computationally complex quantization schemes crosses from memory bandwidth being the limit into compute is the limit, Byteshape tries to keep the models performant while also using techniques appropriately.
Like, for me, their biggest version saves enough space that I can actually fit the dflash2 kernel without digging into context, yet score virtually identical to Unsloth's 3.0 Q4_K_M.
Thats like going from, say, 15-20 tok/s with MTP to 50+ with DFlash2.
If you're looking to run something from the Qwen family, you should jump right to Splash. It's faster than mlx or llama. You can run the unsloth gguf's.
Hi! Unsloth Studio is great, and I was wondering where exactly you are with multi-GPU and multi-node training and fine-tuning? I have been checking your documentation and GitHub issues from time to time and there doesn't seem to be as much development as I initially anticipated. Is making this aspect as user friendly as other parts of Studio and the docs on your near term roadmap or are your prioritizing other areas for now?
Oh we have not announced it, but in the past month, Unsloth has FSDP2 support - full finetuning, LoRA and QLoRA all work! We also support pipeline parallel, tensor parallel, and more!
In order to run the model, I need to install the Unsloth desktop, on Ubuntu. The link at https://unsloth.ai/download/linux says it should be 44.3MB, but the download itself is 23M.
Is this an oversight on the download page or am I getting the wrong package?
Oh it's correct - that's the installer size - but we will need to download binaries from https://github.com/unslothai/llama.cpp and also PyTorch etc which will take 2-3GB more
Just downloaded it because I finally have time to work through Sam Witteveen's Qwen 3 variants video; it looks like your new dynamic quants mean that I perhaps only additionally need ThinkingCap's variant for the few occasions I use xhigh.
Is this new technique limited to Qwen-type architectures, or can e.g. Muse Glimmer be updated this way?
P.S. the literal only thing that occasionally has me going back to LM Studio instead of Unsloth Studio: slightly better PDF export. Is there a way to export PDFs with the thinking hidden?
P.P.S. thank you for all of this. While I have posted about it a fair bit here, I can still hardly begin to explain how much more confident I have been with AI from having the ability to tinker with things like Unsloth Studio. It has let me get from total cynicism to a kind of cynical pragmatism through a process of exploration.
Great work as always Dan! I've used Qwen3.8 quite a bit and it's very verbose - its propensity for tool calling is insane and it uses 2x as many tokens as GLM-5.2 (which itself uses huge amount of tokens). I'm wondering if you have any plans to incorporate Bottlecap AI Qwen3.8 [1] checkpoints? They have done post-training where they significantly improve the token efficiency.
Oh yes I think I saw bottecap's work - very interesting!
For Qwen3.8-Flash-Next (I know not 27B haha), we actually cut reasoning by up to 50% esp for the small quants - we found quantizing models naively coincidentally also made them more verbose, as it would "correct" itself during the reasoning process. We managed to make the extra token usage of lower bit quants now only be +5% vs BF16 vs +50% extra from other quants.
27B should have similar benefits, but it's less as it's a smaller model!
Interesting, thanks for the info! Incidentally we've been testing nvfp4 weights vs vanilla 16-bit, GLM-5.3 in this case, and found that the vanilla 16 bit ones use 10x more tokens, but the output is more stable and better quality (less Chinese characters appearing, better coding solutions). So to me it's all just alchemy at this point.
Yes as requested by the community :) We also run all benchmarks multiple times, and provide error bars! For eg Terminal Bench and DeepSWE are run 3x, and LiveCodeBench 6x or more, and other faster benchmarks are run 12x
I could not find one that's why I built my own. It only covers python and C++ but at least it is not benchmaxed. I specifically tried to focus on models that are runnable in consumer GPUs https://sevangelatos.github.io/spiruxbench-results/
I see you already have average tokens per task but one aspect of model comparisons I find helpful to see is tokens per second or a general sense of time till completion.
Obviously a MOE is going to be faster than a similarly sized dense model, but getting a wrong answer with a smaller / faster model in a third of the time can allow for a second try, sometimes before the smarter / slower model can finish its first response. I feel like that is a useful metric to compare models on.
I will add an average time per task collumn when I get the chance. I feel tokens per second is a very fluid metric though as it varies wildly by hardware, inference engine and configuration as well as MTP config. If you can fit these models in VRAM (aka you have at least 24gigs of VRAM) you should be able to get at least 50/60 tps from the slower dense models and easily more than 100 tps from the MOE models. But it depends on lots of factors.
In my home grown (not grass fed) benchmark, the V3 and V3.1 Qwen3.8-27B-UD-Q2_K_XL have roughly the same capability with the new one being a bit worse consistently in debugging tasks but still having some edge overall. I tried this config with Q8 KV cache as it fully fits in a 16Gig graphics card. The new quant is also significantly less token hungry with 46.5k vs 61.7k tokens/task.
If you look through the ollama codebase, it has a bunch of helpers for just pointing claude code, codex, etc at the ollama api. You can just use ollama and it's helpers or just write your own wrapper to make it work however you want.
I assume you really just want claude code, Qwen 3.8 is practically Claude Opus 4.8-ish out of the box.
Well, I want Claude Code, Opus 5.5 specifically, but locally. I assume that if you give e.g. Qwen a certain set of instructions (skills, etc.) or complete harness, you can make it behave more like Claude. And I assume there are people who are trying to do this. So that's what I'm asking.
Is it possible to use these Qwen3.8-Flash-Next quants as a coding variant, with half the experts removed, like the ISTA-DASLAB Coder version? The performance and quality of that model is quite good when run with Splash and DFlash. But I wonder if there’s room left to improve, as the Coder model is I_Q3S variant, so it seems like doing 5 or 6-bit quants with half of experts removed might be even better. (And using the Dynamic v3.1 quantization)
Just tested Q4_K_M. About 100 MB larger than the earlier version. Quality appears same, for an identical coding task, with a run of about 10 minutes. YMMV.
jan_Sate | 21 hours ago
danielhanchen | 12 hours ago
ilaksh | 21 hours ago
DiabloD3 | 20 hours ago
Bonsai's technology would be better suited for 100s of B models.
ycui7 | 17 hours ago
sroussey | 16 hours ago
alex43578 | 15 hours ago
magicalhippo | 15 hours ago
But I'm sure if you make them less unique they'll start to struggle.
jedbrooke | 18 hours ago
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
lowbloodsugar | 3 hours ago
nout | 20 hours ago
aliljet | 19 hours ago
3eb7988a1663 | 18 hours ago
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
jedbrooke | 18 hours ago
alex43578 | 15 hours ago
badsectoracula | 16 hours ago
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
3eb7988a1663 | 8 hours ago
Plus, using a bigger model makes me feel a little better that I am not taking a hit on quality when I do not even care about the turnaround time.
Incipient | 17 hours ago
stkdump | 17 hours ago
otabdeveloper4 | 16 hours ago
NooneAtAll5 | 16 hours ago
girvo | 16 hours ago
anakaine | 12 hours ago
entrope | 10 hours ago
girvo | 3 hours ago
LeBit | 14 hours ago
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
badsectoracula | 14 hours ago
LeBit | 10 hours ago
God damn it!
ReptileMan | 14 hours ago
What is the price of not having your ssh keys uploaded to the NSA?
danielhanchen | 16 hours ago
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
dev_dan_2 | 14 hours ago
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
danielhanchen | 14 hours ago
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
testtestmetest | 10 hours ago
hacker_88 | 19 hours ago
aliljet | 18 hours ago
quaddoggy | 18 hours ago
dajonker | 17 hours ago
DiabloD3 | 13 hours ago
Continually compressing the model down with more computationally complex quantization schemes crosses from memory bandwidth being the limit into compute is the limit, Byteshape tries to keep the models performant while also using techniques appropriately.
Like, for me, their biggest version saves enough space that I can actually fit the dflash2 kernel without digging into context, yet score virtually identical to Unsloth's 3.0 Q4_K_M.
Thats like going from, say, 15-20 tok/s with MTP to 50+ with DFlash2.
zmmmmm | 12 hours ago
sroussey | 16 hours ago
Less important parameters should get quantized more, seems odd that we would quantize every parameter the same. At least for storage.
danielhanchen | 16 hours ago
We also updated https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF with UD3.1 quants
And also https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF as well!
We provide our benchmarking methods like what graders we use, infra fixes and issues and more in https://unsloth.ai/docs/basics/dynamic-3.0-ggufs/dynamic-3.1...
rdslw | 15 hours ago
what would you recommend re: software stack for best pp/tg overall performance?
danielhanchen | 14 hours ago
rdslw | 8 hours ago
regexorcist | 7 hours ago
logicalmind | 5 hours ago
https://github.com/incoai/splash
dr_kiszonka | 14 hours ago
danielhanchen | 14 hours ago
See https://github.com/unslothai/notebooks/blob/main/fsdp2_scrip...
mirod1 | 14 hours ago
danielhanchen | 14 hours ago
dofm | 9 hours ago
Is this new technique limited to Qwen-type architectures, or can e.g. Muse Glimmer be updated this way?
P.S. the literal only thing that occasionally has me going back to LM Studio instead of Unsloth Studio: slightly better PDF export. Is there a way to export PDFs with the thinking hidden?
P.P.S. thank you for all of this. While I have posted about it a fair bit here, I can still hardly begin to explain how much more confident I have been with AI from having the ability to tinker with things like Unsloth Studio. It has let me get from total cynicism to a kind of cynical pragmatism through a process of exploration.
doctorpangloss | 6 hours ago
puzzlingcaptcha | 14 hours ago
armcat | 14 hours ago
[1] https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-G...
danielhanchen | 14 hours ago
For Qwen3.8-Flash-Next (I know not 27B haha), we actually cut reasoning by up to 50% esp for the small quants - we found quantizing models naively coincidentally also made them more verbose, as it would "correct" itself during the reasoning process. We managed to make the extra token usage of lower bit quants now only be +5% vs BF16 vs +50% extra from other quants.
27B should have similar benefits, but it's less as it's a smaller model!
armcat | 13 hours ago
danielhanchen | 12 hours ago
stared | 14 hours ago
danielhanchen | 14 hours ago
serf | 13 hours ago
Q8 stomps Q6 on personal benchmarks here for me wrt Qwen 3.8 27b.
nkmnz | 13 hours ago
nunodonato | 11 hours ago
whatifitoldyou | 9 hours ago
plipt | 6 hours ago
I see you already have average tokens per task but one aspect of model comparisons I find helpful to see is tokens per second or a general sense of time till completion.
Obviously a MOE is going to be faster than a similarly sized dense model, but getting a wrong answer with a smaller / faster model in a third of the time can allow for a second try, sometimes before the smarter / slower model can finish its first response. I feel like that is a useful metric to compare models on.
whatifitoldyou | 5 hours ago
whatifitoldyou | 9 hours ago
So kudos to unsloth. The difference is measurable. Check it out: https://sevangelatos.github.io/spiruxbench-results/
xrd | 9 hours ago
And, reading through the ggml update on the front page of HN this morning, I realize I might do better than llama.cpp.
I've also really fascinated by llama-modchip and running these models in RAM (https://github.com/synchopate/llama-modchip.git)
Is there a place to find the best way to optimize my hardware setup?
I know about this:
https://github.com/noonghunna/club-3090
Are there others? Are there better nooks to look?
I assume unsloth studio (since it uses llama.cpp behind the scenes) is not going to do better than HEAD of llama.cpp from github, but am I wrong?
strix_varius | 7 hours ago
What do you mean by this? (from another llama.cpp user)
xrd | 7 hours ago
https://news.ycombinator.com/item?id=50494982
amelius | 9 hours ago
SamInTheShell | 8 hours ago
I assume you really just want claude code, Qwen 3.8 is practically Claude Opus 4.8-ish out of the box.
amelius | 7 hours ago
exe34 | 5 hours ago
jeffyaw | 8 hours ago
WhyNotHugo | 2 hours ago
Even with non-hosted models, for me, this consistently yields better results than Claude while being noticeably lighter.
yearolinuxdsktp | 8 hours ago
zkmon | 8 hours ago