Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
I run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.
7900 XT is a sleeper card. When I initially bought it, it was priced at the lowest wattage per $ per GB VRAM (not normalized for token speeds...) Although I ended up swapping for the XTX because that 4GB means everything in just increasing the context window. At 8bit KV my window is over 200k, and although qwen3.8 loves vomiting out tokens as part of its reasoning chain I trust it enough to get assigned tasks done eventually, which I could not say of any model before its release.
How has software/driver support been? I got burned hard by AMD last generation or the one before. Things smoother now, or do you have to baby it like hell and pick and choose software that works?
I don't do anything fancier than inference, and I only use llama.cpp, which supports rOCM. I've had few issues; most GGUFs I download work right out of the box. Nearly any popular model has a quant that just works. But as you can see I don't use my GPU for anything weird or nonstandard.
I'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.
I run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/
"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."
150 tokens per second on a ternary model implies that it’s GPU bound, I’d bet a Q6 model is even faster because it’s existed longer and seen more optimization. You’d have to be insane to not run an NVFP4 quant over a ternary quant on Blackwell if they both fit.
I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.
cd /tmp
# Get the Prism macOS runtime
curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
tar -xzf bonsai-runtime.tar.gz
# Get the ~5.95 GB GGUF model:
curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf
# Run the server, I used port 8331
./llama-prism-b10685-7dffb15/llama-server \
-m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
--port 8331 -ngl 99 -fa on -c 32768
Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:
uvx llm openai endpoint http://127.0.0.1:8331/v1 \
--model bonsai-2-27b --responses hi
That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".
Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:
export HF_TOKEN=xxx # optional, speeds up the download
./llama-prism-b10685-7dffb15/llama serve \
-hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
--port 8331 -ngl 99 -fa on -c 32768
Thanks for all of your exploration in public Simon.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
If we were talking about speed instead of size, i think it would be perfectly reasonable to say 9x faster. I'm not sure I agree that 9x smaller is unintuitive. It makes sense to me.
"Nine times" literally means multiplied by nine, but here we're dividing by nine. It's not unintelligible (because the corrupted verbiage is so commonplace) but it is needlessly awkward. Like saying "resulted in a size reduction increase of 10 megabytes."
Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
"faster" relates to speed. Speed is related to time and speed of a thing is usually defined by time. 9x faster speed translates to time/9. There is an extra step of related conversion there.
Conversely, 9x [filesize/natural number] is bigger. Every time. At least in the basic maths used by most people. There is no conversion into other units.
Therefore "9x smaller" when talking about a natural number like filesize is a nonsense statement in logic terms. If you strive for unambiguous phrasing - which is a significant part of the programming experience - this logical nonsense might well perturb you.
But english language is a flexible thing and if the phrase communicates your intent to your audience then that's fine by me.
I think more intuitively for speed - you are multiplying the things per period. If I’m making widgets 9x faster I have 9 x the previous number of units per period of time. To get the amount of time, you invert the ratio to period over units.
Multiplying some scale by units per period makes sense and is both linguistically and mathematically sound.
I agree it makes little sense in a literal mathematical take but "it's 9x smaller" or is too much of linguistic advantage compared to "the original is 9x larger" or "it's 1/9th as large" to expect a change with. You don't have to invoke fractions, it keeps the thing in focus as the first subject, it matches the pattern of the inverse statement, and it's just plain short... so that's what people will adapt and interpret the meaning to be.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then "smaller"/"larger" tells you which end of that separation the subject is (rather than specifying whether the multiplication builds up or down).
11% is also a really saying a fraction, 11 per-cent or 11/100, but there's nothing wrong with that feeling more natural to some and it is at least a nice shorthand way for the written form. "A ninth the size" is a similar alternative. All are really fine, there's always someone who has trouble with a given representation compared to another.
Eh, when I read smaller with an integer multiplier, I mentally switch to the reciprocal. Easier than convincing the world not to use "9x smaller". Do you feel the same way about "9x faster"? What you're actually measuring is time, and "faster" is the reciprocal of time, similarly to "smaller" being the reciprocal of size.
yes, actually I do think the phrase 'N times faster' is sensible and logical
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster.
i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
Ah, but in this case the customer is buying the coffee for the caffeine content. And you've doubled the caffeine content per volume! Twice as efficient a delivery mechanism, similar to this model!
They probably rephrased it from some more technical form like "we compressed the model by a factor of 9" or "we've improved the packing efficiency of the model by 9x". Where these are measurements of the transformation the model is undergoing, not measurements of the resulting model.
This sounds like the same kind of error as writing "0.10 cents" because it's less than a dollar when the number is in dollars regardless of how big or small it is
Came to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).
The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
Yep more hops from the lower Q is likely going to skew the vectors further over time.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
That's correct. I think there can certainly be issues even with 4bpw with naive quantization (IE you'll notice far better results from a QAT 4bpw vs a naive 4bpw).
One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
1.76 bpw number is kinda misleading if you compare it directly to IQ2/Q2. The encoding is ternary, but the quantization procedure is way more sophisticated than "round Qwen weights to {-1,0,+1}."
They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.
I wonder how well it performs in practice, because I can't help seriously doubting those benchmarks. That would put this 6GB model in Opus 4.6+ ballpark. Granted, that's mostly to Qwen 3.8's credit, but it's hard to believe that Qwen's already unbelievable capability density can still be compressed this much more.
(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.)
Cautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much
GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
That's a rough place to land on a spark. It seems unlikely to be memory bandwidth at this model size, but maybe just lack of tuned kernels? The chip is missing some CUDA features but with tuning you should be able to hit way more than that even without a drafter.
I was wondering if those ternary bits get expanded into full floats internally in the kernels - you’re probably right about lack of tuned kernels. I’m not sure you could just tune your way out of that easily though. Any suggestions on trying particular solutions?
I wonder how their talks with Apple went. Having this run on the TPU opposed to just the GPU, which drains a significant amount of battery life by comparison, is what I'm really interested in.
Without context, it should be number of params * 1.76 (the “effective bits per weight”) / 8
So for this one, 27B * 1.76 / 8 = 5.94 GB
For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:
300GB/s / 5.98 GB = 50.5t/s.
Realistically it’s probably a bit slower, but that is your theoretical maximum.
What is never totally clear with a lot of these releases is the scope of what it's good at. Models that can run with good speed on affordable consumer hardware for coding only is the dream. I am never going to use this for writing, images, or "general knowledge". Coding only
I tried their WebGPU version and it immediately started looping. Yeah "near lossless" my ass. Plus the reasoning that it looped on was clearly wrong and unlike the non quantized 27B
LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.
For example, the Hadamard activation transform used here feels a lot like multiplying
Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.
The issue is I have yet to find a useful agentic local llm that I can run on this machine.
Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.
Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
I tried this one, but I found Ornith1.5 to be a better MoE model for me, also very fast. But I still couldn't make it implement a real task on a real repo :( it only worked with an extremely clear directions and very small tasks.
There's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release!) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if you are GPU starved and want local, but you might have trouble finding things it is good at.
kamranjon | 11 hours ago
kadoban | 11 hours ago
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
spijdar | 11 hours ago
orsorna | 8 hours ago
kadoban | 5 hours ago
orsorna | 5 hours ago
lta | 7 hours ago
redox99 | 6 hours ago
djkoolaide | 7 hours ago
Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
kennywinker | 3 hours ago
abraxas | 11 hours ago
kamranjon | 11 hours ago
pizza234 | 11 hours ago
sisve | 10 hours ago
And speed matters a lot for many use cases
selectodude | 10 hours ago
wincy | 4 hours ago
pwython | 4 hours ago
blurbleblurble | 4 hours ago
azatom | 10 hours ago
_kulang | 10 hours ago
azatom | 9 hours ago
https://xkcd.com/3038/
Havoc | 11 hours ago
Aurornis | 11 hours ago
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
outofpaper | 11 hours ago
SXX | 8 hours ago
trvz | 8 hours ago
SXX | 8 hours ago
14u2c | 7 hours ago
z2 | 11 hours ago
all2 | 9 hours ago
codebje | 3 hours ago
[OP] JonSchneider | 10 hours ago
sroussey | 10 hours ago
verdverm | 9 hours ago
simonw | 10 hours ago
This should work:
Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this: That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".simonw | 10 hours ago
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
kadoban | 10 hours ago
Forgeties79 | 10 hours ago
bigwheels | 9 hours ago
tomcam | 6 hours ago
kadoban | 3 hours ago
rahimnathwani | 5 hours ago
raylad | 30 minutes ago
For my "Please recite Jabberwocky" test the bf16 almost passes but the ternary and even fp8 versions fail badly.
refibrillator | 10 hours ago
They have a demo repo with a setup.sh script:
https://github.com/PrismML-Eng/Bonsai-demo
The release tag and weight file you suggest doesn’t match what they wrote.
simonw | 10 hours ago
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
refibrillator | 9 hours ago
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
nikwen | 9 hours ago
iJohnDoe | 7 hours ago
Zetaphor | 6 hours ago
hedgehog | 4 hours ago
rahimnathwani | 8 hours ago
francisjp | 7 hours ago
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
jb_briant | 22 minutes ago
miffy900 | 10 hours ago
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
hamandcheese | 10 hours ago
simondotau | 9 hours ago
D-Machine | 3 hours ago
Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
_carbyau_ | 9 hours ago
Conversely, 9x [filesize/natural number] is bigger. Every time. At least in the basic maths used by most people. There is no conversion into other units.
Therefore "9x smaller" when talking about a natural number like filesize is a nonsense statement in logic terms. If you strive for unambiguous phrasing - which is a significant part of the programming experience - this logical nonsense might well perturb you.
But english language is a flexible thing and if the phrase communicates your intent to your audience then that's fine by me.
jonhohle | 4 hours ago
Multiplying some scale by units per period makes sense and is both linguistically and mathematically sound.
UI_at_80x24 | 10 hours ago
Kinrany | 5 hours ago
zamadatix | 9 hours ago
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then "smaller"/"larger" tells you which end of that separation the subject is (rather than specifying whether the multiplication builds up or down).
kevinwang | 6 hours ago
zamadatix | 6 hours ago
rpdillon | 8 hours ago
miffy900 | 7 hours ago
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster. i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
rpdillon | 6 hours ago
comradesmith | 6 hours ago
foobarbecue | 6 hours ago
dofm | 3 hours ago
peey | 6 hours ago
If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9
It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number
derefr | 5 hours ago
Kinrany | 5 hours ago
nicbor | 5 hours ago
I don't see what makes it hard to understand.
stkdump | 35 minutes ago
2001zhaozhao | 10 hours ago
kennywinker | 3 hours ago
Best you could realistically run in 32gb is like 28gb, or a 127B param model
jokethrowaway | 22 minutes ago
danbrooks | 10 hours ago
0xbadcafebee | 9 hours ago
nulld3v | 8 hours ago
kadoban | 6 hours ago
anana_ | 6 hours ago
SillyUsername | 2 hours ago
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
WithinReason | 2 hours ago
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
The 3-bit quant is lossless based on benchmarks.
adrian17 | 10 hours ago
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
https://news.ycombinator.com/item?id=49611128
edflsafoiewq | 10 hours ago
yowlingcat | 9 hours ago
One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
https://huggingface.co/AngelSlim/Hy4-preview-GGUF https://arxiv.org/abs/2602.21233
Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
om8 | 8 hours ago
edflsafoiewq | 8 hours ago
0x457 | 10 hours ago
They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.
nulld3v | 9 hours ago
The table claims it performs on par with UD-Q4_K_XL except on OCR.
Balinares | 16 minutes ago
logicallee | 10 hours ago
[1] https://news.ycombinator.com/item?id=49732931
Havoc | 9 hours ago
jedbrooke | 9 hours ago
So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
flutetornado | 9 hours ago
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
cmrdporcupine | 9 hours ago
flutetornado | 8 hours ago
mmastrac | 6 hours ago
flutetornado | an hour ago
circularfoyers | 9 hours ago
cmrdporcupine | 9 hours ago
That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
hedora | 8 hours ago
Also, perf speedup?
jjcm | 8 hours ago
kennywinker | 3 hours ago
So for this one, 27B * 1.76 / 8 = 5.94 GB
For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:
300GB/s / 5.98 GB = 50.5t/s.
Realistically it’s probably a bit slower, but that is your theoretical maximum.
g023 | 8 hours ago
Fordec | 8 hours ago
ncr100 | 5 hours ago
blactuary | 7 hours ago
redox99 | 7 hours ago
respectattentio | 6 hours ago
Yet, seems like there is still another year for improvements.
I like local models (but not mainly using them) for offline needs.
avaer | 6 hours ago
For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
[1] https://arxiv.org/abs/2507.13575
Dwedit | 5 hours ago
huseyinkeles | 5 hours ago
~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.
The issue is I have yet to find a useful agentic local llm that I can run on this machine.
Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.
aetherspawn | 4 hours ago
huseyinkeles | 4 hours ago
`cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh`
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
piyh | 4 hours ago
sean_pedersen | 4 hours ago
huseyinkeles | 4 hours ago
zhiyan | 4 hours ago
nilsherzig | an hour ago
PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)
https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d209...
verytrivial | an hour ago
mpweiher | 44 minutes ago
If true, that would be a very welcome development.
hvhvubufyvycjcx | 18 minutes ago
hvhvubufyvycjcx | 12 minutes ago