Bonsai 27B: A 27B-Class Model that runs on a phone

148 points by xenova 2 hours ago on hackernews | 41 comments

TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1

Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.

NitpickLawyer | an hour ago

There's two variants of this (or, as the joke goes, for very big values of bit):

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.

1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

PcChip | 51 minutes ago

this is a really dumb question, but how is -1 represented?

is it a float? if so, how many bits is the float?

I've never heard of a bit ever having more than two possible values

: zawaideh | 36 minutes ago
It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
: petu | 33 minutes ago
packing multiple trits together
e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing

liuliu | 2 hours ago

The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.

: liuliu | 2 hours ago
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.

Havoc | an hour ago

This must be some sort of unpublished app?

I can just see their image tool on the app store

: Catloafdev | an hour ago
It's a LLM model, not a phone app.
Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b

simonw | an hour ago

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models

I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

: trollbridge | 48 minutes ago
Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.
: bansaltushar | 42 minutes ago
Depending on which model you're running, you might need to use the custom forks.
Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

xyzsparetimexyz | an hour ago

That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?

: Catloafdev | an hour ago
Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.

syntaxing | an hour ago

For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.

erelong | an hour ago

I was trying Ornith 9B locally (it's up on Ollama) which claims:

> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.

https://deep-reinforce.com/ornith_1_0.html

Only tried it so much so far; it did a little better than Qwen 9B

liuliu | an hour ago

Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).

: gunalx | 27 minutes ago
3.5 9B can do thinking. Its just disabled by default in its gguf chat template.

janalsncm | an hour ago

Is that a 1-bit LLM? I don’t understand the connection with this article.

: erelong | 55 minutes ago
Oh, I don't actually know the difference if you want to explain it
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol

verdverm | 24 minutes ago

Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals

syntaxing | an hour ago

I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.

: pulse7 | an hour ago
Most probably not optimized yet for this model...

sigbottle | an hour ago

What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?

trollbridge | 49 minutes ago

If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.

: doctoboggan | 22 minutes ago
Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.

thomasjb | an hour ago

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

: dakolli | 21 minutes ago
start saving your money.

kristianp | an hour ago

Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...

luckystarr | an hour ago

Tried it on Android and got "!!!!!!!!!!!!!" for answers.

: gunalx | 30 minutes ago
The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
: verdverm | 29 minutes ago
That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.
When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.

erwan577 | an hour ago

The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.

I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.

: verdverm | 27 minutes ago
quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.

wy35 | 19 minutes ago

Entire blog post seems to be AI-generated :/

: wmf | 8 minutes ago
Do you think people who work on AI for a living are not going to use it?

theLiminator | 14 minutes ago

This is useful research, but this particular model itself is likely absolutely useless.

: oceansweep | 12 minutes ago
Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.

Arcuru | 6 minutes ago

Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds.

[1] https://jackson.dev/post/dont-sleep-on-bitnet/

comandillos | 6 minutes ago

Quite weird that heavy quantization methods on a huge dense model gives better results than slightly quantized MoE models like 35B-A3B from Google.

At this point all the different quantization and 'compression' (look at MPO applied to LLMs...) techniques start feeling a bit like snake oil. It's just a gut feeling - or scores on benchmarks models are optimized for - what ends up deciding whether a technique is good enough or not.