Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

29 points by anemll 4 hours ago on hackernews | 10 comments

Anyone know how hot an iPhone gets running the Neural Engine flat out for 10 minutes?

: [OP] anemll | 4 hours ago
It's warm but not hot. On M6 it's about 50% less energy consumption in my tests vs GPU. I think iPhone should be similar
: GeekyBear | 3 hours ago
The neural engine is optimized for energy efficiency, and the GPU is optimized for performance.
You're going to get much better battery life and much less heat running on the neural engine.

Hi. Could anyone share any recommendations (hopefully with actual links) for App Store apps to try running such local models? I’m not really interested in side-loading since it currently seems (to me) more of an hassle than it’s worth.

App Store search is broken in a sense that there might be so many apps claiming to be local but once you read reviews, it’s just another scam where it’s it’s usually sending your query to remote server.

simonw | 3 hours ago

My current favorite is this one: https://apps.apple.com/us/app/locally-ai-by-lm-studio/id6741...

It's free, not annoying, and has a bunch of model options.

: layer8 | 3 hours ago
Non-app-store link: https://locallyai.app/
However this uses the GPU, not the Neural Engine like the submission.
: jdgoesmarching | 3 hours ago
Note that the model discussed in this article isn’t available in this app running on the phone. Still, it has some good local options like Gemma 4 and can link to the desktop LM Studio app which is cool.

woadwarrior01 | 3 hours ago

I'm the dev behind the Private LLM app, which was the first on-device LLM app on the App Store. Don't download it yet. I'm currently finalizing a brand new v2 rewrite which can run models that the copycat app (locally) cannot run, like Bonsai 2 27B, Gemma 4 12B etc.

https://x.com/private_llm/status/2101769928252473488

Retr0id | 3 hours ago

OP, asking for upvotes is against the HN guidelines: https://x.com/anemll/status/2111531629529456771

createful | 3 hours ago

Not Qwen, Bonsai. Although it is based on Qwen, and achieves similar scores, it is not the "full" Qwen.

Last I remember, when trying to run a model above 1-2 GB on iOS, it would refuse to do so, at least from Locally AI. iOS only allows apps to use about 1-2 GB of RAM before being killed. Though using their Neural APIs might bypass this.