There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon.
At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo.
The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.
I'm sorry this makes it seem like I didn't do my research. I did a TON. To fix it I'll add a benchmark/comparison table. Also, I wouldn't call it market research since this is not commercial AT ALL.
I'm only joking anyway — it's more a comment on the whole AI-accelerated trend of everyone having their own version of a thing.
I do agree that, ultimately, combining your efforts with others working in this whole area is probably really worth it, but I can see how there's an ease of pushing forward on your own these days.
I do not have fast internet so I am not sure when I'll really be able to download the weights but I do have an M1 Max to try this on, so I will at some point!
I see! yes, downloading the weights part is painful. I tried a couple fixes and it is as fast as it can get downloading from HuggingFace. I think the field is heading toward smaller, more capable models soon, so you won't have to wait that long!
I mostly use Muse Glimmer, which runs quite well on my M1 Max as it is 30B but it also reasons very efficiently. I have tried the Qwen 3.8 27B which is also usable if quite slow to reason, but I guess like everyone the idea of the Flash-Next model holds some intrigue, because the 35B-A3B is pretty good.
I see your point. As an oss defender myself, I agree, however, the spirit of this is to see how fast I can make it. I'm sharing this with the community, which I think is aligned with the original oss spirit.
It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.
Hey carloslfu, kudos from the other side of the internet, don't get down on people nitpicking everything here, experimenting and discovering is part of learning so keep going!, remember this is the place that said dropbox was dumb and could be replaced by a script.
This is one of the aspects of this year that I've been finding very grating and wasteful. Collaboration still happens among people with the ability to do so and the technical skills, but everyone else is taking their own helicopter to the top of the mountain, "putting it out there", and there's just a ton of redundant projects that do the same thing.
Also for the record, I wasn't trying to take a personal shot at you or your project—nor am I sure of how valid that would be, if that were to have been my intention—it's just a gripe I have in general what I think is somewhat degrading the trust I can have in certain types of projects, especially those that someone shits out, puts on the app store, appears on the surface to visibly look ok, but ultimately has no uniquely valuable contribution or long-term outlook and is just someone's zero to one replication of something that has an api and already exists. It is admittedly cynical, but I now scrutizinize what I pay for more aggressively as a consequence.
I think multiple people working on the same thing is great.
Everyone comes at it from a different point of view, and some approaches work, some don't. And when people do this themselves they learn. Existing projects have their mistakes worked out already.
Maybe one of these people is going to come up with the thing that nobody else thought of because of their experience working the problem from scratch. You may not get that from someone working from an existing project, because existing projects have their approach "baked in."
What all these projects are showing so far is that it's possible to stream from disk, but that the performance isn't ideal. But I'm sure you could take this approach with smaller models and get better performance.
In addition, it's a given that when you work with large data sets performance means organizing the data to take advantage of caches, both disk and cpu. It's not clear how that would work, exactly, given that each run is a not-quite-random walk through the data. The Big Data way is to prebuild all of that as much as possible, which is probably impossible with a big model. But what about a smaller model?
Multiple people working on the same thing is great. I'm less convinced about multiple people asking the same LLMs to redo the same project and coming up with a repo with a llm slop readme full of "Disk bites first", it's not this, it's that etc...
I don't think that this will bread any innovations.
README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project".
I hate this AI style writing because since it doesn’t really understand flow, it’s being inserted in irrelevant places and it is extremely irritating to read.
For what it’s worth, this comment was not targeted at you, but rather the model kinda forcing it. I get the sense that Anthropic did not think much of this, but it seems to have gotten worse with recent models and it really comes off as a kind of nails on the chalkboard writing style.
I have to image whatever style of writing this was trained on is a lot more pleasant to read and I feel bad for whoever writes like this now being associated as bad AI writing.
most of this stuff isn't really possible at this point _without_ models, so pissing and moaning about people publishing work and not humanizing is really silly.
I used similar prompts before. Now I simply say to "remove historical cruft" and results are good enough. It's the model itself that first used this wording, I found it concise.
Honestly? Me neither, I'd be embarrassed if I got caught not even looking at my own work before shipping it publicly.
However, feels like the battle is loosing, and now it's just about softening the landing instead of being vehemently against something the vast majority seems to be engaging in. I refuse to participate myself, but at least hopefully I could help steer some of them in a better (more concise) direction.
yes! I'm bullish on this. there is a lot of work to do. I've been experimenting with pruning, distillation, and retraining too. I'm sure your 32gb m6 will run a badass local model!
Yes, but also 12 tok/s versus Claude is so far from comparable. I know that it’s not exactly 1:1, but it’s a long way from an easy trade-off, especially considering hardware prices for high levels of RAM.
It's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat
Now I'm feeling pretty good about getting 10-11 tokens/sec running Qwopus 3.6-35B-A3B Q6_K on an old Mac Pro 2013 (trashcan) with 128GB RAM (DDR3), 12 core Xeon, dual D700s. Arch Linux and llama.cpp.
The laptops definitely can't hang but the minis don't really care. I threw mine down in the basement just to put the heat somewhere else, can tell when the dehumidifer next to it is on because it's a few C lower but that has no impact on performance. I don't think it's ever seen anything north of 70
As someone who is just looking at the theoretical benchmarks of each of these models I'm curious if anyone could share what are the problems (maybe around code) that flash-next was able to solve which 27b was not able to
Reading causes insignificant wear ("read disturb") that likely isn't a problem, but I don't think it's possible to issue pure reads to modern SSDs. The NVMe spec mandates tracking the amount of data read, and this has to be written to the drive. I'd hope the firmware buffers this and writes it at low frequency, but on the other hand, I doubt the firmware was tested in extreme random-read regimes. Unexpected failures from excessive statistics recording could be possible.
Using macos on low memory regimes will make it use disk-based swap.
For example, a Macbook Neo (so in theory, something with around 4GiB of free RAM lying around) might eat around 900GB of writes a day while not doing much at all, because it's basically on low on RAM and swapping all the time.
I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?
We are running 35b-A3b with 264k context (the model's default max) using vllm and the "frog" jinja templates: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates and had good luck. We are mostly running agentic workloads though, rather than coding. 27b has a slightly higher agentic job completion rate (95% vs 92%) but the 3% trade off is worth it because the A3B is sooooo much faster, and we reprocess the other jobs with a different model. Don't sleep on the froggeric templates.
Qwen: Looking at you for a new ~35B MoE! Please and thank you
I am running 3.8 27b at q6 quant with 160k context on a 32gb video card (arc b70 pro) - I quantized the kv cache at q8 - that is the only trick really - works great.
I love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing at some point in the near future. Keep in mind that there is no guarantee that the pendulum has to swing back, it can swing one way and get stuck, and then you're going to have to beg for crumbs from the haves.
I think there's a very good chance that history will rhyme a bit.
DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as a good entry point for new users. As such they spread like wildfire and became the standard while the expensive alternatives either became hardcore niche or vanished.
32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model approach here?
Both projects are different in scope. Think of slotstream as optimizing for memory and for this specific model for now, my intention is not to build an inference engine the same as oMLX
Not a mac/UMA discussion point, but is it time to add additional, installable, DDR5 to GPUs? I can see this as a win/loose. PCIe 5x16 is close to maxing out the bandwidth available from high end dual channel DDR5 now, but not quite. I'm not a hardware person but I suspect putting it on the card could lead to significant performance improvements over using system ram so allowing systems like this, where MOE weights are shed, to get even higher performance than just adding that DDR5 to the system. Bigger models become closer to reality and it provides more of a pathway for developing technologies that take advantage of it. Of course the loose side is that you just put a lot of specialized ram on a card instead of into the system where it could be used for other things. I could see a place for a 16GB card with 64GB(or more) of DDR5 especially if we start seeing MOE and similar technologies really start being designed for this concept.
Probably not with DIMM modules, as longer traces mean higher latency (speed of light is ~30 cm in 1ns). GDDR typically uses larger buses (more wires) for higher bandwidth, even more so for HBM, so DIMM would be hard. Maybe CAMM would be up to the task?
It certainly seems feasible from an engineering perspective (though it does make cooling harder), at least for mid-range, not H100-class HW, but it prevents market segmentation, so EOMs may not be too interested (as long as no competitor does it).
For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s).
This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B.
I’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?
I just used it, went through the whole installation (took like 1 hour approx). Long but straightforward. If I put my computer to sleep will it continue?
I started the server (very curiously I was running oLlama in the same prot slotserve uses by default, instead of switching it which I know you can do, I just ditched oLlama, perhaps an insight for you) and built a small html hello world served via Python. The thing pointed me to the localhost link, nice!
As an early user, my advice is to focus on efficiency. The efficiency of the installation but more importantly, the efficiency of running the thing. 8.1GB per slotserve process is a lot! Is that in your control?
Also, I've seen an interest of certain kinds of programmers for open-weight models. "We all know agree that LLMs for coding are very useful but we're giving money to a small set of big, evil corporations. They're Trump donors. I heard it's bad for the environment because it uses water". If it's local and open-weight, this could be marketed this way I think.
Finally, what's the actual, real use case for slotserve?
AmazingTurtle | 6 days ago
At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo.
The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.
api | 6 days ago
That's open source since forever, unfortunately.
docheinestages | 6 days ago
[OP] carloslfu | 6 days ago
EyMaddis | 6 days ago
[OP] carloslfu | 6 days ago
genxy | 6 days ago
oceanplexian | 6 days ago
[OP] carloslfu | 6 days ago
I genuinely want to contribute. And hey! I was doing oss this since 2014 so waay before AI was cool.
Barbing | 6 days ago
[OP] carloslfu | 6 days ago
Barbing | 5 days ago
(The comments under the parent indicate it was improperly flagged/made dead (maybe could happen just from downvoting?) so glad I hit the Vouch.)
dofm | 6 days ago
[OP] carloslfu | 6 days ago
noir_lord | 6 days ago
[OP] carloslfu | 6 days ago
dofm | 6 days ago
I do agree that, ultimately, combining your efforts with others working in this whole area is probably really worth it, but I can see how there's an ease of pushing forward on your own these days.
I do not have fast internet so I am not sure when I'll really be able to download the weights but I do have an M1 Max to try this on, so I will at some point!
[OP] carloslfu | 6 days ago
dofm | 5 days ago
[OP] carloslfu | 6 days ago
It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.
xlayn | 6 days ago
kzrdude | 6 days ago
genxy | 6 days ago
brailsafe | 5 days ago
brcmthrowaway | 5 days ago
[OP] carloslfu | 5 days ago
brailsafe | 5 days ago
mannyv | 5 days ago
Everyone comes at it from a different point of view, and some approaches work, some don't. And when people do this themselves they learn. Existing projects have their mistakes worked out already.
Maybe one of these people is going to come up with the thing that nobody else thought of because of their experience working the problem from scratch. You may not get that from someone working from an existing project, because existing projects have their approach "baked in."
What all these projects are showing so far is that it's possible to stream from disk, but that the performance isn't ideal. But I'm sure you could take this approach with smaller models and get better performance.
In addition, it's a given that when you work with large data sets performance means organizing the data to take advantage of caches, both disk and cpu. It's not clear how that would work, exactly, given that each run is a not-quite-random walk through the data. The Big Data way is to prebuild all of that as much as possible, which is probably impossible with a big model. But what about a smaller model?
[OP] carloslfu | 5 days ago
sersi | 5 days ago
I don't think that this will bread any innovations.
ErenayDev | 6 days ago
[OP] carloslfu | 6 days ago
karmakaze | 6 days ago
0x457 | 6 days ago
[OP] carloslfu | 6 days ago
rzzzt | 5 days ago
0x457 | 5 days ago
[OP] carloslfu | 6 days ago
drcongo | 6 days ago
AI;DR
drums8787 | 6 days ago
How I have come to detest certain phrases.
bogzz | 6 days ago
drcongo | 6 days ago
brailsafe | 5 days ago
thirtygeo | 6 days ago
embedding-shape | 6 days ago
README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project".
[OP] carloslfu | 6 days ago
Eufrat | 6 days ago
[OP] carloslfu | 6 days ago
Eufrat | 6 days ago
I have to image whatever style of writing this was trained on is a lot more pleasant to read and I feel bad for whoever writes like this now being associated as bad AI writing.
cyanydeez | 5 days ago
makira | 5 days ago
trollbridge | 5 days ago
embedding-shape | 5 days ago
However, feels like the battle is loosing, and now it's just about softening the landing instead of being vehemently against something the vast majority seems to be engaging in. I refuse to participate myself, but at least hopefully I could help steer some of them in a better (more concise) direction.
ricardobeat | 5 days ago
This is the first line of the README. I can't believe people are becoming ok with this, and I'm 100% on the AI train.
whartung | 6 days ago
Folks talking about how 32G is not enough for local use, but then there's been work like this to empower it.
My hope is that the new 32G M6 will be "useful" locally, possibly because of work like this.
[OP] carloslfu | 6 days ago
tyre | 5 days ago
trollbridge | 5 days ago
prometheus1992 | 6 days ago
[OP] carloslfu | 6 days ago
Balooga | 5 days ago
prometheus1992 | 5 days ago
trollbridge | 5 days ago
monster_truck | 5 days ago
atif089 | 6 days ago
[OP] carloslfu | 6 days ago
About the specifics, I have only anecdotal evidence, but I guess this info can be found somewhere
red_hare | 5 days ago
jonplackett | 6 days ago
cromka | 6 days ago
[OP] carloslfu | 6 days ago
[OP] carloslfu | 6 days ago
mrob | 5 days ago
egorfine | 6 days ago
ElectricalUnion | 5 days ago
For example, a Macbook Neo (so in theory, something with around 4GiB of free RAM lying around) might eat around 900GB of writes a day while not doing much at all, because it's basically on low on RAM and swapping all the time.
Gigachad | 5 days ago
mulemisterX | 5 days ago
ig0r0 | 5 days ago
metadat | 5 days ago
ig0r0 | 5 days ago
pram | 5 days ago
hadlock | 5 days ago
Qwen: Looking at you for a new ~35B MoE! Please and thank you
kamranjon | 5 days ago
nikanj | 5 days ago
jacquesm | 5 days ago
cosmic_cheese | 5 days ago
DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as a good entry point for new users. As such they spread like wildfire and became the standard while the expensive alternatives either became hardcore niche or vanished.
jacquesm | 5 days ago
Though to be fair it was Linux more than Windows that killed them. Dos and Windows were competition for DEC and - ironically - IBM.
kethinov | 5 days ago
kzrdude | 5 days ago
siris9476 | 5 days ago
amelius | 5 days ago
pornel | 5 days ago
c0rruptbytes | 5 days ago
[OP] carloslfu | 5 days ago
[OP] carloslfu | 5 days ago
jmward01 | 5 days ago
MayeulC | 5 days ago
It certainly seems feasible from an engineering perspective (though it does make cooling harder), at least for mid-range, not H100-class HW, but it prevents market segmentation, so EOMs may not be too interested (as long as no competitor does it).
jwr | 5 days ago
This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B.
I use a 64GB MacBook Pro (M4 Max).
nixon_why69 | 5 days ago
Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.
ch_sm | 5 days ago
nixon_why69 | 5 days ago
baristaGeek | 5 days ago
I started the server (very curiously I was running oLlama in the same prot slotserve uses by default, instead of switching it which I know you can do, I just ditched oLlama, perhaps an insight for you) and built a small html hello world served via Python. The thing pointed me to the localhost link, nice!
As an early user, my advice is to focus on efficiency. The efficiency of the installation but more importantly, the efficiency of running the thing. 8.1GB per slotserve process is a lot! Is that in your control?
Also, I've seen an interest of certain kinds of programmers for open-weight models. "We all know agree that LLMs for coding are very useful but we're giving money to a small set of big, evil corporations. They're Trump donors. I heard it's bad for the environment because it uses water". If it's local and open-weight, this could be marketed this way I think.
Finally, what's the actual, real use case for slotserve?
[OP] carloslfu | 5 days ago
> ditched oLlama"
yeah! this is interesting.
> 8.1GB per slotserve process is a lot! Is that in your control?
yes, it is hard, but I agree the smaller the better. I'll work on that
> If it's local and open-weight, this could be marketed this way I think.
I like this!
> what's the actual, real use case for slotserve?
I'm working rn on an app on top of it that closes the loop and is a fully local AI app, an experiment. I'll publish it as soon as it is usable!
> built a small html hello world served via Python
What did you use as a harness here?
baristaGeek | 5 days ago
[OP] carloslfu | 5 days ago