No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10.
99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.
If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.
Qwen is weirdly expensive. Deepseek v4 flash is dirt cheap. You'd need at least 128gb of ram to run this model and in my experience, a days work with it costs around 80 cents.
That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week.
With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?
Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.
Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.
> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive.
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
>You couldn’t buy one of these if you wanted to right now.
You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.
I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not.
A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.
Ha fair, I'd definitely confirm with a salesperson before wiring them $300k. But most of the signs on the configurator seem to point to it including the GPUs? Not going to make 30k BTUs/hr of heat without the 8kw of GPUs.
Baseline yeah. But part of the reason you run open models is how much nicer fine tuning them is. Granted, you probably don't want to try and make LoRAs on a 4x RTX6000 setup, but you could if you really wanted to and there are other ways to modify models. And yes, if you're good at it, you can turn a piddly mid-range model that's only good at benchmarks into a heavyweight clanker (for a specific domain).
Ah thanks for the solid info, too bad. I'd seen them come up as a pretty good price for 6000 RTX's in the past, which seem generally pretty available, good source for those?
Yeah, they're good source. But the price for those GPUs is 5 figs even with the nvidia startup program nowadays. Also, I went back and looked. Most of my GPUs are actually from Central Computers who were great, but Exxact is real too. So "lots of" was inaccurate.
Also, the lead time I quoted was for individual 8x nodes.
> Pretty expensive is an understatement. [...] If you could it would be multiple hundreds of thousands of dollars.
Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and important distinction?
> You couldn’t buy one of these if you wanted to right now.
You could have spent all of 5 seconds of searching rather than just assuming[1]. You're not buying an Nvidia Superpod™.
> You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users.
That's certainly fair a point. Although in the English language, especially in legal contexts, the multi- prefix is used inclusively for fractional values. That is it's strictly >1, not >=2. IE an 18 month contract is a multi-year contract, or a $1.6 million dollar asset is a "multi-million" dollar asset. But this is uninteresting semantics.
You are right, but it also doesn't matter. The gap is just that big. You can run 1 single user of Kimi K3 and still not even come remotely close to the $70k or so that a single Opus 4.8 user can burn over the course of a month on left on max. An honestly lowballed amount I know from anecdote. The per-token cost is just really expensive.
> Your math is way off across this post.
You made one technical point above, one that doesn't ever arrive at a relevant rebuttal to the substance of my post. But please, I'd love to hear you elaborate, especially because I didn't actually give much math at all.
If you want math though, here's the math. Let's say you are paying a ridiculous amount of money for electricity, a price nobody in the US pays -- $2 per kilowatt hour. That's about 5x the average rate in California, 4x as in Hawai'i. 17kW @ $2/kWh * ~8766 hours in a year puts that cluster's electrical costs at just shy of ~$298k annually assuming it takes no breaks. Let's make matters worse and round that up to $300k. It's also assuming you didn't invest in a solar hookup for your building, which I don't know why you haven't at this point. 12 months of Claude burning $70k a month is $840k. For a buy in of, you know what, let's call it $500k. Why not? It still doesn't matter. The operating cost is so much lower it's paid for itself plus an additional $40k in the first year. Even at a ridiculous penalty in electricity that nobody pays, even overinflating the amount of money you'd pay for the cluster and the infrastructure to get it set up, it's not even remotely close for a single user where the gap is smaller (IE, you're not wasting "a million dollars" in a year by maxing out the $200k scaling limit every month)
You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! It doesn't matter. If the work it was doing today was useful, it will be useful next year too. And with the rapidly encroaching diminishing returns from parameter scaling, you're probably going to be just fine for a while. Maybe grab a quantized version of a newer Chinese model at the end, before grabbing a newer generation of AMD node. Those MI400s are looking pretty sweet after all.
> If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months
If you're locked in, then you're locked in. But don't pretend like you're saving money. You're not.
> it wouldn’t be some little secret that we only discover in a comment online.
Why does this have you so nasty and defensive? It's not a "little secret" that running your own infrastructure is cheaper. Of course it is. You know what else is cheaper? Owning your own office building out in the sticks, rather than leasing part of one in the city. Not everybody can make that work, there are no free lunches after all.
History repeats, these same exact lines were rolled out ad nauseum during the cloud craze. Datacenters are businesses, not charities. Frontier companies rent quite a fair amount of their infrastructure. Even if they resold that compute below cost (they don't), there's a pretty steep cliff before the economics start to look attractive.
Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?
In normal times in which hardware used to depreciate (lately that's not the case and HW even appreciates, but let's not get distracted), if you calculate only with depreciation costs, plus the fact that when you have such a setup, it'd take many 200$ subs to cover your lack of limits in the other, I think it'd not be a clear victory for any side.
If you just ask "who spent more in the first year" (100% depreciation) then even with 5-6 max accounts, buying HW will be a couple of times more expensive. But when does it make sense to ask that question?
Maybe the SotA models will need better hardware so your investment will not be useful after a year or you'd need very expensive upgrades? But then (as in Fable case) subscribers need to spend more too.
It’s not so clear after 5 years that you’ll come out ahead. You’ll have spent $20k. The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware.
Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.
The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.
(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber.
That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B
How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed.
Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though.
A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!
Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model
I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)
Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.
You have tried Qwen 3.8 27B before coming to this conclusion, I hope? It's an incremental improvement over 3.6, but I mostly want to make sure you didn't just try running some old junker before coming to this conclusion.
I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.
These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.
The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.
I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that?
If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?
amanzi | 5 hours ago
argee | 5 hours ago
dolebirchwood | an hour ago
Which are...?
argee | 14 minutes ago
pcarolan | 5 hours ago
Gigachad | 4 hours ago
api | 4 hours ago
asteroidburger | 2 hours ago
srcreigh | 2 hours ago
99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.
If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.
Gigachad | an hour ago
whatsThisBtn4 | 4 hours ago
He was a lead engineer, so after he announced it wasn't going to work, everyone pretended it never happened. But we all knew.
copper-float | 2 hours ago
ux266478 | 4 hours ago
With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?
Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.
Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.
Aurornis | 2 hours ago
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
ericd | 2 hours ago
You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.
CamperBob2 | an hour ago
A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.
ericd | an hour ago
Aurornis | an hour ago
Okay I love the open models, but the hype is getting ridiculous. The models you can run on 4 X RTX6000 are not Fable level.
CamperBob2 | an hour ago
And Opus is no slouch. I'm satisfied that GLM 5.3 is just as strong as Opus.
ux266478 | 23 minutes ago
Aurornis | an hour ago
No, you can get a quote for possibly being allocated one in the distant future.
The backlog for these is huge. You cannot buy one any time soon.
ericd | an hour ago
arjie | an hour ago
They're a good provider but you have to be a big shot buying NVL72s before you're getting anything within your payback period.
ericd | 44 minutes ago
arjie | 11 minutes ago
Also, the lead time I quoted was for individual 8x nodes.
ux266478 | an hour ago
Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and important distinction?
> You couldn’t buy one of these if you wanted to right now.
You could have spent all of 5 seconds of searching rather than just assuming[1]. You're not buying an Nvidia Superpod™.
> You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users.
That's certainly fair a point. Although in the English language, especially in legal contexts, the multi- prefix is used inclusively for fractional values. That is it's strictly >1, not >=2. IE an 18 month contract is a multi-year contract, or a $1.6 million dollar asset is a "multi-million" dollar asset. But this is uninteresting semantics.
You are right, but it also doesn't matter. The gap is just that big. You can run 1 single user of Kimi K3 and still not even come remotely close to the $70k or so that a single Opus 4.8 user can burn over the course of a month on left on max. An honestly lowballed amount I know from anecdote. The per-token cost is just really expensive.
> Your math is way off across this post.
You made one technical point above, one that doesn't ever arrive at a relevant rebuttal to the substance of my post. But please, I'd love to hear you elaborate, especially because I didn't actually give much math at all.
If you want math though, here's the math. Let's say you are paying a ridiculous amount of money for electricity, a price nobody in the US pays -- $2 per kilowatt hour. That's about 5x the average rate in California, 4x as in Hawai'i. 17kW @ $2/kWh * ~8766 hours in a year puts that cluster's electrical costs at just shy of ~$298k annually assuming it takes no breaks. Let's make matters worse and round that up to $300k. It's also assuming you didn't invest in a solar hookup for your building, which I don't know why you haven't at this point. 12 months of Claude burning $70k a month is $840k. For a buy in of, you know what, let's call it $500k. Why not? It still doesn't matter. The operating cost is so much lower it's paid for itself plus an additional $40k in the first year. Even at a ridiculous penalty in electricity that nobody pays, even overinflating the amount of money you'd pay for the cluster and the infrastructure to get it set up, it's not even remotely close for a single user where the gap is smaller (IE, you're not wasting "a million dollars" in a year by maxing out the $200k scaling limit every month)
You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! It doesn't matter. If the work it was doing today was useful, it will be useful next year too. And with the rapidly encroaching diminishing returns from parameter scaling, you're probably going to be just fine for a while. Maybe grab a quantized version of a newer Chinese model at the end, before grabbing a newer generation of AMD node. Those MI400s are looking pretty sweet after all.
> If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months
If you're locked in, then you're locked in. But don't pretend like you're saving money. You're not.
> it wouldn’t be some little secret that we only discover in a comment online.
Why does this have you so nasty and defensive? It's not a "little secret" that running your own infrastructure is cheaper. Of course it is. You know what else is cheaper? Owning your own office building out in the sticks, rather than leasing part of one in the city. Not everybody can make that work, there are no free lunches after all.
History repeats, these same exact lines were rolled out ad nauseum during the cloud craze. Datacenters are businesses, not charities. Frontier companies rent quite a fair amount of their infrastructure. Even if they resold that compute below cost (they don't), there's a pretty steep cliff before the economics start to look attractive.
[1] - https://www.avadirect.com/GIGABYTE-G893-ZX1-AAX4-Dual-AMD-EP...
pcarolan | an hour ago
ericd | an hour ago
egeozcan | 44 minutes ago
If you just ask "who spent more in the first year" (100% depreciation) then even with 5-6 max accounts, buying HW will be a couple of times more expensive. But when does it make sense to ask that question?
Maybe the SotA models will need better hardware so your investment will not be useful after a year or you'd need very expensive upgrades? But then (as in Fable case) subscribers need to spend more too.
srcreigh | 37 minutes ago
Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.
The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.
(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)
ericd | an hour ago
hkchad | 5 hours ago
brettdav | 2 hours ago
hkchad | 2 hours ago
ericd | 4 hours ago
That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
gruez | 4 hours ago
How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed.
[1] https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks
ericd | 4 hours ago
A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
villish | 3 hours ago
https://quesma.com/benchmarks/babaisbench/
ericd | 3 hours ago
Great showing from Sol, though.
But also, it's Baba Is You :-D
whatsThisBtn4 | 4 hours ago
If you are work from home and do dishes between prompts you can get a gpt3-like result.
I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.
ramgine | 3 hours ago
cdnsteve | 3 hours ago
xydac | 5 hours ago
alexgoodhart | 4 hours ago
Not many people share setup with actual setup handholding so that was very G of you
mkagenius | 4 hours ago
https://x.com/mkagenius/status/2093730391429685732
(xcancel seems to have received a cease and desist)
jumploops | 4 hours ago
I've since acquired two DGX Sparks, and it feels so much snappier.
c0rruptbytes | 4 hours ago
the sparks have much slower memory bandwidth is the trade off
jumploops | an hour ago
Another benefit of the 2x spark setup is that you can parallelize to ~6 streams pretty efficiently.
All depends on the workflows you’re using it for.
I’m quite excited for the M7 class machines.
shell0x | 3 hours ago
jumploops | 3 hours ago
[0]https://github.com/antirez/ds4
shell0x | 3 hours ago
mintflow | 4 hours ago
I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)
Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.
arcanemachiner | 4 hours ago
miles_io | 4 hours ago
whatsThisBtn4 | 4 hours ago
Agents require at least DeepSeek pro and even that is the minimum.
You might be able to get a good model to write instructions and run it in smaller models.
Otherwise, cool your AI got the current weather.
max979 | 4 hours ago
whatsThisBtn4 | 4 hours ago
Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.
crossroadsguy | 2 hours ago
Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?
brainless | 2 hours ago
These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.
The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.
I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
ttul | 30 minutes ago
If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?