You can easily build a localized, access-controlled compute grid using Apple Silicon hardware. Haven't tested but as far as I can see, the key to making it performant would be keeping full models resident on individual nodes rather than attempting to distribute the tensor processing across the network.
> [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to complete larger tasks, similar to Folding@home and other distributed compute setups.
Maybe it's time to refresh this idea then. There's a clear demand for running LLMs at home, but the good ones are too big, and you're not using it most of the time. Sharing with others sounds like a perfect solution to me.
That explains the very old models mentioned on the page. There are models you can run locally on anything that would be more useful than llama 2 70B. e.g. the new Prism Bonsai 27B ternary model is 6GB, runs on any modern Mac, any PC with a gaming GPU bigger than 8GB (12GB or 16GB would be better), and probably runs on CPU faster than this runs any of those models. Gemma 4 E2B or E4B or 12B in the QAT quantization are fantastic for their size, and run on a lot of stuff. Models at every level have gotten a lot better since 2022.
It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately.
It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (Alpha Centauri ever relevant).
What? LLMs are perhaps the least bandwidth-constrained thing you can do over the network.
If you run a local model at a good pace, you're emitting tokens at, what, 60 tokens/sec? That's, very roughly, 300 bytes/sec. You can get that speed with dialup from the 80's! And when it takes a half second just to ingest the input tokens (to get to the part where you start generating the reply), you're not gonna care about even a crazy 250ms latency.
Unless you think that the model should be downloaded on-the-fly in response to a user request, there's zero reason for an LLM to need any sort of fast connection.
I have been thinking of something on these lines but with much smaller models. The entire model has to fit on a single computer. Host owner would choose the model they prefer, perhaps because they already use it. Then it is more about utilizing the GPU for LLM requests.
Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be equal - I have not thought through the fairness part. Perhaps initially it starts with "create your friends group and have access to all the host nodes".
I do not know what/who Sybil is but yes, there is a lot of plumbing in order to make sure that host nodes are ZDR* compliant and much more. Zero visibility of source prompt.
Also, this is the reason I want to start with trust based groups only - invite people you already know. I have friends who have Macs with 32GB or 48GB unified memory but sending prompts to them is not a out of the box thing.
> A Sybil attack is a type of attack on a computer network service in which an attacker subverts the service's reputation system by creating a large number of pseudonymous identities and uses them to gain a disproportionately large influence.
AI Horde has some measures to prevent Sybil attack that returns wrong results, but not enforce zero data retention. Prompts belong to the whole open source community. For example https://huggingface.co/datasets/la-ji/sd-prompt-in-the-wild
Petals is from 2022. Nowadays intelligence of smaller models, quantization techs, and optimizations to run models faster on consumer GPUs have improved a lot.
For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that powers SillyTavern. It also has an interesting economy model of kudos.
> For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines
But isn't what everyone want almost the complete opposite of this? For smaller LLMs you can already run them with consumer hardware, sometimes consumer hardware 2x, but it's the rest of the stuff where distributed inference could actually "democratize" access, as almost no one has 500GB VRAM available at home, or whatever the latest SOTA local models take nowadays.
I've occasionally written comments about the difficult of achieving proof-of-useful-work mechanisms and I should probably find or write a standard post about this question.
In order to be as decentralized as Bitcoin and related PoW mechanisms, a proof-of-work mechanism should have
* the ability to create an unbounded number of instances of the problem deterministically from numeric seeds
* the ability to scale difficulty up and down using a difficulty parameter
* in a way where anyone can easily confirm that a given problem instance corresponds to a given seed, and anyone can easily confirm that a given solution correctly solves a given problem instance
I'm not aware of a proof-of-useful-work mechanism that meets these criteria. (I'm also not aware of an argument that it's impossible to have one. Zero-knowledge proofs might actually go far toward making it possible in the future. but it's kind of complicated.)
The recently discussed https://meshllm.cloud/ is the one I've been playing with but don't have the hardware to try with a model split between nodes, which apparently is supported and just not part of the public demo.
https://public.meshllm.cloud/ - This should give the status of their public mesh. On an average I notice around 10 nodes on their public mesh serving various SLMs. You can use meshllm to share your GPUs like SETI@Home. Wondering how security is handled at a random node.
Distributed training is much harder than distributed inference but not impossible. See the recent development of DiLoCo at Nous Research and Prime Intellect.
I did a triple take. Many years old but an early ground breaking project. Has some exploratoy ideas a year or two ago last I looked. Crazy to see Petals at 16 frontage (at time of writing)
lostmsu | 18 hours ago
gatuamgb | 17 hours ago
wxw | 18 hours ago
Doesn't look like it's very active nowadays: https://github.com/bigscience-workshop/petals
Article with more details: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...
> [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to complete larger tasks, similar to Folding@home and other distributed compute setups.
mcv | 14 hours ago
SwellJoe | 14 hours ago
N_Lens | 18 hours ago
It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (Alpha Centauri ever relevant).
woctordho | 18 hours ago
AlecSchueler | 14 hours ago
N_Lens | 8 hours ago
hahahaa | 18 hours ago
behnamoh | 17 hours ago
c7b | 16 hours ago
qmmmur | 11 hours ago
fwip | 6 hours ago
If you run a local model at a good pace, you're emitting tokens at, what, 60 tokens/sec? That's, very roughly, 300 bytes/sec. You can get that speed with dialup from the 80's! And when it takes a half second just to ingest the input tokens (to get to the part where you start generating the reply), you're not gonna care about even a crazy 250ms latency.
Unless you think that the model should be downloaded on-the-fly in response to a user request, there's zero reason for an LLM to need any sort of fast connection.
brainless | 18 hours ago
Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be equal - I have not thought through the fairness part. Perhaps initially it starts with "create your friends group and have access to all the host nodes".
inigyou | 18 hours ago
brainless | 18 hours ago
Also, this is the reason I want to start with trust based groups only - invite people you already know. I have friends who have Macs with 32GB or 48GB unified memory but sending prompts to them is not a out of the box thing.
* typo
fragmede | 17 hours ago
https://en.wikipedia.org/wiki/Sybil_attack
woctordho | 18 hours ago
woctordho | 18 hours ago
For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that powers SillyTavern. It also has an interesting economy model of kudos.
[0] https://stablehorde.net/
embedding-shape | 12 hours ago
But isn't what everyone want almost the complete opposite of this? For smaller LLMs you can already run them with consumer hardware, sometimes consumer hardware 2x, but it's the rest of the stuff where distributed inference could actually "democratize" access, as almost no one has 500GB VRAM available at home, or whatever the latest SOTA local models take nowadays.
crowcroft | 18 hours ago
You could even hand out some kind of digital 'currency' to the people proportional to the amount of work their doing!
bravoetch | 18 hours ago
goda90 | 17 hours ago
gverrilla | 17 hours ago
schoen | 17 hours ago
In order to be as decentralized as Bitcoin and related PoW mechanisms, a proof-of-work mechanism should have
* the ability to create an unbounded number of instances of the problem deterministically from numeric seeds
* the ability to scale difficulty up and down using a difficulty parameter
* in a way where anyone can easily confirm that a given problem instance corresponds to a given seed, and anyone can easily confirm that a given solution correctly solves a given problem instance
I'm not aware of a proof-of-useful-work mechanism that meets these criteria. (I'm also not aware of an argument that it's impossible to have one. Zero-knowledge proofs might actually go far toward making it possible in the future. but it's kind of complicated.)
maccam912 | 18 hours ago
atum47 | 18 hours ago
tesnorindian | 11 hours ago
wsc981 | 17 hours ago
---
[0]: https://www.gnus.ai/
htrp | 17 hours ago
matheusmoreira | 17 hours ago
woctordho | 16 hours ago
petterroea | 17 hours ago
dang | 17 hours ago
Run LLMs at home, BitTorrent‑style - https://news.ycombinator.com/item?id=37546810 - Sept 2023 (125 comments)
ProofHouse | 16 hours ago
novia | 11 hours ago