Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
This is exciting, I have been wishing for good ternary models for a while as it seems more effective and combining it with FPGA I think this is a super promising path forward.
Someone mentioned that FPGAs though amazing to create bespoke chips that are fast, run into a fab capacity issue. Someone explained briefly to me that even if you could make an fpga that could do inference better than current GPUs it was economically possible to make them at scale. I can't recall whether that was for technology related issues or just because fab capacity is completely filled up.
One issue is that FPGAs are very fine grained and they need to gaurentee an image the tool confirms works for a chip must work across all produced chips. There is a level of binning for speed and binning for some disabled portions of the chip but it is a lot less flexible then disabling shader cores, memory controllers, and portions of the cache on a GPU.
With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
FPGAs should only really be used for:
Pre-silicon verification
Designs where power/cost aren't important (a lot of high end industrial or military for example)
Or low quantity designs that won't make an ROI when taping out an ASIC.
That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)
I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.
I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.
Hbm really is the magic that makes it work (economically)
> With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.
Isn't it like FPGAs use more transistors per equivalent transistor it simulates?
And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?
Over the week or so I have vibe coded a Lean-like language with
- quantitative types
- a Rust-ish DSL that lowers into the base language
- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language
- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.
- A category theory and real number library
- A graph storage engine with queries written using profunctor optics
- A datalog with an egglog in progress
- A Joern-like code analysis engine for C and C# using tree-sitter to parse
- A React style framework
- Editor plugins for my language
All at varying stages of completeness.
What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)
My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.
datadrivenangel | 5 hours ago
datadrivenangel | 5 hours ago
hmate9 | 5 hours ago
dofm | 4 hours ago
dvh | 4 hours ago
exe34 | 4 hours ago
ralph84 | 3 hours ago
surgical_fire | 3 hours ago
drob518 | an hour ago
stronglikedan | 3 hours ago
nyc_data_geek | 3 hours ago
hoppp | 2 hours ago
risyachka | 4 hours ago
Zero surprise it did it
ilaksh | 3 hours ago
But that still not a breakthrough or anything for such a small active set of parameters that is quantized
codekansas | 4 hours ago
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
chalkycrimp | 4 hours ago
cpldcpu | 4 hours ago
It seems implementing transformers is the new "hello world" of AI assisted coding.
saejox | 4 hours ago
Archit3ch | 3 hours ago
IshKebab | 3 hours ago
https://architectlabs.com/blog/frostbite
marvinborner | 3 hours ago
spwa4 | 3 hours ago
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
lxe | 3 hours ago
hoppp | 2 hours ago
jluxenberg | 2 hours ago
ricardobeat | 2 hours ago
mixtureoftakes | 2 hours ago
itissid | 2 hours ago
ShadowBlades512 | an hour ago
mikeayles | an hour ago
FPGAs should only really be used for:
Pre-silicon verification
Designs where power/cost aren't important (a lot of high end industrial or military for example)
Or low quantity designs that won't make an ROI when taping out an ASIC.
That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)
I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.
I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.
Hbm really is the magic that makes it work (economically)
femto | an hour ago
In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.
numpad0 | 22 minutes ago
And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?
mococa | 41 minutes ago
seanjhkwon | 29 minutes ago
reuben364 | 11 minutes ago
- quantitative types
- a Rust-ish DSL that lowers into the base language
- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language
- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.
- A category theory and real number library
- A graph storage engine with queries written using profunctor optics
- A datalog with an egglog in progress
- A Joern-like code analysis engine for C and C# using tree-sitter to parse
- A React style framework
- Editor plugins for my language
All at varying stages of completeness.
What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)
My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.