This was a great read, thank you! David, Austin and Junyang on the Go team put in a lot of work to make the SIMD package a reality and it's nice to see it pay off.
For some context about the "Go Even Faster?" section: the Go compiler is deliberately designed to compile your code very fast and as such, gives up on some optimizations. The linear scan register allocator sometimes finds itself making bad choices around loops that would be avoidable if we used a more expensive graph-based algorithm, but linear scan is very fast and allows for quick iteration times. Being able to recompile the compiler in less than a second has been a massive boon for me working on it.
As for the prove pass getting smarter: I am currently working on a redesign, partly inspired by the new "shapes" of loops we'll see with SIMD intrinsics. Right now, the "remainder" loops often don't have bounds checks removed and combined with having to loop over per-item makes it particularly painful.
I'm use DCS regulary (maybe twice a month - IDK if that's regular though ;) ) and a paper of ours that's just been accepted at a security conference used DCS to find packages that used a pattern we were looking for. The pattern we looked for was pretty generic, but it still returned the packages really fast. Another paper of ours (currently in submission) looked for another generic pattern that turned up a million (or more?) references to packages using it... and it barely took a few seconds lol.
FWIW, the server behind DCS is a single Hetzner AX52, i.e. an AMD Ryzen 7 7700 8-Core Processor with 64 GB of RAM and two Samsung PM9A1 1TB SSDs. It’s pretty amazing to see what performance is available in today’s day and age!
What a great post! I'll need to do a second pass to understand everything. In the past, AVX512 was too power hungry and throttled the CPU, making it not as efficient it could be. I don't know if it applies to your AMD CPU or if it is still a concern at all with modern CPUs. Did you check with turbostat during your benchmarks?
Thanks, that means a lot coming from you, as your blog is also full of great posts! :)
I just had Claude run turbostat to check and AFAICT, on the AMD Ryzen 9950X3D I’m using, there is no AVX512 penalty: when running on a single core, all versions (scalar, AVX2, AVX512) run at 5.1 GHz. When running on all 16 cores at once (thereby maxing out the thermal limit of the CPU), the scalar versions throttles down to 4.9 GHz, the AVX512 version down to 4.7 GHz. So there is a measurable difference (the AVX512 version also uses 10W more power), but the AVX512 version is a lot faster than scalar in any case.
Anecdotally, I have the impression that this was an Intel problem of certain CPU generations, but I’m not an expert on this and a quick research didn’t find much. If anyone knows more, please add!
I'm curious whether AVX512 on that AMD CPU is much faster than AVX256. I should try benchmarks myself. I have vague recollection of hearing it mentioned that at least one generation of AMD CPUs implemented 512 bit SIMD by running the 256 bit wide ALUs twice, (taking less than double the time to do, so it was still a win.)
For my TurboPFor encoder, going from AVX2 to AVX512 yields almost as big a speed-up on Zen 4 as on Zen 5:
Zen 4: +84% Decode, +52% Encode
Zen 5: +97% Decode, +66% Encode
Claude Fable 5.1 tries to explain this as follows (emphasis mine):
Double-pumping is how Zen 4 supports 512-bit AVX-512 instructions without having 512-bit-wide execution hardware. It is a property of the vector execution back end, not the front end.
What Zen 4 actually has. The vector/FP unit in Zen 4 is 256 bits wide, the same physical width as Zen 2 and Zen 3. Its register file holds 512-bit architectural registers (zmm0-31) as pairs of 256-bit halves, and there are four 256-bit vector execution pipes (two FMA/ALU, two ALU/store-data style). Loads from L1 are 2×256 bits per cycle, stores 1×256 bits per cycle.
What happens to a 512-bit instruction. The decoder emits one macro-op for a zmm instruction, so the front end, rename and retirement see a single op. When it reaches the scheduler, the execution pipe processes it as two 256-bit halves on consecutive cycles, occupying the pipe twice. That is the "double pump". The result is that a 512-bit op has the same latency as a 256-bit op plus one cycle and half the per-instruction throughput. Two zmm ops per cycle on two pipes deliver the same 1024 bits per cycle as four ymm ops would, so peak arithmetic throughput per cycle is unchanged versus AVX2 code. You cannot go faster with 512-bit ops, only denser.
What it does not affect. Fetch, decode, the op cache, rename and the reorder buffer all handle a zmm op as one op. This is exactly why AVX-512 still helps on Zen 4: the codec is bound by instruction count, branch and mask handling and front-end work, not by execution width. Halving the instruction count halves the front-end and scheduler work regardless of how the back end executes it. It also gets the AVX-512 ISA features (masking, vpermb, vpcompress, 32 registers) that have nothing to do with width.
Where it does bite. Anything bound by pure execution throughput on 512-bit data: FMA-heavy kernels, wide stores (only 256 bits per cycle on Zen 4), and shuffles that cross the two 256-bit halves, which need extra µops. Those are the cases where the bw7 pure bitunpack rows showed the largest Zen 5 lead in the numbers above.
Zen 5 by contrast has a true 512-bit datapath: 512-bit execution pipes, 2×512-bit loads and 1×512-bit store per cycle (on desktop parts, the full-width path is enabled; some mobile parts keep 256-bit). A zmm op occupies a pipe once. So on Zen 5 AVX-512 gives both the instruction-count win and a doubling of per-cycle vector throughput, which is why its AVX-512 speedup over AVX2 is a bit larger (+97% vs +84% on decode).
When introduced on Skylake, our fear was that enabling VM to use AVX512, it would throttle unrelated VMs due to thermal constraint. I think it was a widespread believe at the time, so we didn't try it. Also, when introducing a new CPU in a fleet, it's interesting to keep emulating an older one to ensure we can migrate VMs from one host to the other. So, maybe we weren't that interested in enabling AVX512.
DMorsing | 15 hours ago
This was a great read, thank you! David, Austin and Junyang on the Go team put in a lot of work to make the SIMD package a reality and it's nice to see it pay off.
For some context about the "Go Even Faster?" section: the Go compiler is deliberately designed to compile your code very fast and as such, gives up on some optimizations. The linear scan register allocator sometimes finds itself making bad choices around loops that would be avoidable if we used a more expensive graph-based algorithm, but linear scan is very fast and allows for quick iteration times. Being able to recompile the compiler in less than a second has been a massive boon for me working on it.
As for the prove pass getting smarter: I am currently working on a redesign, partly inspired by the new "shapes" of loops we'll see with SIMD intrinsics. Right now, the "remainder" loops often don't have bounds checks removed and combined with having to loop over per-item makes it particularly painful.
sneela | 20 hours ago
This is pretty awesome and very interesting!
I'm use DCS regulary (maybe twice a month - IDK if that's regular though ;) ) and a paper of ours that's just been accepted at a security conference used DCS to find packages that used a pattern we were looking for. The pattern we looked for was pretty generic, but it still returned the packages really fast. Another paper of ours (currently in submission) looked for another generic pattern that turned up a million (or more?) references to packages using it... and it barely took a few seconds lol.
[OP] stapelberg | 20 hours ago
Thanks, glad to hear you find DCS useful!
It’s great to see that your queries are fast :)
FWIW, the server behind DCS is a single Hetzner AX52, i.e. an AMD Ryzen 7 7700 8-Core Processor with 64 GB of RAM and two Samsung PM9A1 1TB SSDs. It’s pretty amazing to see what performance is available in today’s day and age!
vbernat | 19 hours ago
What a great post! I'll need to do a second pass to understand everything. In the past, AVX512 was too power hungry and throttled the CPU, making it not as efficient it could be. I don't know if it applies to your AMD CPU or if it is still a concern at all with modern CPUs. Did you check with turbostat during your benchmarks?
[OP] stapelberg | 19 hours ago
Thanks, that means a lot coming from you, as your blog is also full of great posts! :)
I just had Claude run turbostat to check and AFAICT, on the AMD Ryzen 9950X3D I’m using, there is no AVX512 penalty: when running on a single core, all versions (scalar, AVX2, AVX512) run at 5.1 GHz. When running on all 16 cores at once (thereby maxing out the thermal limit of the CPU), the scalar versions throttles down to 4.9 GHz, the AVX512 version down to 4.7 GHz. So there is a measurable difference (the AVX512 version also uses 10W more power), but the AVX512 version is a lot faster than scalar in any case.
Anecdotally, I have the impression that this was an Intel problem of certain CPU generations, but I’m not an expert on this and a quick research didn’t find much. If anyone knows more, please add!
0x2ba22e11 | 17 hours ago
I'm curious whether AVX512 on that AMD CPU is much faster than AVX256. I should try benchmarks myself. I have vague recollection of hearing it mentioned that at least one generation of AMD CPUs implemented 512 bit SIMD by running the 256 bit wide ALUs twice, (taking less than double the time to do, so it was still a win.)
[OP] stapelberg | 16 hours ago
You’re describing AMD’s Zen 4 :)
For my TurboPFor encoder, going from AVX2 to AVX512 yields almost as big a speed-up on Zen 4 as on Zen 5:
Claude Fable 5.1 tries to explain this as follows (emphasis mine):
vbernat | 13 hours ago
When introduced on Skylake, our fear was that enabling VM to use AVX512, it would throttle unrelated VMs due to thermal constraint. I think it was a widespread believe at the time, so we didn't try it. Also, when introducing a new CPU in a fleet, it's interesting to keep emulating an older one to ensure we can migrate VMs from one host to the other. So, maybe we weren't that interested in enabling AVX512.
David Lemire said at the time that this was documented but he wasn't able to reproduce. https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-instructions-are-maybe-not-so-dangerous/