Slightly different, it was ret vs br x30, but same idea - ret got predicted whilst br x30 did not. This time it was memory protected by STM's RIF (resource isolation framework).
We solved it a slightly different way, by only mapping the necessary memory, but then we're a bootloader not a hypervisor.
Funnily enough, the ZynqMP (ZCU102) board had been having periodic failures booting in 32-bit mode every so often (not consistent even with the same binary), and this solved it
CPUs don’t signal page faults for speculative accesses
This is not strictly true - the CPU core won't (which is what produces page faults, so technically correct), but on ARM, such as in the case with the RIF, speculative accesses that make it out to the system bus and to various peripherals can produce SErrors.
Oh wow. The first issue found on the way is a lot more "nightmare fuel" than the main event:
modifications to the data do not automatically propagate to the instruction fetches
I guess I never really even realized that that was the case because… well, modifying instructions is generally not something I would do just for fun, but that's quite the footgun for those rare cases when that happens.
x86 is somewhat weird here not requiring it: they have special code in the frontend that handles even self-modifying-code and will flush the pipeline (that doesn't work when modified through a different address - that requires a pipeline flush); and then i-cache and d-cache are coherent as you mentioned.
I'm more of an ARM person, I've barely touched x86, but it seems weird to me that they'd be coherent: after all, that's extra work the processor needs to do to maintain coherency for a relatively rare operation. (This is true also for loading programs into memory - not just modifying existing operations; which is arguably the same thing).
Arm server-grade cores have coherent icaches because explicit flushes are pretty expensive especially when with the system-wide broadcast needed combined with huge core counts.
However, on Arm's server cores, opting into the coherent icache comes at the price of having the L1i be strictly included within the L2. Arm does not offer that option on client-oriented cores.
Apple's design has a non-coherent L1i but Qualcomm's Oryon cores even on phones ship with a coherent L1i.
Great info as always! Not surprised about Oryon considering the server heritage there :)
Oh right— I haven't even realized until now that non-coherent L1i means it doesn't need to be included within L2 and you can just "win" some L2 space there! That makes a lot of sense.
On Arm high-end designs, the L1d is also strictly included within the L2, with the L2 handling core-wide coherency. Having the L1i be strictly included within the L2 followed that design, and with piggybacking on the L2 for coherency - to be able to retrofit it into an existing core.
Core lineages designed for coherent L1i from the get-go do not necessarily do that and a number of them have the L1i integrated into the coherency infrastructure.
Self-modifying code used to be relatively common back in the DOS days, when x86 chips didn't have cache (that was introduced in the 486). The coherent behavior in later chips was presumably to keep backwards compatibility with existing self-modifying code.
In retrospect maybe x86 family CPUs would be a little faster now if they had implemented it with a flag that makes you need to manually invalidate icache after you modify code, switched off by default but with a promise that the chip will run slightly faster if you switch it on.
juliaaa | 19 hours ago
We ran into this in the seL4 boot loaders as well(unverified code). More specifically, someone from STM32 contributing platform support for the STM32MP2 did, on a Cortex-A35.
Slightly different, it was
retvsbr x30, but same idea -retgot predicted whilstbr x30did not. This time it was memory protected by STM's RIF (resource isolation framework).We solved it a slightly different way, by only mapping the necessary memory, but then we're a bootloader not a hypervisor.
Funnily enough, the ZynqMP (ZCU102) board had been having periodic failures booting in 32-bit mode every so often (not consistent even with the same binary), and this solved it
This is not strictly true - the CPU core won't (which is what produces page faults, so technically correct), but on ARM, such as in the case with the RIF, speculative accesses that make it out to the system bus and to various peripherals can produce SErrors.
valpackett | 19 hours ago
Oh wow. The first issue found on the way is a lot more "nightmare fuel" than the main event:
I guess I never really even realized that that was the case because… well, modifying instructions is generally not something I would do just for fun, but that's quite the footgun for those rare cases when that happens.
juliaaa | 18 hours ago
x86 is somewhat weird here not requiring it: they have special code in the frontend that handles even self-modifying-code and will flush the pipeline (that doesn't work when modified through a different address - that requires a pipeline flush); and then i-cache and d-cache are coherent as you mentioned.
I'm more of an ARM person, I've barely touched x86, but it seems weird to me that they'd be coherent: after all, that's extra work the processor needs to do to maintain coherency for a relatively rare operation. (This is true also for loading programs into memory - not just modifying existing operations; which is arguably the same thing).
never_released | 8 hours ago
Arm server-grade cores have coherent icaches because explicit flushes are pretty expensive especially when with the system-wide broadcast needed combined with huge core counts.
However, on Arm's server cores, opting into the coherent icache comes at the price of having the L1i be strictly included within the L2. Arm does not offer that option on client-oriented cores.
Apple's design has a non-coherent L1i but Qualcomm's Oryon cores even on phones ship with a coherent L1i.
valpackett | 8 hours ago
Great info as always! Not surprised about Oryon considering the server heritage there :)
Oh right— I haven't even realized until now that non-coherent L1i means it doesn't need to be included within L2 and you can just "win" some L2 space there! That makes a lot of sense.
never_released | 7 hours ago
On Arm high-end designs, the L1d is also strictly included within the L2, with the L2 handling core-wide coherency. Having the L1i be strictly included within the L2 followed that design, and with piggybacking on the L2 for coherency - to be able to retrofit it into an existing core.
Core lineages designed for coherent L1i from the get-go do not necessarily do that and a number of them have the L1i integrated into the coherency infrastructure.
juliaaa | 5 hours ago
Interesting. I suppose flushing the appropriate lines in D&I-caches can get expensive if it needs to propagate to every core.
Do they do something similar and make DMA coherent? I would expect that would have a bigger impact if flushes/invalidates have to go to every core.
pbsd | 13 hours ago
Self-modifying code used to be relatively common back in the DOS days, when x86 chips didn't have cache (that was introduced in the 486). The coherent behavior in later chips was presumably to keep backwards compatibility with existing self-modifying code.
0x2ba22e11 | 9 hours ago
In retrospect maybe x86 family CPUs would be a little faster now if they had implemented it with a flag that makes you need to manually invalidate icache after you modify code, switched off by default but with a promise that the chip will run slightly faster if you switch it on.