have request buffering off in your reverse proxy in front
From what I can tell, nginx is a bit of an outlier in having request buffering, and all of Traefik, Envoy (thus by extension many k8s gateway implementations), and HAProxy have it off by default. (I expect the mix of Traefik + Envoy will cover a lot of k8s setups.) The article also states that whether or not this affects it depends on the target in question: "Request buffering is not a general mitigation. Elasticsearch runs from one complete request, so this setting does not stop that chain."
use AVX512 on the host
The article says "AMD EPYC and Ryzen: Zen 4 or newer", and Zen 4 came out in 2022. As some arbitrary examples, Hetzner's CPX VPSs seem to currently all be at least on Zen 4 Genoa, and Netcup seems to already have Zen 5 VPSs available (and G11 was already Zen 4).
use java + base64
Their initial testing describes pre-auth RCE on the default setups for Keycloak, Jira, Confluence, Artifactory, and Jenkins; and post-auth for Elasticsearch, Gerrit, and Kafka. I don't think this is that rare, per se. At minimum, it probably affects a decent number of cloud infrastructure setups.
This might depend on whether the hosting setups you're familiar with look more like "nested VMs running on a single host with nginx in front" or "k8s cluster across 1 or more nodes" (in which case you don't have the VM unless [unless using something like Kata] and nowadays are very very likely not using nginx).
It's a bit unfortunate this claim inflation we've had on the security vulnerability side. The next time we have something of the same gravity as log4shell (which was in an entirely different league of bad) it'll struggle a lot more to get visibility and traction given how often the alarm is being rung.
At the same time this sort of vulnerability isn't nothing either. I don't know what's a good solution.
Compared to log4shell this feels like a nothingburger from the guys who already took 2h of my time for a previous nothingburger.
Not even a mention of minecraft, the reason for lingering custom java installations running on big machines 24/7 - and which log4shell famously was an attack against.
Like php2shell, this requires a somewhat tailored exploit - but that’s definitely possible; I liked the write up of a php2shell skeptic who put their Bitcoin where their mouth was, and ended up convincing themselves that the risk was real (but indeed hard to exploit).
Hacking Minecraft servers may motivate people, but that is obviously going to have a different level of impact than hacking business applications. Which run a lot of 24/7 Java.
Which seems to have been one of the worst setups and ended up using the php CLI dev server - as the author simply let claude do the job and didn't even have any php running at the start. So not sure how much I would give on that.
I think something that might help is a metric for the likelihood of mass exploitation. Afaik CVSS only measures complexity and CIA impact. A non-exhaustive list of questions that would be helpful to ask about a vulnerability:
Does the exploit need to be tailored per application that uses the library/system?
Does the exploit need to be tailored per build?
Does it require mulitiple consecutive requests to the same process?
Does it network latency hinder reliability of the exploit?
How common is the configuration?
(I think the claim inflation was inevitable. Relying on the person/org who found the vulnerability to correctly assess it's seriousness is a flawed approach. Even if ignoring commercial incentives, in the excitement of finding a serious vulnerability in a popular piece of software, it's difficult for a researcher to be perfectly objective.)
Interesting! The current EPPS score for this CVE is 0.00734 (between 0 and 1). But from https://www.first.org/epss/how-it-works.html it seems like most of the signals they use come later, like when the exploits get adopted in tooling, honey pots, threat intel feeds. It does include this:
Beyond CVSS, EPSS derives a set of descriptive tags directly from vulnerability descriptions which capture characteristics like whether the vulnerability allows remote code execution or other common labels found in descriptions. Vendor and some product information is also included as individual features in this category.
What makes it odd? Java is a pretty popular language, base64 is a normal way to encode binary data as text, many CPUs have had AVX512 for a decade, you have to disable request buffering for a host of reasons. To me it seems like 3 fairly common orthogonal conditions, nothing odd about it?
have request buffering off in your reverse proxy in front
From my reading, that's just a prerequisite for their chosen approach to do the necessary Heap Feng Shui to achieve RCE. It is not said this is the only way to exploit the bug. It may be easier to crash a JVM or send it off into some weird unpredictable behaviour by corrupting random objects.
I'd hesitate to give that as a mitigation. Their RCE might not work as-is against ZGC, but the bug itself isn't in G1 and at the very least can still be used for denial of service with ZGC.
"RCE everywhere" is a bit of a stretch. Requires AVX-512, which if you squint a bit is anything Intel, and AMD Epyc but only Zen 4 and up (Genoa onward).
But still, hardly everywhere. Java's slogan of Build Once, Run Everywhere that this is alluding to is specifically that it doesn't have such limitations.
I have not checked what the big cloud hosters are running, but if e.g. AWS is actually running mostly those for x64, I think it's the figure of speech as "AWS is everywhere". If you hit 50% of companies, it's everywhere enough ;)
We who don't use Java, and especially those of us who use and love Rust, may be tempted to be smug, but I think something like this could just as easily happen to us. IMO, the Rust ecosystem in particular tends toward maximum cleverness in libraries, so everyone can reuse the most efficient, most optimized possible implementation behind a safe interface just by adding a dependency.
Maybe what we need to avoid such vulnerabilities in the future is a strict policy about writing safe, purely portable code in a language like Rust, and the conservatism of someone writing in C or C++ who knows they're in a minefield and thus keeps their code as un-clever as possible, but again applied to a language like Rust. In other words, just get the thing working, with a bare minimum of features and optimizations. Then again, in user interfaces, some complicated features like internationalization and accessibility really do matter, and I'm sure there are equivalent considerations in server-side software.
Certainly Rust depends on some limited but non-trivial amount of internally unsafe code, like Java did here, so that will always be a potential target (even if the target is reduced).
...that being said, Rust does also have safe SIMD libraries, so I think you wouldn't need to go as low-level in order to achieve optimizations like this. (Of course, Java could have its own safe SIMD built on top of an unsafe core, but I don't think anyone's done that there just yet.)
Sadly, the built-in safe SIMD library. core::simd, is still nightly-only, which nudges folks towards the unsafe stuff that is available on non-nightly.
Earlier this year we found Squidbleed, an older cousin of Heartbleed, hiding in the Squid proxy since 1997. The bug was in Squid's FTP module. But who uses FTP anymore? It's a dead protocol and should be removed.
Isn't Automated Clearing House (ACH) basically a bunch of glorified FTP servers? That's pretty much anyone using direct deposit to get paid (so probably most salaries employees in the US, not sure about elsewhere).
I agree with the point that we should deprecate and remove old and unused features, or make them opt-in. But I suspect in a lot of cases that would affect more systems than this author realizes. The problem is vast.
proctrap | 21 hours ago
So if I read this right you need to
That's an odd combination in my books
refi64 | 20 hours ago
I don't think this is that odd.
From what I can tell, nginx is a bit of an outlier in having request buffering, and all of Traefik, Envoy (thus by extension many k8s gateway implementations), and HAProxy have it off by default. (I expect the mix of Traefik + Envoy will cover a lot of k8s setups.) The article also states that whether or not this affects it depends on the target in question: "Request buffering is not a general mitigation. Elasticsearch runs from one complete request, so this setting does not stop that chain."
The article says "AMD EPYC and Ryzen: Zen 4 or newer", and Zen 4 came out in 2022. As some arbitrary examples, Hetzner's CPX VPSs seem to currently all be at least on Zen 4 Genoa, and Netcup seems to already have Zen 5 VPSs available (and G11 was already Zen 4).
Their initial testing describes pre-auth RCE on the default setups for Keycloak, Jira, Confluence, Artifactory, and Jenkins; and post-auth for Elasticsearch, Gerrit, and Kafka. I don't think this is that rare, per se. At minimum, it probably affects a decent number of cloud infrastructure setups.
proctrap | 20 hours ago
I'm specifically thinking about typical VM setups that do not expose AVX512 in my experience.
Nginx though is widely used.
refi64 | 20 hours ago
This might depend on whether the hosting setups you're familiar with look more like "nested VMs running on a single host with nginx in front" or "k8s cluster across 1 or more nodes" (in which case you don't have the VM unless [unless using something like Kata] and nowadays are very very likely not using nginx).
proctrap | 20 hours ago
That's fair - my experience is VMs + HA where you have to opt-in to newer CPU specs.
marginalia | 20 hours ago
It's a bit unfortunate this claim inflation we've had on the security vulnerability side. The next time we have something of the same gravity as log4shell (which was in an entirely different league of bad) it'll struggle a lot more to get visibility and traction given how often the alarm is being rung.
At the same time this sort of vulnerability isn't nothing either. I don't know what's a good solution.
proctrap | 20 hours ago
Compared to log4shell this feels like a nothingburger from the guys who already took 2h of my time for a previous nothingburger.
Not even a mention of minecraft, the reason for lingering custom java installations running on big machines 24/7 - and which log4shell famously was an attack against.
JoachimSchipper | 12 hours ago
Like php2shell, this requires a somewhat tailored exploit - but that’s definitely possible; I liked the write up of a php2shell skeptic who put their Bitcoin where their mouth was, and ended up convincing themselves that the risk was real (but indeed hard to exploit).
Hacking Minecraft servers may motivate people, but that is obviously going to have a different level of impact than hacking business applications. Which run a lot of 24/7 Java.
proctrap | 8 hours ago
Which seems to have been one of the worst setups and ended up using the php CLI dev server - as the author simply let claude do the job and didn't even have any php running at the start. So not sure how much I would give on that.
lucasluitjes | 19 hours ago
I think something that might help is a metric for the likelihood of mass exploitation. Afaik CVSS only measures complexity and CIA impact. A non-exhaustive list of questions that would be helpful to ask about a vulnerability:
(I think the claim inflation was inevitable. Relying on the person/org who found the vulnerability to correctly assess it's seriousness is a flawed approach. Even if ignoring commercial incentives, in the excitement of finding a serious vulnerability in a popular piece of software, it's difficult for a researcher to be perfectly objective.)
mariell | 8 hours ago
AFAIU, this is practically the niche EPSS fills.
lucasluitjes | an hour ago
Interesting! The current EPPS score for this CVE is 0.00734 (between 0 and 1). But from https://www.first.org/epss/how-it-works.html it seems like most of the signals they use come later, like when the exploits get adopted in tooling, honey pots, threat intel feeds. It does include this:
But as part of a much larger set of signals.
mort | 19 hours ago
What makes it odd? Java is a pretty popular language, base64 is a normal way to encode binary data as text, many CPUs have had AVX512 for a decade, you have to disable request buffering for a host of reasons. To me it seems like 3 fairly common orthogonal conditions, nothing odd about it?
gerikson | 20 hours ago
At least this constant flood of CVEs is keeping me gainfully employed.
sjamaan | 10 hours ago
From my reading, that's just a prerequisite for their chosen approach to do the necessary Heap Feng Shui to achieve RCE. It is not said this is the only way to exploit the bug. It may be easier to crash a JVM or send it off into some weird unpredictable behaviour by corrupting random objects.
klingtnet | 9 hours ago
This was introduced with the intel Skylake family of processors in 2017. A big portion of servers is probably using newer CPUs than that.
marginalia | 4 hours ago
On the AMD side you need a CPU newer than 2023 though.
andersmurphy | 7 hours ago
Also looks like the only exploit is currently on G1. So won't work if you're using ZGC as far as I can tell.
sjamaan | 6 hours ago
I'd hesitate to give that as a mitigation. Their RCE might not work as-is against ZGC, but the bug itself isn't in G1 and at the very least can still be used for denial of service with ZGC.
marginalia | 20 hours ago
"RCE everywhere" is a bit of a stretch. Requires AVX-512, which if you squint a bit is anything Intel, and AMD Epyc but only Zen 4 and up (Genoa onward).
mort | 19 hours ago
That... is a lot of CPUs
marginalia | 19 hours ago
But still, hardly everywhere. Java's slogan of Build Once, Run Everywhere that this is alluding to is specifically that it doesn't have such limitations.
wink | 8 hours ago
I have not checked what the big cloud hosters are running, but if e.g. AWS is actually running mostly those for x64, I think it's the figure of speech as "AWS is everywhere". If you hit 50% of companies, it's everywhere enough ;)
mwcampbell | 9 hours ago
We who don't use Java, and especially those of us who use and love Rust, may be tempted to be smug, but I think something like this could just as easily happen to us. IMO, the Rust ecosystem in particular tends toward maximum cleverness in libraries, so everyone can reuse the most efficient, most optimized possible implementation behind a safe interface just by adding a dependency.
Maybe what we need to avoid such vulnerabilities in the future is a strict policy about writing safe, purely portable code in a language like Rust, and the conservatism of someone writing in C or C++ who knows they're in a minefield and thus keeps their code as un-clever as possible, but again applied to a language like Rust. In other words, just get the thing working, with a bare minimum of features and optimizations. Then again, in user interfaces, some complicated features like internationalization and accessibility really do matter, and I'm sure there are equivalent considerations in server-side software.
refi64 | 4 hours ago
Certainly Rust depends on some limited but non-trivial amount of internally unsafe code, like Java did here, so that will always be a potential target (even if the target is reduced).
...that being said, Rust does also have safe SIMD libraries, so I think you wouldn't need to go as low-level in order to achieve optimizations like this. (Of course, Java could have its own safe SIMD built on top of an unsafe core, but I don't think anyone's done that there just yet.)
hsivonen | 3 hours ago
Sadly, the built-in safe SIMD library. core::simd, is still nightly-only, which nudges folks towards the unsafe stuff that is available on non-nightly.
coby | 3 hours ago
Isn't Automated Clearing House (ACH) basically a bunch of glorified FTP servers? That's pretty much anyone using direct deposit to get paid (so probably most salaries employees in the US, not sure about elsewhere).
I agree with the point that we should deprecate and remove old and unused features, or make them opt-in. But I suspect in a lot of cases that would affect more systems than this author realizes. The problem is vast.