I'll evaluate this from my perspective, as someone who self hosts Nextcloud, and try to judge the points as they apply to myself.
Even a moderately popular production website will have multiple web VMs/containers running multiple php workers behind a load balancer and constant bot traffic.
This may be true, but I don't. I'm betting most people who just self host a PHP thing don't.
Let’s say you try this for the 50 most popular php/OS combinations, and any number of workers between 10 and 1010.
Against myself, you could've looked at the latest version of Nextcloud's official Docker image. It has my PHP version, my OS version, and my number of workers. I haven't changed the defaults.
(stuff about WAF)
I don't run a WAF, and I have seen security people complain about them and essentially call them useless. Maybe they're wrong, and maybe I should've been running a WAF. In any case, Nextcloud's default configuration doesn't include one, and I don't think most other self hosted services do either.
Rate limiting is obviously a good idea independent of WAFs but since we're talking about 153 requests here I don't know that that would've kicked in.
Against very low-traffic websites that do not have any bot protection, leak information about their version, running close to stock configuration, you might get lucky.
Well there you have it. That's me, and probably most people who self host Nextcloud.
I don't know how typical my situation is, but I'm feeling good about having taken both my Nextcloud instances offline once the vulnerability was teased.
From that perspective, it makes sense. But also, in that case the consequences of turning it off for an afternoon are very low. So you see the announcement, turn it off, wait until the patch commes, turn it on again, no big deal.
On the other hand, if you're running a medium or large site, the consequences of turning it off are significant. I spent the entire day frantically looking through hn/reddit/lobsters/fediversetwitter for new information, implementing remediation, compiling php because some distros were slow, tweaking WAF rules, turning assets off, communicating with clients about the risks of incidents vs downtime.
Assuming my take in the original post is correct, that was all basically unnecessary.
I still think it should've been communicated that it was a very serious vulnerability that needed to be patched asap, but a little "requires days of very noticeable/blockable DoS'ing to exploit moderate to high traffic websites" would have saved devops/infosec teams around the world a whole lot of time, prevented a lot of panic. I did write that in the post, though not particularly prominent. Several countries disabled half their government services in response to this vuln.
Since the AI security vendors benefit from fear of powerful hacking agents, I feel like many people got used for their marketing campaigns. And I'm concerned that if this behaviour becomes the new normal, the next heartbleed/shellshock style vulnerability that truly warrants the panic, where a single malicious request can get you root, will not get taken as seriously as it should.
I don't run a WAF, and I have seen security people complain about them and essentially call them useless
I'm a bit of a WAF hater. I consider them mostly fuzzy bug injectors. Especially since the headline features are usually like "Stop SQL injection" which isn't possible to do reliably.
But this is exactly what WAFs are great at, blocking untargeted exploitation of vulnerabilities in popular software. Especially if you have a 24/7 security team (likely outsourced) deploying rules to mitigate new vulnerabilities until I woke up and figure out how to patch them. It doesn't prevent you needing to patch, and it often won't stop a targeted attack. But, if I was running Wordpress on the public internet I would strongly consider paying someone to provide a WAF for me.
But more to your precise point, yes, I don't think we should ever dismiss a vulnerability because a WAF could mitigate it. WAFs with quality and up-to-date rulesets shouldn't be assumed.
This may be true, but I don't. I'm betting most people who just self host a PHP thing don't.
Same but then there is a trivial work-around: limit PHP processes to handle a single request. The additional load won't affect you. I'm a bit sad that there weren't more advice on work-arounds. I'm not more than a bit sad because updating was easy and considering how stretched and under-funded the PHP security team is, talking about work-arounds is only going to be more work (and possibly more criticism).
Yeah I'd rather not find out that my nextcloud with the data of me and friends got breached just because it's not behind cloudflare or has hundreds of users. I see it as happy accident that it turns out to be not that easily to be abused, could have been far far worse.
And maybe this was the necessary drill to prepare for the upcoming CVSS 10 bugs we will surely soon see just about anywhere. I've seen enough people tell me they can't afford to shut down their php stuff.
And I really wonder how bad shit has to get for them to just damn shut down until the storm literally breaching your servers and user data is over. As if their stuff has to have more uptime than AWS or Github and they rather have to explain to their local government agency why they lost everyone's data.
As someone else who self-hosts Nextcloud, I have to wonder....if you are not a user of my Nextcloud, how vulnerable am I really?
Sure, I can imagine an actual user of my Nextcloud could exploit this, but the users of my Nextcloud are my spouse and me. If my spouse wanted code execution on my server, she could just ask for the password if she really wanted it.
The only webpage you can access is the login page, which, yes, has a way to POST info, but AFAICT, when Nextcloud was put as "vulnerable", it was just that it's docker image has a vulnerable PHP version. I would certainly hope (yes, I know, security shouldn't rest on hope, but here we are) that Nextcloud understands that the login page needs to handle un-trusted input and is structured accordingly.
I also have been monitoring my Nextcloud instance, I didn't see any weird logs and see no odd processes (thank you Debian for not having random processes like "svchost.exe").
Granted, when the patch dropped. I was at work and couldn't update until 10 hours later (because I have job and family obligations), but really, like the website said, you need to have a very specific combination to really exploit this, and given that, I was probably attacked with some lazy script kiddies that failed ans they moved on. Could it have been exploited? Theoretically yes. Was it? Based on what I am seeing, and what I know about the exploit, I doubt it.
(Not to minimize how you feel, trust me, I get it! But this is what I am thinking with my Nextcloud instance)
AIUI, the exploit doesn't care whether your app accepts POST requests or not. The exploit succeeds or fails before necessarily interacting with your app because it is attacking the PHP runtime.
If the language maintainers themselves say it's a bad vuln and to take the server offline before patches come out then I don't really blame most orgs for reacting as they did. I suspect there was also a bit of a telephone game going on from https://lobste.rs/s/t73wqi/just_rumour_bug_is_enough_find_security where automated attacks starting in minutes from patches being pushed was conflated with finding said vuln + developing a reliable poc that works everywhere in minutes of knowing that a patch will be pushed.
I do blame the vendors who irresponsibly use such vulns for clout (calif couldn't be bothered to check dates before uploading their poc video?), break embargoes they agreed to (chainguard!) and put others at risk. And also everyone's trying to take credit for finding it? See: https://shellsharks.com/php2chaos
This all just leaves a very bad taste in my mouth seeing how vendors act.
I myself specifically don't blame chainguard, they published images only once the patch was already virtually as good as leaked, and with exploit attempts in the wild.
Not extremely clued in to the whole chain of events, so my reading might be off, but it seems to me that the panic was fuelled by the recent over-confidence in, and over-reliance on, AI agents and LLMs. As soon as rumours of the exploit hit the internet, everyone and their grandma put their chatbots on the task - and chatbots are very good at very quickly producing a large volume of results that look very convincing but only half-work or don't hold up of the further scrutiny, and also immediately try to attack targets if controlled by a black hat. The result was a deluge of "confimed exploits" and "active attacks" hitting the internet as soon as the news broke.
People scrambling to respond to this probably themselves had to resort to LLMs to analyze the situation due to the lack of reliable information and the speed in which everything was moving, and knowing how the current models are primed to overemphasize security issues (and be overly dramatic in general) it would not surprise me if they defaulted to confirming people's worst fears: "The exploit has been thoroughly corroborated. This isn’t just a bug. It’s a genuinely profound vulnerability woven into a load-bearing part of the whole internet."
Combine this with a pre-existing fear that AI actually would be able to exploit something like this in a catastropic manner, and a reasonable better-safe-than-sorry stance, and the reports of widespread severe exploits become very believable even for experts.
Actually analyzing the situation and the real impact still required human effort and thoughtful analysis, but such an approach moves with a glacial speed in comparison.
Problem here is that there has been a steady stream of local privilege escalation bugs for linux for the past ... what, 18 months? Combine any of these with any kind of RCE and you're toast. So the danger is real, especially as people are usually lazy in patching LPE bugs on dedicated servers.
yeah, and the PHP maintainers issued that advisory which described it as very high severity and turning off the server as the only way to mitigate before the patch. so everybody did that, because that's the correct action when a language maintainer tells you to.
I hope people don't get a false sense of security that leads them to under-react next time.
I'll be interested to learn more, once there's been full analysis, about whether other security people agree that the maintainers made the right call.
There are many, many PHP installations that have few users, live on an old VPS or are in a long forgotten Docker container. Those all seem like they are very exploitable.
This is obviously a non-trivial bug to actually exploit in the wild, with a lot of variables, but i think it's perfectly reasonable to assume that some bad actor would be able to weaponize this against a small amount of targets at least, which is why i think the response is reasonable (shutting down sites, wait for update).
As for the mass-exploitation scare, it was obviously a bit exaggerated, and this was obvious from the first POCs becoming public, but if no one thought this was serious and just kept their site running someone could have eventually developed a POC for whatever version of software they were running (Wordpress, Nextcloud, etc) and started mass exploiting vulnerable public instances.
That’s actually, for me, one of the more subtle pieces of it:
non-trivial bug to actually exploit in the wild
If it were trivial to exploit, there would likely be a massive armada of shit disturbers pointing their mass DDoS tools at the entire internet and popping every PHP server in existence. That might still happen, but if hitting any particular server is a non-trivial exercise it might not be worth the effort.
On the other hand, because there isn’t visible mass exploitation, I’m speculating that the massive long tail of small services running older versions of PHP will just continue on running unpatched, ready to be exploited in the future in a more targeted campaign. Bob’s wordpress blog that isn’t on anyone’s radar until he posts something that upsets someone and gets some publicity, then boom… popped.
Maybe this ends up being one of the few times when things like CloudFlare and their WAF stuff ends up actually being a good thing? Hard to say for sure, but for me personally the fact that this isn’t mass-automated-exploitable suggests that the door’s going to remain open for a long long time and will probably be talked about in security circles like MS08-067 (literally the only Microsoft security advisory that I know by memory)
Bad ol' Conficker, forever clanking around on some WinXP machines around the world... I remember having to manually configure that service to not restart-pc-on-crash before applying a patch. Amazing staying power.
Also unclear, what I did see is all the usual suspects (cloudflare, aws) claiming to seen attack attempts. I am not sure I have seen any reports about websites being compromised.
The Doylist answer is that no one who would be compromised from this, especially after it went fully public, would have capacity to know that they got compromised.
I apologize for nitpicking, but I don't understand in what sense this is Doylist. reality doesn't have a writer whose reasoning we are party to and can analyze? is it some sort of metaphor?
Yeah, I was aiming for a metaphor, might have misused it. Maybe a rephrasing will be clearer: as a corollary, any organisation running PHP that didn't patch or take down the site within few hours of the public patches being available is perhaps best considered to be unable to monitor their own systems too.
Speaking for myself: I tried to shut down a site I (barely) maintain before the patch was released. Unfortunately I bungled it (I stopped but did not mask the Apache service, but the machine rebooted for some reason), and the site came back up before the patch dropped, and remained up until I got home from work.
That machine is showing no obvious signs of compromise (eg crypto miners eating all the CPU).
I doubt I'm the only person in my position, who knows enough to recognize a compromised machine, but for whatever reason failed to cleanly handle the patch.
mort | 20 hours ago
I'll evaluate this from my perspective, as someone who self hosts Nextcloud, and try to judge the points as they apply to myself.
This may be true, but I don't. I'm betting most people who just self host a PHP thing don't.
Against myself, you could've looked at the latest version of Nextcloud's official Docker image. It has my PHP version, my OS version, and my number of workers. I haven't changed the defaults.
I don't run a WAF, and I have seen security people complain about them and essentially call them useless. Maybe they're wrong, and maybe I should've been running a WAF. In any case, Nextcloud's default configuration doesn't include one, and I don't think most other self hosted services do either.
Rate limiting is obviously a good idea independent of WAFs but since we're talking about 153 requests here I don't know that that would've kicked in.
Well there you have it. That's me, and probably most people who self host Nextcloud.
I don't know how typical my situation is, but I'm feeling good about having taken both my Nextcloud instances offline once the vulnerability was teased.
[OP] lucasluitjes | 12 hours ago
From that perspective, it makes sense. But also, in that case the consequences of turning it off for an afternoon are very low. So you see the announcement, turn it off, wait until the patch commes, turn it on again, no big deal.
On the other hand, if you're running a medium or large site, the consequences of turning it off are significant. I spent the entire day frantically looking through hn/reddit/lobsters/fediversetwitter for new information, implementing remediation, compiling php because some distros were slow, tweaking WAF rules, turning assets off, communicating with clients about the risks of incidents vs downtime.
Assuming my take in the original post is correct, that was all basically unnecessary.
I still think it should've been communicated that it was a very serious vulnerability that needed to be patched asap, but a little "requires days of very noticeable/blockable DoS'ing to exploit moderate to high traffic websites" would have saved devops/infosec teams around the world a whole lot of time, prevented a lot of panic. I did write that in the post, though not particularly prominent. Several countries disabled half their government services in response to this vuln.
Since the AI security vendors benefit from fear of powerful hacking agents, I feel like many people got used for their marketing campaigns. And I'm concerned that if this behaviour becomes the new normal, the next heartbleed/shellshock style vulnerability that truly warrants the panic, where a single malicious request can get you root, will not get taken as seriously as it should.
kevincox | 9 hours ago
I'm a bit of a WAF hater. I consider them mostly fuzzy bug injectors. Especially since the headline features are usually like "Stop SQL injection" which isn't possible to do reliably.
But this is exactly what WAFs are great at, blocking untargeted exploitation of vulnerabilities in popular software. Especially if you have a 24/7 security team (likely outsourced) deploying rules to mitigate new vulnerabilities until I woke up and figure out how to patch them. It doesn't prevent you needing to patch, and it often won't stop a targeted attack. But, if I was running Wordpress on the public internet I would strongly consider paying someone to provide a WAF for me.
But more to your precise point, yes, I don't think we should ever dismiss a vulnerability because a WAF could mitigate it. WAFs with quality and up-to-date rulesets shouldn't be assumed.
adrien | 19 hours ago
Same but then there is a trivial work-around: limit PHP processes to handle a single request. The additional load won't affect you. I'm a bit sad that there weren't more advice on work-arounds. I'm not more than a bit sad because updating was easy and considering how stretched and under-funded the PHP security team is, talking about work-arounds is only going to be more work (and possibly more criticism).
zem | 7 hours ago
wouldn't anyone who is aware and capable enough to deploy a workaround also be capable of deploying the patched php instead?
proctrap | 15 hours ago
Yeah I'd rather not find out that my nextcloud with the data of me and friends got breached just because it's not behind cloudflare or has hundreds of users. I see it as happy accident that it turns out to be not that easily to be abused, could have been far far worse.
And maybe this was the necessary drill to prepare for the upcoming CVSS 10 bugs we will surely soon see just about anywhere. I've seen enough people tell me they can't afford to shut down their php stuff.
And I really wonder how bad shit has to get for them to just damn shut down until the storm literally breaching your servers and user data is over. As if their stuff has to have more uptime than AWS or Github and they rather have to explain to their local government agency why they lost everyone's data.
kop316 | 14 hours ago
As someone else who self-hosts Nextcloud, I have to wonder....if you are not a user of my Nextcloud, how vulnerable am I really?
Sure, I can imagine an actual user of my Nextcloud could exploit this, but the users of my Nextcloud are my spouse and me. If my spouse wanted code execution on my server, she could just ask for the password if she really wanted it.
The only webpage you can access is the login page, which, yes, has a way to POST info, but AFAICT, when Nextcloud was put as "vulnerable", it was just that it's docker image has a vulnerable PHP version. I would certainly hope (yes, I know, security shouldn't rest on hope, but here we are) that Nextcloud understands that the login page needs to handle un-trusted input and is structured accordingly.
I also have been monitoring my Nextcloud instance, I didn't see any weird logs and see no odd processes (thank you Debian for not having random processes like "svchost.exe").
Granted, when the patch dropped. I was at work and couldn't update until 10 hours later (because I have job and family obligations), but really, like the website said, you need to have a very specific combination to really exploit this, and given that, I was probably attacked with some lazy script kiddies that failed ans they moved on. Could it have been exploited? Theoretically yes. Was it? Based on what I am seeing, and what I know about the exploit, I doubt it.
(Not to minimize how you feel, trust me, I get it! But this is what I am thinking with my Nextcloud instance)
cmcaine | 14 hours ago
AIUI, the exploit doesn't care whether your app accepts POST requests or not. The exploit succeeds or fails before necessarily interacting with your app because it is attacking the PHP runtime.
7tehdt3cnw6kir6o | 20 hours ago
If the language maintainers themselves say it's a bad vuln and to take the server offline before patches come out then I don't really blame most orgs for reacting as they did. I suspect there was also a bit of a telephone game going on from https://lobste.rs/s/t73wqi/just_rumour_bug_is_enough_find_security where automated attacks starting in minutes from patches being pushed was conflated with finding said vuln + developing a reliable poc that works everywhere in minutes of knowing that a patch will be pushed.
I do blame the vendors who irresponsibly use such vulns for clout (calif couldn't be bothered to check dates before uploading their poc video?), break embargoes they agreed to (chainguard!) and put others at risk. And also everyone's trying to take credit for finding it? See: https://shellsharks.com/php2chaos
This all just leaves a very bad taste in my mouth seeing how vendors act.
kwas | 19 hours ago
I myself specifically don't blame chainguard, they published images only once the patch was already virtually as good as leaked, and with exploit attempts in the wild.
mxey | 19 hours ago
How was that the case?
kwas | 19 hours ago
EDIT: Oh, okay, perhaps not patch, but the vuln.
AWS saw xploit attempts already on Saturday.
https://aws.amazon.com/blogs/security/amazon-threat-intelligence-identifies-php2shell-exploitation-prior-to-public-disclosure-in-madpot/
On Tuesday afternoon it was already widely talked that it's something related to multipart messages.
https://x.com/k_firsov/status/2112641845926989944
mxey | 19 hours ago
Exploit attempts are not exploits.
And it was in fact not specific to multipart messages. That was just one possible vector.
pocket_walrus | 20 hours ago
Not extremely clued in to the whole chain of events, so my reading might be off, but it seems to me that the panic was fuelled by the recent over-confidence in, and over-reliance on, AI agents and LLMs. As soon as rumours of the exploit hit the internet, everyone and their grandma put their chatbots on the task - and chatbots are very good at very quickly producing a large volume of results that look very convincing but only half-work or don't hold up of the further scrutiny, and also immediately try to attack targets if controlled by a black hat. The result was a deluge of "confimed exploits" and "active attacks" hitting the internet as soon as the news broke.
People scrambling to respond to this probably themselves had to resort to LLMs to analyze the situation due to the lack of reliable information and the speed in which everything was moving, and knowing how the current models are primed to overemphasize security issues (and be overly dramatic in general) it would not surprise me if they defaulted to confirming people's worst fears: "The exploit has been thoroughly corroborated. This isn’t just a bug. It’s a genuinely profound vulnerability woven into a load-bearing part of the whole internet."
Combine this with a pre-existing fear that AI actually would be able to exploit something like this in a catastropic manner, and a reasonable better-safe-than-sorry stance, and the reports of widespread severe exploits become very believable even for experts.
Actually analyzing the situation and the real impact still required human effort and thoughtful analysis, but such an approach moves with a glacial speed in comparison.
pegasus | 21 hours ago
Problem here is that there has been a steady stream of local privilege escalation bugs for linux for the past ... what, 18 months? Combine any of these with any kind of RCE and you're toast. So the danger is real, especially as people are usually lazy in patching LPE bugs on dedicated servers.
mxey | 19 hours ago
You don’t need to escalate to root if PHP already gives you all the access you need, like reading the web app’s database.
Irene | 20 hours ago
yeah, and the PHP maintainers issued that advisory which described it as very high severity and turning off the server as the only way to mitigate before the patch. so everybody did that, because that's the correct action when a language maintainer tells you to.
I hope people don't get a false sense of security that leads them to under-react next time.
I'll be interested to learn more, once there's been full analysis, about whether other security people agree that the maintainers made the right call.
artem | 12 hours ago
It certainly seems like many of the constraints make exploitation harder but not impossible. Ref: https://x.com/dguido/status/2113825891021144306
There are many, many PHP installations that have few users, live on an old VPS or are in a long forgotten Docker container. Those all seem like they are very exploitable.
aster | 19 hours ago
This is obviously a non-trivial bug to actually exploit in the wild, with a lot of variables, but i think it's perfectly reasonable to assume that some bad actor would be able to weaponize this against a small amount of targets at least, which is why i think the response is reasonable (shutting down sites, wait for update).
As for the mass-exploitation scare, it was obviously a bit exaggerated, and this was obvious from the first POCs becoming public, but if no one thought this was serious and just kept their site running someone could have eventually developed a POC for whatever version of software they were running (Wordpress, Nextcloud, etc) and started mass exploiting vulnerable public instances.
tonyarkles | 15 hours ago
That’s actually, for me, one of the more subtle pieces of it:
If it were trivial to exploit, there would likely be a massive armada of shit disturbers pointing their mass DDoS tools at the entire internet and popping every PHP server in existence. That might still happen, but if hitting any particular server is a non-trivial exercise it might not be worth the effort.
On the other hand, because there isn’t visible mass exploitation, I’m speculating that the massive long tail of small services running older versions of PHP will just continue on running unpatched, ready to be exploited in the future in a more targeted campaign. Bob’s wordpress blog that isn’t on anyone’s radar until he posts something that upsets someone and gets some publicity, then boom… popped.
Maybe this ends up being one of the few times when things like CloudFlare and their WAF stuff ends up actually being a good thing? Hard to say for sure, but for me personally the fact that this isn’t mass-automated-exploitable suggests that the door’s going to remain open for a long long time and will probably be talked about in security circles like MS08-067 (literally the only Microsoft security advisory that I know by memory)
https://learn.microsoft.com/en-us/security-updates/securitybulletins/2008/ms08-067
ibookstein | 11 hours ago
Bad ol' Conficker, forever clanking around on some WinXP machines around the world... I remember having to manually configure that service to not restart-pc-on-crash before applying a patch. Amazing staying power.
mxey | 21 hours ago
Has there been any compromise resulting from this yet?
creesch | 21 hours ago
Also unclear, what I did see is all the usual suspects (cloudflare, aws) claiming to seen attack attempts. I am not sure I have seen any reports about websites being compromised.
kwas | 20 hours ago
The Doylist answer is that no one who would be compromised from this, especially after it went fully public, would have capacity to know that they got compromised.
Irene | 20 hours ago
I apologize for nitpicking, but I don't understand in what sense this is Doylist. reality doesn't have a writer whose reasoning we are party to and can analyze? is it some sort of metaphor?
kwas | 20 hours ago
Yeah, I was aiming for a metaphor, might have misused it. Maybe a rephrasing will be clearer: as a corollary, any organisation running PHP that didn't patch or take down the site within few hours of the public patches being available is perhaps best considered to be unable to monitor their own systems too.
Irene | 20 hours ago
oh, fair enough. thanks for explaining.
Gaelan | 12 hours ago
Speaking for myself: I tried to shut down a site I (barely) maintain before the patch was released. Unfortunately I bungled it (I stopped but did not mask the Apache service, but the machine rebooted for some reason), and the site came back up before the patch dropped, and remained up until I got home from work.
That machine is showing no obvious signs of compromise (eg crypto miners eating all the CPU).
I doubt I'm the only person in my position, who knows enough to recognize a compromised machine, but for whatever reason failed to cleanly handle the patch.
rsalmond | 21 hours ago
Given the need to run consecutive requests to the same process, wouldn't that make an application that uses sticky sessions easier to exploit?
mxey | 19 hours ago
Sticky sessions will route you to the same server but not the same process. But yes that would make it easier.