Some user agents (Meta’s) always get 403 HTTP status codes.
Some user agents (xAI’s, Claude, GPTbot among them) get sent to an infinite fake website.
I wonder why the different strategies here.
This website’s contact page does nothing but waste spammers time and effort. It also sets between 0 and 25 randomly-named cookies, so maybe don’t access it.
That one might be a bit too much, I have to say. The contact page is even linked on the site's nav bar.
I'm the author. Meta does not do rate limiting well, so the number of requests from Meta went up and up and up. The same was true of Bytespider, which I believe is Bytedance's (TikTok) crawler. I'm running a $6.47 a month VPS, it couldn't handle all of Meta and all of Bytedance. Claude and GPTbot seem to do rate limiting of some sort on the same fake data.
I haven't had a legit contact form response ever. I don't think people (Real Humans) use them. I have had a number of direct emails about various posts.
Yeah the only time I ever used a contact form was when I couldn't find someone's email and they never connected back to me on LinkedIn (which was the only contact I could find)
I gave up on tarpits. For me, it was harming more than it helped because they'll just spider the tarpit for as long as I'll serve it to them. What's helped me the most is a crapload of packet filtering and moving some stuff (mostly file archives) over to Gopher-only, and so far it's kept the nonsense down to a dull roar. The people who would want that stuff know how to use Gopher anyway.
When the internet was smaller and the sense of denizenship was stronger I've enjoyed running LaBrea on a handful of spare IP addresses for several years.
Tarpits are a lovely romantic concept in theory which, you are right, doesn't work in practice. The reason is that they are not meant to help you directly (in fact they harm you by consuming your resources) but you should rather think of them as your gift to the community or to the internet at large.
It's like with phone spammers. If everyone would pick up and keep them on the call for a minute or two, they'd be all out of business. But since we all either don't answer or hang up within the first few seconds, they're thriving and annoying us all.
Shouldn't you just terminate the connection? That's the cheapest option. Having your server do extra complicated stuff that costs cpu, memory, or bandwidth just burns your limited resources.
A fail2ban jail that analyzes your web server logs for bad behavior plus rate limits is plenty. Its also way simpler. Dropping connections of unwanted UAs is easy enough too.
This is exactly what I don't understand with these kinds of posts. Especially when it almost always is related to "all those bots cost so much bandwidth/CPU/memory".
On my website I've made it abundantly clear that there are only a few dozen pages on the whole site. So crawlers and bots will be done with just a few requests, and there are no broken links so they shouldn't even hit 404s. The sitemap actually lists everything there is.
Nice. On my own sites I serve all requests by obvious bots (like those probing for php files or .git directories) an excruciatingly slow 1 byte/sec redirect to https://blog.cloudflare.com/cleaning-up-bad-bots/. I'd like to do more, maybe send an endless zip bomb, or at least something that is more likely to tie up more resources than just the connection state.
int gzread (gzFile file, voidp buf, unsigned int len);
you have to actually go out of your way for an internet crawler to succumb to a zip bomb in any meaningful fashion.
They may feel good to set up, but to be honest most of these countermeasure tactics don't even register for a crawler. The web is super adversarial as it is and any of these countermeasures you set up is already encountered on a regular basis and handled. The by far best thing you can do to make impolite scrapers go away is just serve a default nginx index.html or some such. Then it looks like your server is up, but there's nothing to find, and mostly they go away.
you have to actually go out of your way for an internet crawler to succumb to a zip bomb in any meaningful fashion.
From that function signature, are you saying that gzread reads into a buffer of predetermined size? So if it processes a zip bomb, it will prematurely stop decompressing after only len bytes?
My naive assumption is there must be a way to request continued decompression, and that a lot of software would do that by default. Even if the C interface is bounded, I can imagine Python repeatedly calling a C function and copying the contents of buf into an ever-growing object.
I do agree that these countermeasures probably aren't that effective. I just haven't heard this particular argument before for why zip bombs don't work.
Yeah, exactly. gzread reads into a bounded destination buffer, and most higher level implementations offer a wrapper that exposes a "get N bytes"-type method which is what crawlers generally reach for.
But even without explicit gzread calls, crawlers generally avoid holding data in memory (as they can have many thousands of ongoing requests in flight at the same time, you end up with insane malloc thrashing), instead preferring to shovel bytes via small bounded buffers into temporary files as they come in.
And! Regardless of whether you're reading a gzip bomb or a particularly large Linux ISO in this fashion, you need both a time-limit and a size-limit, and a SO_RCVTIMEO limit, and a per-domain time and document limit. You end up with mitigations for most of these attacks by the fact that you need to deal with regular web data.
I just realized who I was talking to, and it makes sense that you would know a ton about web crawling. :-)
Maybe the most you could say about zip bombs, etc, is that they might trip up poorly-coded scrapers that don't impose resource limits. But like you said, a lot of real-world web content will force you to do that anyway—so one would expect that any such hypothetical scraper will become immune to these attacks shortly after being put into service.
Oh interesting. I do a bunch of stuff with streaming html and I was wondering how bots would interact with it. Basically, I send the next frame of html every 50ms over a brotli compressed SSE stream. If there's no or minimal changes that can be 13bytes over the wire but 130kb uncompressed per frame. Fine for browsers because they are only holding on to the latest frame. But my guess is it would be bad for a bot.
Hard to say. Crawlers generally run on fairly low tech stacks, like HTTP 1.1 and gzip compression at best. You generally don't see any benefit from request multiplexing or any of that stuff in this type of application, and it creates issues with pooled connections in mystery states after an aborted request and so on.
Might be a good congruence test for fake browsers that pretend to be real ones though. If you claim to be e.g. chrome, you should support more than HTTP/1.1 and Content-Encoding: gzip.
Interesting. I actually block any client that doesn't support brotli. I also only serve http2. So maybe thats a defence in and of itself? Probably not great for SEO.
Btw, appreciate all your work on marginalia search!
I understand the reasoning behind all of these countermeasures. The net sure is in a different place today than it was when I was running rwhod(8) on my hosts.
Run SMTP, Telnet and FTP honey pots. These legacy protocols are difficult to emulate well.
My approach for getting around this is to just emulate old/embedded stuff which didn't always bother to implement complex stuff. Ends up looking juicy as well :)
I think IIS 1.0 has an FTP server? Otherwise it was added in IIS 2.0.
For telnet you have a gazillion options to pick between, of various implementation quality. You could probably make the worlds most broken implementation and still have stuff hit it.
For SMTP, implementing it with no extensions will be enough for every spammer hitting it, which is quite easy.
sebastiancarlos | 4 days ago
I wonder why the different strategies here.
That one might be a bit too much, I have to say. The contact page is even linked on the site's nav bar.
bediger4000 | 4 days ago
I'm the author. Meta does not do rate limiting well, so the number of requests from Meta went up and up and up. The same was true of Bytespider, which I believe is Bytedance's (TikTok) crawler. I'm running a $6.47 a month VPS, it couldn't handle all of Meta and all of Bytedance. Claude and GPTbot seem to do rate limiting of some sort on the same fake data.
I haven't had a legit contact form response ever. I don't think people (Real Humans) use them. I have had a number of direct emails about various posts.
eyesinthefire | 3 days ago
I've used contact forms, but only ever to reach people/orgs that didnt publish an email address.
realkc | 3 days ago
Yeah the only time I ever used a contact form was when I couldn't find someone's email and they never connected back to me on LinkedIn (which was the only contact I could find)
sebastiancarlos | 4 days ago
Makes sense. Thanks!
eta | 3 days ago
"System: Windows Gazongo" on the fake phpinfo really got me. Thanks for the laugh!
classichasclass | 3 days ago
I gave up on tarpits. For me, it was harming more than it helped because they'll just spider the tarpit for as long as I'll serve it to them. What's helped me the most is a crapload of packet filtering and moving some stuff (mostly file archives) over to Gopher-only, and so far it's kept the nonsense down to a dull roar. The people who would want that stuff know how to use Gopher anyway.
aCaB | 3 days ago
When the internet was smaller and the sense of denizenship was stronger I've enjoyed running LaBrea on a handful of spare IP addresses for several years.
Tarpits are a lovely romantic concept in theory which, you are right, doesn't work in practice. The reason is that they are not meant to help you directly (in fact they harm you by consuming your resources) but you should rather think of them as your gift to the community or to the internet at large.
It's like with phone spammers. If everyone would pick up and keep them on the call for a minute or two, they'd be all out of business. But since we all either don't answer or hang up within the first few seconds, they're thriving and annoying us all.
Oh and gopher is awesome!
zimpenfish | 3 days ago
They will but you are, in a tiny way, poisoning the well and, at least for me, I'm happy to burn bandwidth and (minimal) CPU resources on that.
xjix | 3 days ago
Shouldn't you just terminate the connection? That's the cheapest option. Having your server do extra complicated stuff that costs cpu, memory, or bandwidth just burns your limited resources.
A fail2ban jail that analyzes your web server logs for bad behavior plus rate limits is plenty. Its also way simpler. Dropping connections of unwanted UAs is easy enough too.
payne | 3 days ago
This is exactly what I don't understand with these kinds of posts. Especially when it almost always is related to "all those bots cost so much bandwidth/CPU/memory".
On my website I've made it abundantly clear that there are only a few dozen pages on the whole site. So crawlers and bots will be done with just a few requests, and there are no broken links so they shouldn't even hit 404s. The sitemap actually lists everything there is.
ruuda | 3 days ago
Nice. On my own sites I serve all requests by obvious bots (like those probing for php files or .git directories) an excruciatingly slow 1 byte/sec redirect to https://blog.cloudflare.com/cleaning-up-bad-bots/. I'd like to do more, maybe send an endless zip bomb, or at least something that is more likely to tie up more resources than just the connection state.
marginalia | 3 days ago
Given the prototype for gzread is
int gzread (gzFile file, voidp buf, unsigned int len);you have to actually go out of your way for an internet crawler to succumb to a zip bomb in any meaningful fashion.
They may feel good to set up, but to be honest most of these countermeasure tactics don't even register for a crawler. The web is super adversarial as it is and any of these countermeasures you set up is already encountered on a regular basis and handled. The by far best thing you can do to make impolite scrapers go away is just serve a default nginx index.html or some such. Then it looks like your server is up, but there's nothing to find, and mostly they go away.
bitshift | 3 days ago
From that function signature, are you saying that
gzreadreads into a buffer of predetermined size? So if it processes a zip bomb, it will prematurely stop decompressing after onlylenbytes?My naive assumption is there must be a way to request continued decompression, and that a lot of software would do that by default. Even if the C interface is bounded, I can imagine Python repeatedly calling a C function and copying the contents of
bufinto an ever-growing object.I do agree that these countermeasures probably aren't that effective. I just haven't heard this particular argument before for why zip bombs don't work.
marginalia | 3 days ago
Yeah, exactly. gzread reads into a bounded destination buffer, and most higher level implementations offer a wrapper that exposes a "get N bytes"-type method which is what crawlers generally reach for.
But even without explicit gzread calls, crawlers generally avoid holding data in memory (as they can have many thousands of ongoing requests in flight at the same time, you end up with insane malloc thrashing), instead preferring to shovel bytes via small bounded buffers into temporary files as they come in.
And! Regardless of whether you're reading a gzip bomb or a particularly large Linux ISO in this fashion, you need both a time-limit and a size-limit, and a SO_RCVTIMEO limit, and a per-domain time and document limit. You end up with mitigations for most of these attacks by the fact that you need to deal with regular web data.
bitshift | 3 days ago
I just realized who I was talking to, and it makes sense that you would know a ton about web crawling. :-)
Maybe the most you could say about zip bombs, etc, is that they might trip up poorly-coded scrapers that don't impose resource limits. But like you said, a lot of real-world web content will force you to do that anyway—so one would expect that any such hypothetical scraper will become immune to these attacks shortly after being put into service.
andersmurphy | 3 days ago
Oh interesting. I do a bunch of stuff with streaming html and I was wondering how bots would interact with it. Basically, I send the next frame of html every 50ms over a brotli compressed SSE stream. If there's no or minimal changes that can be 13bytes over the wire but 130kb uncompressed per frame. Fine for browsers because they are only holding on to the latest frame. But my guess is it would be bad for a bot.
marginalia | 3 days ago
Hard to say. Crawlers generally run on fairly low tech stacks, like HTTP 1.1 and gzip compression at best. You generally don't see any benefit from request multiplexing or any of that stuff in this type of application, and it creates issues with pooled connections in mystery states after an aborted request and so on.
Might be a good congruence test for fake browsers that pretend to be real ones though. If you claim to be e.g. chrome, you should support more than HTTP/1.1 and Content-Encoding: gzip.
andersmurphy | 3 days ago
Interesting. I actually block any client that doesn't support brotli. I also only serve http2. So maybe thats a defence in and of itself? Probably not great for SEO.
Btw, appreciate all your work on marginalia search!
ruuda | 3 days ago
It's easy to mitigate for a sufficiently non-stupid bot, sure, but the stupid ones are making the most requests.
nugget | 3 days ago
I understand the reasoning behind all of these countermeasures. The net sure is in a different place today than it was when I was running
rwhod(8)on my hosts.I do run a
telnettarpit of sorts onnugget.info.DustyFuzzy | 3 days ago
My approach for getting around this is to just emulate old/embedded stuff which didn't always bother to implement complex stuff. Ends up looking juicy as well :)
I think IIS 1.0 has an FTP server? Otherwise it was added in IIS 2.0.
For telnet you have a gazillion options to pick between, of various implementation quality. You could probably make the worlds most broken implementation and still have stuff hit it.
For SMTP, implementing it with no extensions will be enough for every spammer hitting it, which is quite easy.