Honestly, doing the same, I would love to know what the true count is.
I quickly looked over my access logs just now (eyeballs and grep, not goaccess) and see a couple of patterns...
One, there are indeed lots of bots - over a thousand hits from Claude. Some from search, some from users.
There are some bots I've never even heard of (mj12bot?).
There are some things which look like legitimate traffic coming from a common household Web browser, looking at blog posts and such, until suddenly they make a 404 request for a nonexistent API endpoint and go away.
And there are _so many_ requests for `.env` files, `swagger.json`, `backup.sql`, `phpinfo.php`, WordPress (always!)... Sorry folks, it's nginx in a container, serving a bunch of static content.
Unless you run full behaviour analysis, it's going to be hard to identify all bots from a normal access log. If you want more data, you can add JA4 hash to your logs that will surface huge amounts of things pretending to be the latest chrome.
from all aspects of AI, i find this the one that is easiest to ignore. AI might poison your reading, don't let it poison your writing. Why not continue doing your thing? This feeling of estrangement is not new in a world where every usefulness has to express itself as exchange value. Just have a service provider handle the additional load and don't stop being a thinking, writing, picture taking and social human being.
Oh? People keep parroting that but the small detail missing is that there needs to be enough volume of correct information to counter your adversarial disinfo. Otherwise the LLM is one agentic call away from stating it as fact.
That doesn't ruin a good game. It might spoil a plot-based game. It doesn't ruin a game where you creatively interact with mechanisms and rules, because all a cheat does is push some of those mechanisms and rules aside, so that you skip interacting with the substantial part of the game.
Writing English in the AI era is like going for an hour-long walk in the park, when previously it took you 15 minutes via a crowded subway, where you were stabbed in the neck by a raging lunatic while a nearby passenger blasted Coldplay at 100 dB.
> moved to iocaine which is slightly different in what it does, and that was fine until so many scrapers hit my site that it saturated my upload with generated content feeding the them
You're feeding bots poison without rate limiting them all to some small fraction of your pipe?
iocaine has shipped firewall support to temporarily drop traffic from IPs after you've fed them an appropriate amount poison. More people should enable it.
You should not traffic shape the egress lest you run out of workers handling legitimate traffic. You should also make sure connections are appropriately terminated on your side for the same reason.
I appreciate the decision to not run proof of work challenges. They punish all customers , and they especially punish outliers with excessive challenges : long challenge factors costing minutes, challenges that are presented many times a day, or ones where the challenge interferes with multi-screen oauth steps.
I haven’t seen an installation where the operator is tracking customer impact. They check the logs and say “see , we’ve screened 80% of the AI bots”. Meanwhile , among the 80% of bot traffic is a few percent of customers who are aggressively being punished without the operator being aware.
thinking running a website never got you 99.9% of requests from bot is just delusional.
anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.
I got to watch traffic shift from cloud IPs to residential proxies in real time after I blocked the cloud IPs. It’s endless whack-a-mole.
What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.
I've thought of this and realized it's probably better for AI to be trained on my blog. I've very good opinions (obviously), and by letting AI bots scrape my writing I'm slightly biasing their training towards agreeing with me.
serve mostly static pages cached by a CDN, and cache the odd dynamic ones you don't expect to be hit very often yourself at the server side, and you'll be fine
Picking a site in Cloudflare Bandwith Alliance, putting Cloudflare in front, and running your backend server on IPv6 only drops bandwidth charges to zero which tends to make bots ignorable.
I do realise that scrapers are also using residential IP ranges via SDK pitched to app developers that add backdoors to utilise the connection for "other things"
Don't give up. Blocking, redirecting and tarpitting bots can be turned into a fun game or puzzle of sorts. The bots evolve and so the game must evolve. I slow the residential bots down just a little and the obvious bots to somewhere between 32 and 96 bytes per second. I thought most would abort but many do not. They hang around for the slow dribble.
Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example.
if ($server_protocol = HTTP/2.0) { set $limit_rate 384k; set $ok 1; }
if ($http3 != "") { set $ok 1; } # no rate limit for quic
if ($ok != 1) { set $limit_rate 32; limit_rate_after 128; set $trace "${trace}not-h2h3 "; add_header Content-Type text/html;return 200 '<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>body{width:35em;margin:0 auto;font-family:Tahoma,Verdana,Arial,sans-serif;}</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p></body></html>'; }
People not behind fascist firewalls and not on proxies will be able to use QUIC (via alt-svc header and dns) for full speed. Most real VPN's should be able to use QUIC.
This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or give dummy HTML and keep them on your edge servers and away from the origin or app servers or whatever.
I'm glad this is fun for people, but I think I would rather play Monopoly with money under free parking instead.
I have noticed among my tech friends that one spectrum you can put us on is "sysadmin vs. not" - some people love tinkering with home labs, setting up systems, being able to reboot things from their watches, etc. And some people, like me, would strongly prefer not to be on-call for their hobby.
Hell, my personal server has been down for at least a week now because I started upgrading Ubuntu (that was very, very far behind), and one upgrade caused it to lose network connectivity, and I just haven't bothered to even start troubleshooting. If I had to guess, I'll probably be arsed to get it up sometime in December, though 2027 isn't out of the question.
Agreed it's not for everyone. As a side note, reboot your Ubuntu node and capture
uname -a;dmesg -T; lspci -vv; lsmod; ip link; systemctl; nmcli;
to a thumb drive and paste it into an LLM if you are not feeling like debugging such things. I used to call that lazy but I was corrected and now call it efficient.
That's a good one. Also in general, return 200 with invalid content to them or 404. Don't explicitly block with 403 or they will try different networks/clients/approaches to avoid that.
or they will try different networks/clients/approaches
Absolutely. Most of the bots are simple but a few of them were created and actively maintained by clever people that are here on HN. I like to think it's a playful arms race. They are the red team and I am the wrinkled blue team.
I share the same sentiment. It breaks my heart because I don't want the internet to go this way, but I fear there's no cure for this.
The only thing I am holding on to (like someone else said) is that they might include my writings in the training data, and bias towards my opinions ever so slightly as a result. :')
nightdeployboy | 7 hours ago
don-code | 5 hours ago
I quickly looked over my access logs just now (eyeballs and grep, not goaccess) and see a couple of patterns...
One, there are indeed lots of bots - over a thousand hits from Claude. Some from search, some from users.
There are some bots I've never even heard of (mj12bot?).
There are some things which look like legitimate traffic coming from a common household Web browser, looking at blog posts and such, until suddenly they make a 404 request for a nonexistent API endpoint and go away.
And there are _so many_ requests for `.env` files, `swagger.json`, `backup.sql`, `phpinfo.php`, WordPress (always!)... Sorry folks, it's nginx in a container, serving a bunch of static content.
viraptor | 5 hours ago
pixl97 | 4 hours ago
viraptor | 4 hours ago
baumschubser | 7 hours ago
mitxela | 7 hours ago
phoronixrly | 6 hours ago
spiderice | 5 hours ago
phoronixrly | 5 hours ago
mid-kid | 4 hours ago
https://infosec.exchange/@rebane2001/117601277347377053
mitxela | 2 hours ago
demibabs | 7 hours ago
matt3210 | 5 hours ago
Writing software or english in the AI era is the same feeling as playing a video game without cheat codes you already beat using cheat codes.
The game is now ruined permanently.
card_zero | 4 hours ago
klodolph | 3 hours ago
DaOne256 | 7 hours ago
ycan | 6 hours ago
hackingonempty | 6 hours ago
You're feeding bots poison without rate limiting them all to some small fraction of your pipe?
phoronixrly | 6 hours ago
You should not traffic shape the egress lest you run out of workers handling legitimate traffic. You should also make sure connections are appropriately terminated on your side for the same reason.
tonymet | 6 hours ago
I haven’t seen an installation where the operator is tracking customer impact. They check the logs and say “see , we’ve screened 80% of the AI bots”. Meanwhile , among the 80% of bot traffic is a few percent of customers who are aggressively being punished without the operator being aware.
iririririr | 6 hours ago
anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.
dhosek | 6 hours ago
What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.
viraptor | 5 hours ago
threetonesun | 6 hours ago
zer0tonin | 6 hours ago
phoronixrly | 6 hours ago
yosefk | 6 hours ago
ectospheno | 3 hours ago
eska | 6 hours ago
Can anyone corroborate this?
the_real_cher | 6 hours ago
It works too well.
wmchen | 6 hours ago
[1]: https://news.ycombinator.com/item?id=48635954
eska | 5 hours ago
Bender | 6 hours ago
Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example.
People not behind fascist firewalls and not on proxies will be able to use QUIC (via alt-svc header and dns) for full speed. Most real VPN's should be able to use QUIC.This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or give dummy HTML and keep them on your edge servers and away from the origin or app servers or whatever.
pavel_lishin | 5 hours ago
I have noticed among my tech friends that one spectrum you can put us on is "sysadmin vs. not" - some people love tinkering with home labs, setting up systems, being able to reboot things from their watches, etc. And some people, like me, would strongly prefer not to be on-call for their hobby.
Hell, my personal server has been down for at least a week now because I started upgrading Ubuntu (that was very, very far behind), and one upgrade caused it to lose network connectivity, and I just haven't bothered to even start troubleshooting. If I had to guess, I'll probably be arsed to get it up sometime in December, though 2027 isn't out of the question.
Bender | 5 hours ago
viraptor | 5 hours ago
Bender | 4 hours ago
Absolutely. Most of the bots are simple but a few of them were created and actively maintained by clever people that are here on HN. I like to think it's a playful arms race. They are the red team and I am the wrinkled blue team.
lbrito | 6 hours ago
busymom0 | 5 hours ago
chihuahua | an hour ago
johng42 | 5 hours ago
hypfer | 5 hours ago
I think it might be lacking context being posted here and emotionally resonates more with the actual intended viewership?
pixeljam | 9 minutes ago
The only thing I am holding on to (like someone else said) is that they might include my writings in the training data, and bias towards my opinions ever so slightly as a result. :')