These liars impersonate Googlebot in hopes that it will help them bypass bot controls
I have ended up impersonating Googlebot on my phone for, in part, this reason on some websites.
I keep JavaScript disabled by default, since I'm using a Moto X4 that is sluggish at the best of times. If I load a Discourse forum page, it "helpfully" informs me "HTML content omitted because you are logged in or using a modern mobile device". I initially used the curl user-agent to get around that (the HTML version that gets me, to be clear, is perfectly usable and much snappier than the default Discourse experience), but I found some sites would severy rate limit me (as in, I'd get tempblocked just reading threads normally). I haven't encountered similar rate limits if I use the Googlebot user-agent.
I also claim to be Googlebot to YouTube, so that they at last include the video title in the <title> tag on video pages; with the real Firefox mobile user agent you get zero useful information if you don't allow JavaScript to run.
Bot users seems like a good use case for mutual TLS. Should be very doable for bot operators to issue valid certificates to their bots, and then the site operators can allow bot-clients based on domains.
As mentioned in TFA most legit crawlers do publish their crawler subnets. There have been attempts to standardize request signing for this as well, but as far as I know it hasn't taken off.
mTLS would be quite the lift I think. The current most common mechanism - validating reverse and forward DNS - is effective, simple, and has clear caching mechanics.
TLS termination is handled in all sorts of odd places (CDNs, load balancers, reverse proxies) where you'd have to deploy all the bot certificates you care about - or do some kind of JIT fetching of them, which will necessarily be slower than equivalent DNS operations. They won't normally pass down mTLS certificate info either, whereas there are established mechanisms of passing along remote IPs.
Most of these bots are bad actors, and it’s unfortunate that they’re poisoning the well this way.
However, the fact that impersonating googlebot gets you these extra privileges is also a bad feature of the web ecosystem: https://danluu.com/googlebot-monopoly/.
Offering this without a solution, as I don’t have one.
Most crawlers will officially state they parse robots.txt as if they are google bot. It's the only way you can actually get any useful data. You have to block them by name if you really want to
Back many years ago we hosted an army of WordPress websites and I built a system to block malicious bots. It was a mix of patterns that had to match first before bots could do anything else besides GET and HEAD. Otherwise they would only receive cached content.
I browsed hours through logs and there was really an interesting mix of user agents and behavior:
Googlebots not coming from Google’s ASN was one of them
Internet Explorer 4.0 on Windows 2000 speaking TLS 1.2 was also interesting and quite often for some reason
In that same category as before: old user agents coming from cloud providers via IPv6
On a slightly unrelated note, I remember when I was at one of my first jobs back in the 2000s and the tech lead decided at some point to just entirely ban the Chinese IP space because we were constantly fighting DDoS attacks originating from there. It did in fact help a lot, but there obviously was a deeper issue at play: our systems was still vulnerable, irrespective of where requests originated...
nortti | 7 hours ago
I have ended up impersonating Googlebot on my phone for, in part, this reason on some websites.
I keep JavaScript disabled by default, since I'm using a Moto X4 that is sluggish at the best of times. If I load a Discourse forum page, it "helpfully" informs me "HTML content omitted because you are logged in or using a modern mobile device". I initially used the curl user-agent to get around that (the HTML version that gets me, to be clear, is perfectly usable and much snappier than the default Discourse experience), but I found some sites would severy rate limit me (as in, I'd get tempblocked just reading threads normally). I haven't encountered similar rate limits if I use the Googlebot user-agent.
I also claim to be Googlebot to YouTube, so that they at last include the video title in the <title> tag on video pages; with the real Firefox mobile user agent you get zero useful information if you don't allow JavaScript to run.
adrien | 7 hours ago
Try links'. It's treated nicely by a surprising number of websites.
JulianSildenLanglo | 10 hours ago
Bot users seems like a good use case for mutual TLS. Should be very doable for bot operators to issue valid certificates to their bots, and then the site operators can allow bot-clients based on domains.
marginalia | 9 hours ago
As mentioned in TFA most legit crawlers do publish their crawler subnets. There have been attempts to standardize request signing for this as well, but as far as I know it hasn't taken off.
mjec | 5 hours ago
mTLS would be quite the lift I think. The current most common mechanism - validating reverse and forward DNS - is effective, simple, and has clear caching mechanics.
TLS termination is handled in all sorts of odd places (CDNs, load balancers, reverse proxies) where you'd have to deploy all the bot certificates you care about - or do some kind of JIT fetching of them, which will necessarily be slower than equivalent DNS operations. They won't normally pass down mTLS certificate info either, whereas there are established mechanisms of passing along remote IPs.
mandeep | 4 hours ago
mTLS is a bit heavy. Web Bot Auth seems like the more lightweight way to do this, but it's seen poor adoption so far.
hyperpape | 7 hours ago
Most of these bots are bad actors, and it’s unfortunate that they’re poisoning the well this way.
However, the fact that impersonating googlebot gets you these extra privileges is also a bad feature of the web ecosystem: https://danluu.com/googlebot-monopoly/.
Offering this without a solution, as I don’t have one.
mousetail | 6 hours ago
Most crawlers will officially state they parse robots.txt as if they are google bot. It's the only way you can actually get any useful data. You have to block them by name if you really want to
gcupc | 10 hours ago
Just block all traffic claiming to be googlebot. Problem solved.
Jan200101 | 8 hours ago
Some people want their website indexed so this isn't a universal solution.
gcupc | 5 hours ago
Unless you're selling a product, having your website indexed these days isn't a clear-cut benefit, either. We're in the Dark Forest era of the web.
cr | 6 hours ago
Back many years ago we hosted an army of WordPress websites and I built a system to block malicious bots. It was a mix of patterns that had to match first before bots could do anything else besides GET and HEAD. Otherwise they would only receive cached content.
I browsed hours through logs and there was really an interesting mix of user agents and behavior:
zg | 6 hours ago
On a slightly unrelated note, I remember when I was at one of my first jobs back in the 2000s and the tech lead decided at some point to just entirely ban the Chinese IP space because we were constantly fighting DDoS attacks originating from there. It did in fact help a lot, but there obviously was a deeper issue at play: our systems was still vulnerable, irrespective of where requests originated...
seabre | 4 hours ago
Most of the probes are for checking for paths leaking credentials (think .env and etc).
Take a sample of log traffic for about a week and use that to inform something like rack-attack for targeting traffic to ban.