I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.
But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.
Yes, this is very difficult to solve for sites with many URL variations, like git repos and heavily threaded forums. Your cache is always full, but the hit ratio is abysmal.
no its only difficult for you. and only at this moment... any minute now you will see the way. btw, the OP is just saying "its OK for now". And the OP is just telling us: this is what has been happening... maybe its difficult for OP also, but they didnt say that. they just said their current setup cant handle it. old tech...have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
using the current solution and a cache pattern doesnt solve it, but it is an "affordable" solvable problem. but first we need to ignore OP said its ok for now...have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
Most of those pages are going to be identical if not for some very minor differences. You can make it much much cheaper to put together these pages as opposed to the expensive operations it has to do right now.
i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, but instead of caching instead with maybe compiled streaming renderer(s)... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended
anyway, I have solved this exact scenario many times in the last 30 years... thanks for the dv.. have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.
> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
I guess GitHub is in a similar bunch of sources, it should be also more efficient to crawl by cloning.
Anyway, isn’t it the whole sales pitch that it generates tailored solutions fast?
This tick about "if AI smart how come crawler dumb" is in most complaints I've read about AI crawlers and I've started to find it pretty annoying. The crawlers might be written using AI but they're evidently not actually running AI inference over the pages they get back--besides being able to tell this from the behavior, if this is pretraining input, that's enormous scale, so it'd mean a large increase in effective training cost. Naively assume inference costs are equal to pretraining costs (probably not true but maybe right order-of-magnitude) and it's a doubling.
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
There's nothing "denialist" about recognizing the utter stupidity of systems that are being mislabeled as "AI". You judge a tool by its results, and the results have been very poor indeed. The only heads in the sand are those whose owners continually refuse to recognize the proofs before their very eyes that there's zero intelligence here.
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
>AI is certainly very capable of being dumb (as are humans).
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
Yes, but the article (not to call you out - I just think it's a very important point!) points out that this type of throttling would not be effective:
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
Without something like cookies (which are almost certainly tossed after the IP is rotated) or some other persistent identifier, you are stuck have to apply mitigations that scale with the load you're encountering, which means longer challenges for everyone or degraded functionality, like removing some of the fancier cgit features.
I've had a fair bit of success with increasing the bot mitigation based on a global rate limit. During periods of high request rates, I throw progressively more hurdles at the bots, and during periods of low request rates I disable them all.
Because they're crawling a billion webpages, only a tiny fraction of which can be git cloned, and configuring a special case just for that tiny fraction isn't worth the effort (of the crawlers).
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking.
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
> it's not actually obvious who's running these bots
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
This is bad enough that I'm going to stop serving cgit. I've been doing cvsweb, then subversion, then cgit over my home server for many many years and for the first time ever this is annoying my own bw usage.
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
i wonder what they're all up to. i imagine some are scraping datasets for pre-training, others are probably real-time scrapers looking for security bugs, even more still are agents working on coding tasks and looking at the kernel. also interesting to think about solutions: does everything need to be optimized now for weird access patterns that proliferated ai creates? do the ais need to have behavior trained in to be better netizens? is this the end of anonymous browsing and the beginning of an era where one has to attach an identity to all requests? or the end of community hosted free information services more broadly?
High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls
Unfortunately we're apt to run into some kind of Jeavons Paradox where the hardware gets so much faster in those few years will be able to slurp massive amounts of data cheaply so the problem never really ends.
It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essentially DDoS:ing a website.
And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare.
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?
Has anyone considered having Anubis perform more valuable hashing?
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
> High Anubis difficulty is annoying the hell out of me for several sites.
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
the brutal truth is that if the website operator simply disabled Anubis, your page load would likely take more than 30s.
when a system was designed for 100 req/s and bots hit it with 5000 req/s, nobody entering that queue is having a good time. Anubis is the trade those operators make just to ensure your request gets serviced at all.
30s load time is already a sign that the Anubis approach is breaking down. if there's nothing else ready by the next order-of-magnitude increase in crawler load, those sites quite likely will just disappear from the public internet. hate Anubis all you want: for most of us, the realistic alternative is strictly worse.
I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
Are the AI companies worried about anything but their valuations?
I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need to fund any risk mitigations, security coordination, and other safety work. If these things are not imposed on the market, the most ruthless player will create the biggest mess they can get away with.
I don’t see a better way to do this, right now, than to allow Strips, or CF, to create some way to charge the crawlers. Imagine I have some really expensive content, I can put it behind the paywall, or subscription model, but that’s very specific to the human consumer. What if I do want to search engine or content aggregator to consume some of the content? It would be quite hard for me to sign an exclusive agreement with OpenAI, like what Reddit or NYT are doing. It might be quite beneficial to create some infrastructure to let bots automatically to pay for access. It might even be crypto based, I don’t think it really matters. It just has to be easy for me to integrate it, and the bots need to find it easy to use.
If we all start doing this, I think this could be of benefit to all content creators, such as musicians, movie makers, game developers, and even independent journalists.
We used to pay for software and things, but what has happened? Walled gardens and production monopolies are taking over. This has largely reduced the quality of all the content we consume.
I think that to prove I am not a crawler I probably just need to supply a Safari user agent string. The crawlers on my sites actually self identify for the most part. The malicious ones are banned when they try to post spam.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
Yes. Any number of methods work. From requiring a cookie to requiring CSS to be loaded to a PoW to a JavaScript redirect. Most scraper bots are really dumb. If you do it cookie based, you can also put a poison link on every page that invalidates their cookie and starts returning poison on every page after that, so they're damned if they do and damned if they don't.
I have tried cookie based sessions and I have tried multiple things. I have been battling DDOS and script kiddies longer than CF has been a thing. You’re wrong in that you think I could continue to maintain the site without CF in front of it, I would just shut it down. It’s not a profitable forum, it’s just a hobby.
If I recall by browsing the docs, there's a lot of fine grained client control over the blocking params. Which makes sense since the main product of cf is bot blocking.
So you can definitely block by IP, by fingerprint, both or neither
Cloudflare is also great at playing both sides, and they're trying pretty hard to push for pay-to-crawl because they'll probably get a 30% cut along the way.
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
Because I also use AI, and I actually want the content from the site to show up for me, as well as other AI users.
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
If these were actual legitimate bots using a known user-agent and coming from the same subnet so you could throttle their requests it would be fine.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
All anti-bot measures are inherently flawed, and if you overdo it, your false positives cause real harm by excluding actual people from using your services.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
HoustonArchitecture.com, a site that posts about new development projects in the Houston area, had the exact same thing happen to them. (They also got hacked; don't remember the details.) They now pay the Cloudflare tax.
It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
As the article states, this phenomenon may be documented, but there is no consensus that it describes any practical reality.
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
Normal clones can reuse delta-compressed data the server stored on disk. Shallow clones impose a negative constraint: do not transfer data outside the requested commit depth. Pre-computed delta chains than contain unrequested data become unusable and the server must do delta compression on the fly to satsify the shallow clone.
What I find remarkable is that for at least a decade, i.e., long before LLM scrapers were a thing, GitHub engineers have been reaching out to popular package manager projects, asking them to do away with shallow clones [0] [1]. They basically used the same reasoning as your comment did.
> So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
I'm not saying websites can't demand payments for service, I'm just saying it's bad if it's a necessary fix for "scrapers are destroying the basic social contract of the web".
The internet is already like that, but for some reason the payment only extends as far as the recipient's ISP, not the actual recipient. Most senders pay a flat rate, but their ISP doesn't.
It's a social problem, technical solutions aren't going to work. To solve social problems you need the law to get involved. It should simply be illegal to install malware on someone's TV and use it to hijack their internet connection, and the punishment should be the same as for any theft of service, including imprisonment for people who do it.
You just take a sample of the crawling to whichever police department knows how to deal with DDoS, they subpoena the ISPs involved and trace who's using the proxies?
so true. the article authors wishing crawlers will use git instead is so funny because the crawlers don't care at all. they are scrapping everything with brute force. they don't care about your content or effective alternatives, and one more site driving their real users crazy with Anubis is nothing more than a new blip in their dashboard. the crawler operators will not even look at the url.
Seeing the exact same thing on (somewhat high profile) open data sites I run.
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
Yeah my search engine saw traffic of up to 160 queries per second the other day from some bot that was ostensibly searching for information on Jack Parsons. Just variations on the same query in different permutations of filters and site:-terms.
No I think this is the same case. It seems to just be following local links on the SERP. From a search for jack parsons, you can find hyperlinks to the sorts of requests it's making.
Once detected, don't block them because they'll just change strategies automatically, but you can toy with them, like returning a page full of random numbers instead of real data.
I’d love to, but I’m a bit limited in what I can do from a reputational damage POV. They’re my sites, in that I’m responsible, but they aren’t something where I can return incorrect responses.
WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
I disagree. The kernel finds it effective - 66% of scrapers are turned away directly.
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.
That's still a notable step up in completely and resource investment for the crawler
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.
For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection.
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
Why would they even run a browser engine on the devices they're hosted on?
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
Until you actually do the math and realize that it is not meaningful at all. It’s equivalent to the blogs that have a custom “bot protector” that asks you “what’s 2+2” every time you submit a comment. It might work temporarily as an inconvenience, but nothing more.
> Many of these boxes may not even have the spare CPU power ...
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
If you want to argue that there's excessive enthusiasm about anubis, which is neither a permanent solution nor a solution against all bots run for all purposes (some bot authors are cleverer than others, and you don't have to block everything, just enough to reduce resource use to tolerable levels)... fine.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call.
You're arguing worst-case, and people running anubis are saying it dramatically helps in their experience, because they're not getting too many of the sorts of bots that people like you and Tavis (correctly) argue are theoretically possible. So the argument that it's "bad CS" falls flat.
A better, perhaps the best, argument is that a worse bot mix might appear tomorrow, so anubis should be used as a runway for a better solution, and not as a solution in itself. If only everyone with a website had the desire (or AI tokens) to implement their own better solution. In some cases even desire or a lot of tokens isn't enough. I mean, you think kernel.org folks haven't put some serious thought into it? And yet, as HN argues about it, there's kernel.org running anubis at difficulty 5.
I don't know what you mean by "the best argument". My entire argument is that it's bad CS. I'm not trying to convince people not to run Anubis. If anything, I'm trying to convince Anubis to use better computer science. That would make more people run it! But even that's just a faint subsidiary goal; the real goal, as someone who works in this field and has designed proof-of-work antiabuse systems before, is just to add clarity about what can and can't work and why.
I too would love anubis to use better CS, if available. Perhaps you have some ideas for @xena, instead of just letting anubis continue on the trajectory of iterating its PoW approach?
I had a fairly long conversation with Xe privately about this and didn't feel the need to repeat it here, especially since (as the search box will tell you) I've already talked about this before. But, do go on.
I looked at your comments and there's a bunch saying that Anubis is bad CS but none I could find saying what, exactly, you want Anubis to do instead. You sure are posting a lot about how it's bad, though.
It is interesting, this particular user seems extremely active in this thread. I'm not sure if they have a bone to pick with Anubis in particular or some other reason for disliking the software so much, but to me it does seem overzealous. Regardless of if it's "bad computer science" or what, Anubis works quite well and is easy to customise to target it to specific botnets, as I've done. And this is coming from someone trying to develop her own non-PoW anti-scraper system.
There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints.
I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
Why would they directly use a proxy service? I'd just launder the data scraping through some third party company that I could slough off if it ever turned into a news story. Not that anything would happen to them if they directly used these services anyway.
If you just want to block the bottom of the barrel scrapers you don't need to do any proof of work. It's overkill and wastes user's times for such a purpose.
Yep, all security is about making attack more expensive than defense. If they have a farm of other people's devices, as alleged in the article, then it costs them less to calculate PoW than it costs ordinary users. Ordinary users have to sit there waiting, whereas bots are somebody else's problem.
I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).
Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
They're the exception, not the rule. They get crawled like any other site, but happen to host git repositories. It's not obvious that these are targeted crawls and they likely may just end up in crawling queues a lot generally
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
If you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?)
If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.
The next more advanced version is that URLs are unique to your cookie.
Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.
You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request.
(a cookie url mismatch gets the rate limited version of the page)
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally
2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
I would think that residential proxying would be illegal already, as it is a network intrusion. The trick is chasing down the offenders, proving their actions did harm, and getting them to pay. None of those steps are easy, even if there are laws to assist. Otherwise, spam would be a solved problem.
Is it still an intrusion if the user accepted the shrinkwrap TOS of an app that trades them “free TV” in exchange for allowing that app to operate a proxy (via an “app monetization” SDK) on their network?
Yeah, no one actually agrees to all of the individual terms in EULAs. That’s the first sign that the law will be nearly useless to address any aspect of these problems. It is already one-sided, and that side is not a friend to the consumer or general public.
It isn't illegal, there is no law against "network intrusion" which is a term you just made up, and if it was a real term it probably wouldn't cover this. There are laws against things like "unauthorized access to a protected computer system".
If there was a way of detecting the proxy SDKs, it would be a simple matter of banning the device. Or at least naming and shaming the apps that use them.
I'm sure there are ways of detecting residential proxy clients that are reasonably effective. I'm equally sure that the people in possession of those techniques (thinking CDN providers, big social networks with public links, newsrooms) are unlikely to share their secret sauce in public. Doing that loses a competitive edge in cost-of-hosting, and as soon as you go public the residential proxy makers can adapt.
Good luck enforcing the law against a company whose entire business model is to piggyback residential-proxy-as-a-service over the shitty FNQQOIIWKNURQB brand $60 projectors they flip on Amazon. You call them out, they’ll just say they stopped doing it and go on with business as usual.
This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
I'm assuming they haven't yet sent responses to the bots? Since AI is dumb, you can send errors that tell the bot to git clone rather than crawl. If it's vulnerable to prompt injection, it might listen and do the clone instead and stop trying to solve challenges.
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
I really enjoyed the writing style in this article.
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.
turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set
this is an increasingly common situation. it goes something like:
- i have a free, niche resource
- it becomes too popular
- i make it more efficient
- now it's really popular, and people are "abusing" it
- let's make them proof-of-work
- ... and proof-of-work harder
- but now "legitimate" users can't use it
- ???
the core problem is that the average person uses a mobile device where work is expensive, and the "attackers" use servers where work is cheap. if you require expensive proof-of-work, next comes a cheap-work-as-a-service where inefficient mobile devices pay small amounts of money to get efficient servers to complete their work for them. now everyone has an interest in making their usage efficient, but there's still an obvious inefficiency in the system: why have people pay unknown 3rd parties to burn cpu cycles to reduce costs for a free service, when you could just have people make small payments that cover the service's costs?
which is called l402/x402. micropayments' day has come
What's actually stopping them isn't the PoW, it's the customisation effort. If one site has Anubis nothing scrapes it. If many sites have Anubis they write counter scrapers. Today if you make a slight change to the Anubis algorithm on your site, they'll burn CPU endlessly computing hashes with the original algorithm and submitting wrong ones.
The author of Anubis hates this fact and will ban you if you mention it, so don't. He insists it's the PoW.
Searching around for this, found "make users click the mouse three times"[1] as an anti-bot idea. Generally, makes sense that smaller site owners can make small customizations to existing anti-bot tech and see positive results until they're either (1) deemed valuable enough to receive custom attention or (2) the scrapers include LLM-based anti-antibot methods.
>will ban you if you mention it
Even if mentioned really politely?
[1] would hope anyone trying this makes it accessible to visitors with disabilities
> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
Not sure if you mean the solution, or the problem they were trying to solve.
The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.
Ah yeah I was quoting 20% of their total capacity.
Nevertheless, without knowing what it actually is in cost, it’s impossible to have an objective sense of the severity of this problem.
Is it, “everyone’s mildly annoyed so we can save $10k per year” or is it “everyone’s mildly annoyed so we can save $100k per year?” And has someone attempted to reason about what the $/year cost is of that mild annoyance?
I’ve seen this a lot in my career. Developers coming to conclusions that are ultimately based on what feels expensive or problematic.
Bright Data's business is legal, but they could be subpoenaed to find out which customer is making these requests, but first you would have to prove they were actually involved, because there are many residential proxy providers.
Sometimes the first step of a lawsuit is discovering who you're suing. It's not unusual and there are processes for it. You could bring something like an access log to a court and receive an order for all ISPs involved to unmask the corresponding users.
Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it.
For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.
> You have the power, and should exercise it, to rate limit bad actors
It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
> Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource.
Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.
a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently.
I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.
Cool, so then that invalidates the ASIC problem, right?
My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...
All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)
...That sounds like entropy? As in, the thing computers are bad at (truly random numbers) and /dev/urandom in your kernel already spits out an approximation of?
The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.
Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic.
You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.
It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.
If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust.
Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
A significant amount of the effect is everything outside the proof of work. Not the cost of hashing but the need to run the javascript that submits it.
If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it.
This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers.
That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".
I suspect that's most of the value, and that the deterrent is about software construction and not about the proof-of-work, which I've been saying for the last year is a productive direction for Anubis to head towards. Instead, it's just sort of tormenting normal people, as a kind of performance of anger about AI.
Which: I get! If it's your site, perform all the anger you want.
> There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.
That doesn't matter! Every time they successfully solve the challenge, they're getting the token reward. That's the missing asymmetry. In an anti-spam system, 99.999% of all transactions are unrewarded.
I have a small personal site with an exposed Git repository. I'm getting crawled from about 100000 residential IPs from several ASNs (mostly in Brazil). Each crawler makes a couple of requests per minute.
Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
> I really don't see why Safari on a modern iPhone should be so slow at this
me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?
It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.
So, they can interleave different tokens and get stopless access at normal price?
Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.
Paying a second of CPU time is an inconvenience to a user but trivial for LLM scraping.
The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!
At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.
> If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution.
Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.
I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero".
Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed.
>It's sole purpose is to deny the bots you don't want on your site.
This is fundamentally impossible.
In the end only pay per crawl will work. Like x402 but maybe something similar.
In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
> Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose.
Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".
It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.
And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.
Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with.
Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together.
Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
Most of the problem in this case is that the scrapers are being incredibly lazy and not investing even minimal effort in their approach. If this forces them to revisit how they go about things then mission accomplished.
I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.
My iPad Pro M2 takes around 30 seconds to solve it in Safari, while my Mac Studio solves it in about 3 seconds using 449% logical CPUs. It really makes no sense. Interesting how much slower it is than native - could it be protections against things such as crypto mining?
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
You yourself describe the solution.
Extensions or browser integrations of the Anubis task.
He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.
Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once.
Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...
What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!
> rather than moving to a challenge system that actually impacts scrapers
At this point isn't it basically auth-only?
Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)
I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
Out of curiosity I tried to load this on my iPhone 12 and it took about 8 and a half minutes and once it finally passed it said my browser didn't allow cookies and to enable them before trying again. I would have to go through the whole proof again. I get what they're doing but there definitely needs to be some work done on getting those times down on iphones.
It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.
One natural counter to this is to dynamically alter the computing algorithm per request (variants of Anubis?) so that the scrapers would need to rewrite their accelerator each time or just use the unoptimized JS version...
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience.
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part.
Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
> Anubis wants "slow for bots but fast for humans"
That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.
That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance.
I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?
anubis was a joke on arrival. PoW for this purpose is plain stupid.
the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.
Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
That's the whole raison d'etre for CloudFlare, it was originally a DDoS protection layer, which, as the article mentions, is the final form of malicious traffic, being distributed and hard to attribute traffic to an identity.
If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.
FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.
Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
I think he means, get cloudflare to cache your content, so the traffic never reaches your servers to begin with. Assuming your sites content is cacheable by cloudflare. I agree it's a sad state of affairs if you have to rely on a 3rd party..
Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it has to do it serve up dumb .html pages, needing next to no compute cycles.
A creepy crawly is a South African invention to clean your swimming pool. The company that introduced them in the 70ies is called Kreepy Krauly. Also popular in Australia.
It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.
What prevents people from obtaining or buying such devices and automating them? Using TVs as proxies is just one example of that. People will be willing to give their ID cards away too, if you pay them or beat them enough.
In fact they already do this. Buying 100 android phones and chargers is cheaper than reverse engineering whatever you're trying to automate - or was, before AI.
I run a website with 10k unique pages. If I leave the gates open, Meta hits it 200.000 times per day. Every day. What are you paying developers $500k for Mark?
> [...] when a source is guaranteed to be LLM-free, like the entire history of kernel commits [...]
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...
There is a section in the article answering your question if you read it.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.
That is a ridiculous way to try and deal with the problem of residential proxies.
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
Similar to how people running an open SMTP are complicit in promoting spam, I see people running a wild public proxy as complicit in this malicious scraping activity.
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
Hey, from the beginning of SMTP, running an open relay was an administrative mistake. The MTA administrators were supposed to know what they were doing, because resources were allocated to them. They had privileges granted for the system and the network. It was right if they were blacklisted for misuse of those resources.
Now in 2026, running a "public proxy" doesn't take an administrator. You don't even need to be aware. Most victims are unknowing victims. They simply subscribe to an ISP and they have their own devices. They are being exploited for that innocence and ignorance. Most victims have no visibility to even detect that they're being used as a proxy. Most victims couldn't stop it, even if they wanted to.
I challenge anyone with a home router to list the processes running on that router, and list all current open connections on that router, and list all open, listening sockets on that router. I bet you can't do it. There are no consumer router OS that lend themselves to being secured, or even diagnosed. Malware can easily be planted on any of them and run, completely invisibly.
A residential proxy server could run on routers, could run on a switch, could run on your "Smart TV" or a smartphone, or a notebook computer. It could be anywhere in any form. Perhaps you consented to it, perhaps you didn't notice.
In no way is this the same as an SMTP open relay situation. If you wanna play "whack-a-mole" with a "blackhole list" you're simply going to overwhelm those lists with false positives and collateral damage. The residential proxies have long since moved on. You won't even find the culprits using those addresses you just blocked. You're just clogging up your own machines. It's a total self-own.
> You are, in reality, only hurting the actual owners
How many times do I have to hurt them before they decide to buy a different smart TV?
Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.
The thing that bothers me is why the fuck are they still scraping git.kernel.org or any other site that has already been scraped a million times before. Who would pay for that data?
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
How do you ensure it doesn’t include every non-human?
I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.
You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition.
Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough
That’s a pragmatic and understandable argument. And for an individual hobbyist site owner, that’s fine. Are we okay with excluding 1 person in 20 from the services of a midsized organization? What if they’re integral to the workplace? Or a major transport provider without differentiated competitors? What if the organization is a state government?
Every place on earth has some legal definition of who is human. The system I’m thinking of isn’t a technical/captcha one, it’s a human curated list of humans. Just an electronic ID. Those already exist but the challenge is making them (acceptably) privacy-preserving.
I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.
> phone gets uncomfortably warm as it's doing the number crunching
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally.
If someone else wants to run a public HTML service, let them deal with the bots.
If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.
I think you underestimate the difficulty in effectively gating user registration. If the anti-bot efforts in general don't work for other site pages, they won't work for signup functionality either.
I've been on the other side of this kind of thing (hero rather than villain, though I'm sure someone out there disagrees).
You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's worth for the amount of activity they get from each account.
If it costs your attacker a penny to solve the captcha to make an account, and they can only get 100 pages out of an account, you win.
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
One thing I've thought of trying is, if I ever get around to putting things back online, loading the pages with altered versions of themselves in a manner that human users are unlikely to see (though they could if they wanted to) but scrapers are unlikely to ignore (because they look enough like something users might see, perhaps by opening a <detail> tag).
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
To be clear, making pages with bad training data for bots won't make the bots go away.
It'll just punish the bad actors running the scrapers.
As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
> … making pages with bad training data for bots won't make the bots go away.
It'll just punish the bad actors running the scrapers.
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
Thankfully this is my photography website so what they're mostly after is images, which they have to request. I'm also intercepting all my legit images once the bot is flagged, which will no doubt have unintended consequences but I'll find out at some point I guess.
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
Gemini's surprisingly bad at it but so far I have some voluptuous curvy toasters, toasters in the shape of a butt, and robot girls with toasters for tits. I have a Gemini subscription for the month after mistakenly thinking I'd get cheap Opencode usage through it, so gotta use it for something.
Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
Have you considered releasing any of this as a plug for phoenix or similar?
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
“Expect to lose some functionality, at least when accessing our resources anonymously.”
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
1.4 billions requests, 258 160 cpu hours.
That's 1.5 requests per second ?
I'm starting to believe, the issue is more that their software is not well optimized.
This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date.
If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.
CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.
There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
Content-addressing decouples the hoster from the data itself. Anyone can serve content-addressed data and you can locally verify that you’ve got served the correct bytes. IPFS implements building blocks for such an alternative web.
What irony that this article is about crawling content-addressed data.
Agreed on both points. But it's looking increasingly likely that particular dream is not coming true. Kubo, the reference implementation IPFS node, is maintainerless as of last week.
I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).
I think focusing on filecoin was probably the mistake. You've got to build something that people trust first and then consider adding a money-shaped app. If you start with something money-shaped you're indistinguishable from the legions of scams, and that's a hard position to start from if you're wanting to build something trustworthy.
What I'm after is a way to get away from polling by server name. With a sufficiently interested audience I ought to be able to be offline except when I need to update the merkle root, meanwhile users can just serve the content to each other.
I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
A lot of work is apparently put into bypassing any kind of anti-scraping, no work is apparently put into figuring out if the site freely gives a way to get all that information in a less wasteful way.
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
I run a non source code service and I also get these, also getting hit by literally millions of residential IPs being cycled after 1 - 2 hits. Ingesting thousands of copies of near identical pages. The bots seem to be programmed in an extremely simple manner, using non-obfuscated javascript to write the href attribute seems to stop them from seeing links, where previously they would endlessly crawl generated links. Unfortunately it also means noscript users get shafted.
In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.
I'm not suggesting 'login' because facebook/twitter/etc, just as a mechanism to make the bot problem go away. They clearly want it to stop, they tried obvious methods but the AI platforms are circumventing it (deliberately) which is poor form.
I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.
As the article describes, it doesn't help, because the traffic originates from millions of unique residential IPs across hundreds of ASNs and countries.
Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000
The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution.
And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma
How feasible would it be to only offer a binary git (partial) download and move the html rendering to the client? It would still be a lot of requests, but less work for those servers. Not that I like SPA’s, but they could be useful here.
I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging.
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years.
Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely?
Those ecommerce stacks serve a large fixed number of pages. cgit does not.
Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages.
Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.
Now imagine scrapers are permuting those URLs.
How would you cache that?
That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate.
The following from Claude:
"""
A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average:
8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes
"""
I'm not sure if the website allows for diffs against arbitrary tree states, but if it does than the diff space is completely unbounded, and the ram demand is theoretically infinite.
Per the article, the vast majority of traffic is scrapers that are generating effectively random, permuted URLs. Even if they're already caching their popular-with-real-humans most recent data, that efficiency gain is a tiny drop in a very large bucket.
So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right?
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
At this point they are using residential proxies and stuff, and increasing the difficulty level is not going to help, among other things because they don't pay for it.
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.
I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
That's really funny. It's always been kind of amusing to me that Anthropic has this air about them of trying to be the most ethical AI company, but really exhibits the same behaviour as all the others.
> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data.
Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
Why not use the POW to help cover the costs? Mine an actual coin (Annubis Coin?) and pay for anonymous infra access with it (or log in and get a certain quota for free)?
There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are.
Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.
Some kind of proof of stake might be viable. "I as a visitor stake 1 cent that I'm a genuine user and not a sloppy bot, server is free to withdraw my stake if it's not true". If works, withdrawn money can be used to cover hosting costs.
There are definitely ways of proving you're a human, unfortunately it also means giving up your privacy and anonymity (IRL ID Checks combined with appropriate routing and validation - even going as far as certifying the browser being used)
Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the average redditor.
But! Using a network of real ID-checked humans to scrape the web, what would that be--half a billion times harder than Firecrawl or whatever they use today?
Too bad it's dead in the water today because so many (like me) hate the idea so much. Perhaps a biometric dongle (retinal-scanning orb :-/ ) that the staunchest privacy hawks stamp with their seals of approval because it's somehow nearly impossible to go horribly horribly... horribly... wrong... Yeah, anybody who can crack this issue, hope you have the free time or find the funding to try it, we need ya.
All this happens due to companies like browserbase, Hyperbrowser, Scrapefly. These service exists to facilitate such operation and they aren't doing anything to prevent abuse. They are infact selling way to bypass captchas etc...
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
What makes you think that would accomplish anything?
These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.
Time to F*$k the internet. The whole concept of anonymous IP addresses was broken but worked for a long time. Now it is just stupid. Just like domain names. (Are more names used by squatters than real?). And email as identity? Time to engineer solutions and create a new protocol layer.
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
I wonder how much is for training vs for LLMs doing research. On several occasions Claude has gone digging through kernel archives on my behalf (sometimes at my direction, other times all on its own). Usually to determine the current status of some kernel bug I'm experiencing. Apologies for the load, but I'm sure it was much less than an actual crawler trying to slurp up everything.
TLDR basically old tech is not optimized for scrapers / slurpers / etc. to the point it would take 42^n to solve all possible combinations. Why? Murphy's law. Solution for OP is to ignore for now. Otherwise Use or invent something else. Easy enough. other notes... Anubis and other gatekeepers dont work perfectly, but ok for now.
Just for fun, because I could, I vibed up a `cgit` replacement that runs entirely in the browser -- point it at a git repo where you've run `git update-server-info` and it'll load files as if it's starting to clone the repo, using range requests and browser caching to avoid actually loading more data than necessary for the view you've requested.
I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.
It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb
This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.
Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.
I just had a look through my logs, and I've had over 90,000 requests from known AI bots over the last month. All this to a personal website that doesn't post very often.
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
I have the feeling that the hate might be misplaced.
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org.
What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co.
Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them.
Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room.
Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits
Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.
> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.
> it is a shame that such talented people would not be able to have a proper optimization
It's mostly not static content in the sense that you're implying.
Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).
There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.
Ironically, defense by obscurity may be the way to go here.
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
> Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.
It doesn't matter what the hash is if it is inherently cheaper for a bot farm to compute the hashes than it is for a human to do it on their device. The human pays a greater cost in annoyance, wasted time, battery, and that means the PoW has failed its function. The bot farm owner does not care.
There's something missing from the picture. The bots are:
- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.
- Putting in tons of CPU cycles to surpass the Anubis PoC.
- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.
This sounds more like escalating DDoS than AI scraping.
Possible, but I doubt it. The sheer number of other free-to-read content sites dealing with the exact same problem the last few years (many of whom there's no plausible reason to DDoS) tells us that this is content harvesting, not an attack.
I'm curious and would like to see those reports. There's AI scraping for sure, but intentionally resource-consuming, increasingly-insidious AI scraping I've never actually read about.
AI scraping might be bad, but if a particular case that's actually a DDoS becomes the cause celebre against AI scraping, it will weaken the argument, not strengthen it.
My takeaway: we're degrading the web for real people to slow down bots that will just move forward. The solution seems is worse than the problem. At some point, we need to accept that the open web as we knew it is dying...
One problem with Anubis is that once you've solved the POW once, you just need to hold the cookie to avoid solving it again. Scrapers have probably learnt to do that by now. So Anubis isn't as effective as it used to be before it was widely used.
If you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org similarly to other sites like Wikipedia, but not the majority of scrapers
> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes.
That is super interesting, thank you!
> At that price point, its actually very affordable to many thousands of organizations to get their own copy.
I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening.
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.
Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.
Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.
I don't think the show-commit or file-at-revision routes are what's causing the bot load.
Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.
I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.
This is a much more substantial critique :) I definitely don't think this is a trivial idea. I did a quick analysis and the naivest possible solution isn't very good: only looking at uncompressed objects requires scanning ~340 MB to deliver the <1 MB answer the question "show me the diffs by user Zhang Yi in file fs/ext4/inode.c on branch master in 2025". The bulk of that is scanning commit objects for commits by Zhang Yi in 2025 (171 MB), then fetching trees to filter those to just fs/ext4/inode.c (159 MB).
But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)
I think the solution is for the POW being done by the clients to *actually benefit the site owner*. Users remain just as mildly annoyed as with Anubis, but maybe a bit less knowing that the work they’re doing benefits the site owner/author, and the system helps thwart the bots (or at least makes them do work that benefits the author).
it's getting insane, I have a high profile client, I manage their infrastructure including web server. I swore to them years ago they would not have to turn on the CF managed challenge / under attack / human verification.
I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.
and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.
How about making cgit more efficient at serving these pages. There's no excuse for burning a ton of CPU power on purely static pages when you have generous resources available to you.
There is a HN article on abusive AI crawlers on the front page almost every week, but we rarely talk about the path forward. Web scraping has been around for as long as the internet, and it was fine because we had established best practices (rate limiting, self-identification, robots.txt, etc.) that the industry agreed upon. Now we have AI labs and their crawlers that don't care about any of this gentlemen's agreement.
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
jruohonen | a day ago
dingaling911 | 15 hours ago
jruohonen | 15 hours ago
daveguy | 13 hours ago
Also, many plotting libraries include an xkcd style these days:
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....
And libraries for various languages:
https://github.com/timqian/chart.xkcd
So if you have a preferred dev environment there's probably a way to set it to xkcd style.
electrogas | 14 hours ago
Artoooooor | 15 hours ago
parineum | 14 hours ago
inigyou | 9 hours ago
wredcoll | 14 hours ago
initramfs | 14 hours ago
Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-python
https://github.com/mthcht/Masquerade-Spoofer
Great for AI scrapers, bad for hosters and everyone else.
chuckadams | 14 hours ago
Zariel | 14 hours ago
teo_zero | 14 hours ago
kijin | 14 hours ago
oowa | 9 hours ago
oowa | 6 hours ago
oowa | 9 hours ago
kees99 | 14 hours ago
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
[0] https://git.zx2c4.com/cgit/about/
oowa | 13 hours ago
mik1998 | 14 hours ago
oowa | 9 hours ago
rcxdude | 14 hours ago
charcircuit | 47 minutes ago
oowa | 13 hours ago
oowa | 9 hours ago
skybrian | 12 hours ago
https://bsky.app/profile/xeiaso.net/post/3muasair5uk2j
Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.
lkbm | 14 hours ago
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
phmx | 14 hours ago
kalkin | 13 hours ago
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
bigstrat2003 | 8 hours ago
kalkin | 7 hours ago
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
https://www.scientificamerican.com/article/no-ai-didnt-just-...
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
Barbing | 6 hours ago
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
DarmokTanagra | 12 hours ago
Thats a good enough explanation in my opinion.
Source: someone who reviews AI generated PR's on a daily basis
nicman23 | 14 hours ago
dunder_cat | 14 hours ago
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
Without something like cookies (which are almost certainly tossed after the IP is rotated) or some other persistent identifier, you are stuck have to apply mitigations that scale with the load you're encountering, which means longer challenges for everyone or degraded functionality, like removing some of the fancier cgit features.
marginalia_nu | 12 hours ago
nicman23 | 11 hours ago
PinkaDunka | 14 hours ago
nicman23 | 11 hours ago
Velocifyer | 14 hours ago
lkbm | 14 hours ago
acedTrex | 14 hours ago
rcxdude | 14 hours ago
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
voakbasda | 13 hours ago
ipdashc | 13 hours ago
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
tmp10423288442 | 13 hours ago
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
OAI-AdsBot → advertising landing-page crawl
inigyou | 9 hours ago
Symbiote | 12 hours ago
(No second chances, I blocked their whole IP range.)
inigyou | 9 hours ago
nubinetwork | 8 hours ago
inigyou | 14 minutes ago
DarmokTanagra | 12 hours ago
AshamedCaptain | 14 hours ago
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
rwmj | 13 hours ago
singpolyma3 | 13 hours ago
a-dub | 14 hours ago
yellow_lead | 14 hours ago
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
sethops1 | 14 hours ago
igor47 | 13 hours ago
rpcope1 | 3 hours ago
pixl97 | 12 hours ago
marginalia_nu | 12 hours ago
inigyou | 9 hours ago
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
roughly | 8 hours ago
inigyou | 10 minutes ago
MithrilTuxedo | 13 hours ago
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
IshKebab | 13 hours ago
It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.
inigyou | 9 hours ago
andy81 | 7 hours ago
KomoD | 12 hours ago
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
johneth | 12 hours ago
colinsane | 8 hours ago
when a system was designed for 100 req/s and bots hit it with 5000 req/s, nobody entering that queue is having a good time. Anubis is the trade those operators make just to ensure your request gets serviced at all.
30s load time is already a sign that the Anubis approach is breaking down. if there's nothing else ready by the next order-of-magnitude increase in crawler load, those sites quite likely will just disappear from the public internet. hate Anubis all you want: for most of us, the realistic alternative is strictly worse.
Demiurge | 14 hours ago
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
andai | 14 hours ago
I would promote this idea to all of my competitors. Nah mate, you don't have to ask her out right now. You can wait until next week ;)
Anyway, silver linings, looks like we're finally going to get widely adopted infra for microtransactions.
https://web.archive.org/web/20030202042510/http://www.openp2...
Demiurge | 12 hours ago
Are the AI companies worried about anything but their valuations?
I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need to fund any risk mitigations, security coordination, and other safety work. If these things are not imposed on the market, the most ruthless player will create the biggest mess they can get away with.
I don’t see a better way to do this, right now, than to allow Strips, or CF, to create some way to charge the crawlers. Imagine I have some really expensive content, I can put it behind the paywall, or subscription model, but that’s very specific to the human consumer. What if I do want to search engine or content aggregator to consume some of the content? It would be quite hard for me to sign an exclusive agreement with OpenAI, like what Reddit or NYT are doing. It might be quite beneficial to create some infrastructure to let bots automatically to pay for access. It might even be crypto based, I don’t think it really matters. It just has to be easy for me to integrate it, and the bots need to find it easy to use.
If we all start doing this, I think this could be of benefit to all content creators, such as musicians, movie makers, game developers, and even independent journalists.
We used to pay for software and things, but what has happened? Walled gardens and production monopolies are taking over. This has largely reduced the quality of all the content we consume.
inigyou | 9 hours ago
Demiurge | 9 hours ago
Anyway, what’s wrong with making 2 cents? :)
rankdiff | 14 hours ago
klez | 14 hours ago
Demiurge | 13 hours ago
iririririr | 13 hours ago
they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.
TZubiri | 12 hours ago
1oooqooq | 12 hours ago
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
Demiurge | 12 hours ago
inigyou | 9 hours ago
Demiurge | 9 hours ago
TZubiri | 9 hours ago
So you can definitely block by IP, by fingerprint, both or neither
realusername | 12 hours ago
And Cloudflare is no exception, the "hello world" of bots is Cloudflare bypass.
timpera | 12 hours ago
Demiurge | 12 hours ago
anal_reactor | 13 hours ago
I feel like it's the same issue that public resources are facing.
Step 1: public resource is freely available
Step 2: asshole shows up, drains the resource completely just because they can
Step 3: either resource is taken away, or it has massive security behind it
inigyou | 9 hours ago
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
J-Kuhn | an hour ago
pdonis | 12 hours ago
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
Demiurge | 11 hours ago
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
mbirth | 11 hours ago
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
Hackbraten | 7 hours ago
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
nunez | 11 hours ago
persolb | 11 hours ago
acedTrex | 14 hours ago
petesergeant | 14 hours ago
Is it foolish of me to have expected more from a blog post on kernel.org?
theandrewbailey | 14 hours ago
Lerc | 13 hours ago
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
petesergeant | 7 hours ago
easton | 14 hours ago
jacobvosmaer | 13 hours ago
Hackbraten | 7 hours ago
[0]: https://github.com/Homebrew/brew/pull/9383
[1]: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
ygouzerh | 2 hours ago
feelamee | 14 hours ago
First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.
So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
klez | 13 hours ago
So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly.
EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should resist slowly breaking it further.
feelamee | 13 hours ago
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
klez | 12 hours ago
inigyou | 9 hours ago
rwmj | 13 hours ago
feelamee | 13 hours ago
inigyou | 9 hours ago
zbentley | 3 hours ago
> [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions]
Never going to happen.
inigyou | 12 minutes ago
delichon | 14 hours ago
Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
iririririr | 13 hours ago
wiredfool | 13 hours ago
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
marginalia_nu | 12 hours ago
inigyou | 9 hours ago
marginalia_nu | 9 hours ago
inigyou | 9 hours ago
wiredfool | 8 hours ago
edent | 14 hours ago
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
kardos | 13 hours ago
tptacek | 13 hours ago
https://news.ycombinator.com/item?id=44962529
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
nneonneo | 13 hours ago
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
Y_Y | 13 hours ago
wongarsu | 11 hours ago
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
inigyou | 9 hours ago
tptacek | 13 hours ago
yorwba | 9 hours ago
kstrauser | 2 hours ago
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
akerl_ | an hour ago
kstrauser | 37 minutes ago
lxgr | 12 hours ago
Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.
The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.
[1] https://news.ycombinator.com/item?id=49500040
FridgeSeal | 2 hours ago
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
graemep | 12 hours ago
lxgr | 12 hours ago
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
semiquaver | 12 hours ago
oasisbob | 11 hours ago
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
harshreality | 12 hours ago
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
tptacek | 12 hours ago
I'm just saying, it's bad computer science.
harshreality | 12 hours ago
A better, perhaps the best, argument is that a worse bot mix might appear tomorrow, so anubis should be used as a runway for a better solution, and not as a solution in itself. If only everyone with a website had the desire (or AI tokens) to implement their own better solution. In some cases even desire or a lot of tokens isn't enough. I mean, you think kernel.org folks haven't put some serious thought into it? And yet, as HN argues about it, there's kernel.org running anubis at difficulty 5.
tptacek | 12 hours ago
harshreality | 11 hours ago
rvz | 6 hours ago
It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions.
Unless they come up with said "solution", they can completely ignored in their complaints.
tptacek | 4 hours ago
MallocVoidstar | 4 hours ago
myng111 | an hour ago
rvz | 2 hours ago
So could you give a short elaboration on what was this "better computer science" solution that you came up with in your discussion?
inigyou | 9 hours ago
TeMPOraL | 9 hours ago
- It's the SOTA companies doing it?
- Scrapers are doing it for training data?
Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.
inigyou | 9 hours ago
mschuster91 | 6 hours ago
There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints.
I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
figglestar | 6 hours ago
shakna | 6 hours ago
[0] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61...
inigyou | 8 minutes ago
charcircuit | an hour ago
chr15m | 4 hours ago
nneonneo | 13 hours ago
Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
nxndbebdb | 13 hours ago
nneonneo | 13 hours ago
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
colinsane | 7 hours ago
Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:
``` { "bots": [ { "action": "WEIGH", "expression": "path.startsWith(\"/expensive/endpoint\")", "name": "scrutinize-expensive-endpoints", "weight": { "adjust": 20 } } ] } ```
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
``` { "bots": [ { "action": "WEIGH", "expression": "load_15m <= 16.0", "name": "sustained-low-load", "weight": { "adjust": -10 } }, { "action": "WEIGH", "expression": "load_5m >= 24.0", "name": "intermittent-high-load", "weight": { "adjust": 10 } }, ] } ```
bauerd | 13 hours ago
tarpitt | 13 hours ago
NooneAtAll3 | 13 hours ago
inigyou | 9 hours ago
jay_kyburz | 8 hours ago
sgsjchs | 5 hours ago
jay_kyburz | 2 hours ago
If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.
The next more advanced version is that URLs are unique to your cookie.
Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.
You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)
inigyou | 13 minutes ago
bluedino | 13 hours ago
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
nxndbebdb | 13 hours ago
RussianBot9580 | 13 hours ago
hdbsbs | 10 hours ago
NooneAtAll3 | 13 hours ago
1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally
2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
voakbasda | 13 hours ago
alwa | 12 hours ago
voakbasda | 11 hours ago
inigyou | 9 hours ago
lxgr | 12 hours ago
In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.
_ks3e | 11 hours ago
the8472 | 10 hours ago
johneth | 12 hours ago
zbentley | 3 hours ago
inigyou | 9 hours ago
Hackbraten | 6 hours ago
znnajdla | 13 hours ago
ninglor | 12 hours ago
grep_it | 12 hours ago
“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
jwilk | 10 hours ago
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
inigyou | 9 hours ago
jopsen | 12 hours ago
But if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
inigyou | 9 hours ago
0xbadcafebee | 13 hours ago
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
gib444 | 13 hours ago
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
jdnier | 13 hours ago
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
TZubiri | 12 hours ago
atq2119 | 11 hours ago
inigyou | 9 hours ago
NavinF | 8 hours ago
p_j_w | 6 hours ago
emsign | 5 hours ago
inigyou | 9 minutes ago
NegativeLatency | 13 hours ago
ynniv | 13 hours ago
which is called l402/x402. micropayments' day has come
inigyou | 9 hours ago
The author of Anubis hates this fact and will ban you if you mention it, so don't. He insists it's the PoW.
ynniv | 8 hours ago
Barbing | 6 hours ago
>will ban you if you mention it
Even if mentioned really politely?
[1] would hope anyone trying this makes it accessible to visitors with disabilities
QuantumNomad_ | 2 hours ago
Anyone who responds with something like:
> I don’t have subjective feelings, consciousness, or an inner point of view, so there isn’t a sensation of “being me.”
is definitely an LLM.
Waterluvian | 13 hours ago
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
BowBun | 13 hours ago
The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.
inigyou | 9 hours ago
colinsane | 8 hours ago
> What is the cost of 20% and is that cost worth “mildly annoying” everyone?
from the articled:
> With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.
this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of these bad actors.
Waterluvian | 6 hours ago
Nevertheless, without knowing what it actually is in cost, it’s impossible to have an objective sense of the severity of this problem.
Is it, “everyone’s mildly annoyed so we can save $10k per year” or is it “everyone’s mildly annoyed so we can save $100k per year?” And has someone attempted to reason about what the $/year cost is of that mild annoyance?
I’ve seen this a lot in my career. Developers coming to conclusions that are ultimately based on what feels expensive or problematic.
singpolyma3 | 13 hours ago
Symbiote | 13 hours ago
johneth | 12 hours ago
inigyou | 9 hours ago
singpolyma3 | 10 hours ago
inigyou | 9 hours ago
mmooss | 12 hours ago
For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.
akersten | 11 hours ago
You have the power, and should exercise it, to rate limit bad actors
inigyou | 9 hours ago
mmooss | 6 hours ago
It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.
iririririr | 13 hours ago
dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.
www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
semiquaver | 13 hours ago
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
smallerize | 13 hours ago
semiquaver | 12 hours ago
NobodyNada | 12 hours ago
semiquaver | 12 hours ago
Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.
a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
LatticeAnimal | 12 hours ago
I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
inigyou | 11 hours ago
LatticeAnimal | 8 hours ago
My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...
All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)
CursedSilicon | 10 hours ago
inigyou | 11 hours ago
semiquaver | 5 hours ago
You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.
rstuart4133 | 3 hours ago
If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.
dspillett | 10 hours ago
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
semiquaver | 5 hours ago
Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
dspillett | 4 hours ago
If they are so stupid an idea that they could never work, why do they seem to having the desired effect?
Dylan16807 | 11 minutes ago
If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it.
tptacek | 12 hours ago
That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".
graemep | 12 hours ago
tptacek | 12 hours ago
Which: I get! If it's your site, perform all the anger you want.
progval | 11 hours ago
There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.
tptacek | 8 hours ago
cyberax | 10 hours ago
Anubis doesn't really help against this.
DarmokTanagra | 11 hours ago
wongarsu | 12 hours ago
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
semiquaver | 12 hours ago
paytonjjones | 12 hours ago
They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
pbronez | 11 hours ago
paytonjjones | 11 hours ago
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
inigyou | 10 hours ago
ekidd | 9 hours ago
inigyou | 9 hours ago
afdbcreid | 3 hours ago
(Also, blocking a whole country is likely not what you do, but you probably know that).
inigyou | 8 minutes ago
eru | 3 hours ago
Developers and compute are interchangeable now.
nozzlegear | 2 hours ago
eru | 2 hours ago
Aurornis | 11 hours ago
Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.
So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.
A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.
The amortized per-view cost is highest for the real user.
rplnt | 9 hours ago
JsonCameron | 8 hours ago
afdbcreid | 3 hours ago
Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.
embedding-shape | 8 hours ago
> so I really don't see why Safari on a modern iPhone should be so slow at this
FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.
physicalecon | 5 hours ago
Shank | 3 hours ago
supriyo-biswas | an hour ago
reitzensteinm | 7 hours ago
The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!
At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.
fc417fc802 | 3 hours ago
Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.
TZubiri | 12 hours ago
When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.
lxgr | 12 hours ago
Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
TZubiri | 9 hours ago
Self-Perfection | 12 hours ago
But it would just make Anubis suck a little bit less
Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.
bayindirh | 10 hours ago
Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
Self-Perfection | 5 hours ago
This is fundamentally impossible.
In the end only pay per crawl will work. Like x402 but maybe something similar.
In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
thaumasiotes | 4 hours ago
No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose.
Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".
rzmmm | 12 hours ago
inigyou | 11 hours ago
wbl | 10 hours ago
applfanboysbgon | 9 hours ago
Terr_ | 9 hours ago
Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
wbl | 4 hours ago
fc417fc802 | 3 hours ago
radlad | 12 hours ago
jwilk | 11 hours ago
minimaltom | 11 hours ago
You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...
inventor7777 | 10 hours ago
karlgkk | 10 hours ago
You yourself describe the solution.
Extensions or browser integrations of the Anubis task.
dpark | 9 hours ago
karlgkk | an hour ago
DenisM | 9 hours ago
Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
brokensegue | 8 hours ago
apitman | 8 hours ago
xena | 8 hours ago
tptacek | 8 hours ago
darkmighty | 5 hours ago
At this point isn't it basically auth-only?
Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)
I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
samtheDamned | 8 hours ago
gizmodo59 | 8 hours ago
semiquaver | 7 hours ago
latentsea | 3 hours ago
Dylan16807 | 31 minutes ago
I don't see much connection between that problem and these controls.
nozzlegear | 2 hours ago
All in the name of safety, of course.
zalo | 8 hours ago
stjo | 6 hours ago
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
jcul | 6 hours ago
I might try to tackle it again when I can give it some more attention.
chr15m | 4 hours ago
Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
eru | 3 hours ago
Opportunity costs are real.
margalabargala | 3 hours ago
Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
fc417fc802 | 3 hours ago
That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.
Dylan16807 | 40 minutes ago
I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?
teravor | 2 hours ago
the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.
FergusArgyll | 46 minutes ago
Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds
adverbly | 13 hours ago
TZubiri | 12 hours ago
Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
inigyou | 9 hours ago
TZubiri | 2 hours ago
If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.
inigyou | 14 minutes ago
marginalia_nu | 12 hours ago
Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
Velocifyer | 12 hours ago
inigyou | 9 hours ago
wingworks | 8 hours ago
Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it has to do it serve up dumb .html pages, needing next to no compute cycles.
inigyou | 13 minutes ago
andruby | 12 hours ago
https://kreepykrauly.co.za/about-us/
Symbiote | 10 hours ago
leoqa | 12 hours ago
okanat | 10 hours ago
inigyou | 9 hours ago
leoqa | 6 hours ago
inigyou | 15 minutes ago
DarmokTanagra | 12 hours ago
api | 12 hours ago
jopsen | 12 hours ago
I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
iamniels | 11 hours ago
jopsen | 10 hours ago
That's bad for static content.
Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.
mbirth | 7 hours ago
lxgr | 12 hours ago
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
[1] https://docs.kernel.org/process/coding-assistants.html
afarah1 | 12 hours ago
mattmcal | 11 hours ago
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
mbirth | 11 hours ago
inigyou | 9 hours ago
Some ISPs ban customers based on a single report there - have fun!
mbirth | 8 hours ago
ButlerianJihad | 8 hours ago
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
mbirth | 7 hours ago
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
ButlerianJihad | 5 hours ago
Now in 2026, running a "public proxy" doesn't take an administrator. You don't even need to be aware. Most victims are unknowing victims. They simply subscribe to an ISP and they have their own devices. They are being exploited for that innocence and ignorance. Most victims have no visibility to even detect that they're being used as a proxy. Most victims couldn't stop it, even if they wanted to.
I challenge anyone with a home router to list the processes running on that router, and list all current open connections on that router, and list all open, listening sockets on that router. I bet you can't do it. There are no consumer router OS that lend themselves to being secured, or even diagnosed. Malware can easily be planted on any of them and run, completely invisibly.
A residential proxy server could run on routers, could run on a switch, could run on your "Smart TV" or a smartphone, or a notebook computer. It could be anywhere in any form. Perhaps you consented to it, perhaps you didn't notice.
In no way is this the same as an SMTP open relay situation. If you wanna play "whack-a-mole" with a "blackhole list" you're simply going to overwhelm those lists with false positives and collateral damage. The residential proxies have long since moved on. You won't even find the culprits using those addresses you just blocked. You're just clogging up your own machines. It's a total self-own.
zbentley | 3 hours ago
How many times do I have to hurt them before they decide to buy a different smart TV?
Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.
fer | 9 hours ago
monegator | 11 hours ago
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
alkonaut | 11 hours ago
We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
Arubis | 11 hours ago
How do you ensure it includes every human?
How do you ensure it doesn’t include every non-human?
I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.
Kamq | 11 hours ago
Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough
Arubis | 10 hours ago
inigyou | 9 hours ago
alkonaut | 9 hours ago
I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.
ChocolateGod | 11 hours ago
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
inigyou | 9 hours ago
stickfigure | 11 hours ago
If someone else wants to run a public HTML service, let them deal with the bots.
If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.
Sucks, but here we are.
oasisbob | 11 hours ago
stickfigure | 2 hours ago
You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's worth for the amount of activity they get from each account.
If it costs your attacker a penny to solve the captcha to make an account, and they can only get 100 pages out of an account, you win.
TowerTall | an hour ago
robotmay | 11 hours ago
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
jopsen | 11 hours ago
The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
dspillett | 10 hours ago
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
jopsen | 8 hours ago
It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
dspillett | 8 hours ago
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
robotmay | 9 hours ago
embedding-shape | 11 hours ago
You can't tease us like that without showing screenshots.
robotmay | 9 hours ago
https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg
I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.
altairprime | 9 hours ago
Barbing | 7 hours ago
No hot toaster pics though :( on second thought perhaps better in our imaginations
altairprime | 4 hours ago
worldsavior | 9 hours ago
inventor7777 | 10 hours ago
robotmay | 9 hours ago
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
mlaretallack | 10 hours ago
robotmay | 10 hours ago
someothherguyy | 8 hours ago
hahnbee | 6 hours ago
charles_f | 5 hours ago
solid_fuel | 5 hours ago
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
eloisius | 2 hours ago
bilater | 11 hours ago
inigyou | 9 hours ago
pbronez | 11 hours ago
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
Kuinox | 11 hours ago
Skunkleton | 8 hours ago
Kuinox | 8 hours ago
6d6b73 | 11 hours ago
hnisjafx40 | 11 hours ago
__MatrixMan__ | 11 hours ago
If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.
CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.
There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
dennis-tra | 11 hours ago
What irony that this article is about crawling content-addressed data.
__MatrixMan__ | 11 hours ago
I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).
__MatrixMan__ | 10 hours ago
inigyou | 9 hours ago
__MatrixMan__ | 2 hours ago
inigyou | 11 minutes ago
inigyou | 11 hours ago
mzajc | 11 hours ago
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
vintermann | 10 hours ago
TonyTrapp | 9 hours ago
diegocg | 9 hours ago
emsign | 5 hours ago
jeremyjh | an hour ago
jonhohle | an hour ago
asddubs | 4 hours ago
arlattimore | 11 hours ago
kuschkufan | 9 hours ago
arlattimore | 4 hours ago
asah | 10 hours ago
virgoerns | 10 hours ago
inigyou | 9 hours ago
mzajc | 9 hours ago
inigyou | 9 hours ago
VladVladikoff | 7 hours ago
The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution.
And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma
left-struck | 4 hours ago
superjan | 10 hours ago
forrestthewoods | 10 hours ago
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
cobbzilla | 10 hours ago
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
bourse_lee | 10 hours ago
fer | 9 hours ago
calvinmorrison | 9 hours ago
Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
inigyou | 9 hours ago
> Us PHP developers
I can tell.
calvinmorrison | 9 hours ago
zbentley | 2 hours ago
Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages.
Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.
Now imagine scrapers are permuting those URLs.
How would you cache that?
That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
inigyou | 11 minutes ago
riknos314 | 8 hours ago
The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """
riknos314 | 8 hours ago
calvinmorrison | 7 hours ago
zbentley | 2 hours ago
hei-lima | 9 hours ago
userbinator | 9 hours ago
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
kgeist | 9 hours ago
RGamma | 7 hours ago
oowa | 9 hours ago
javcasas | 9 hours ago
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
beached_whale | 8 hours ago
kdowns | 8 hours ago
myng111 | an hour ago
duplessitous | 8 hours ago
Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
thomasjudge | 8 hours ago
goldenarm | 8 hours ago
innocent_name | 8 hours ago
If you're reading this - go ahead and see HOW Linux Foundation spends their money.
_blk | 7 hours ago
ivanjermakov | 7 hours ago
Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.
brownkonas | 7 hours ago
echelon | 7 hours ago
ivanjermakov | 7 hours ago
zythyx | 7 hours ago
Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the average redditor.
Barbing | 6 hours ago
Reminded of the mules renting Airbnbs to use as USA-based delivery locations (tricking grandma into FedExing cash for one scam or another) - https://getrichslowly.org/scambaiters (probably Jim Browning + Mark Rober specifically https://youtube.com/watch?v=Xvjjpzyiig4 )
But! Using a network of real ID-checked humans to scrape the web, what would that be--half a billion times harder than Firecrawl or whatever they use today?
Too bad it's dead in the water today because so many (like me) hate the idea so much. Perhaps a biometric dongle (retinal-scanning orb :-/ ) that the staunchest privacy hawks stamp with their seals of approval because it's somehow nearly impossible to go horribly horribly... horribly... wrong... Yeah, anybody who can crack this issue, hope you have the free time or find the funding to try it, we need ya.
mkl | 6 hours ago
Why? We are not so special or magical.
MetaWhirledPeas | 4 hours ago
I think I agree with this. There will be difficulties though:
- Will likely have to be pay-to-play. No point in whitelisting IPs if you are bombarded with whitelist requests by a bot.
- Will come with the assurance that any bot traffic will result in an IP ban.
- Will rule out people with dynamic IP addresses.
- Will be difficult to gain new users.
cute_boi | 7 hours ago
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
fizlebit | 7 hours ago
DrJThomasHusk | 7 hours ago
muahahahahah
Sorry, just the thought of it
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
I’m bookish.
phyzome | 2 hours ago
boredatoms | 7 hours ago
zbentley | 3 hours ago
These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.
bjourne | 7 hours ago
I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?
sgsjchs | 5 hours ago
yardstick | 7 hours ago
Allow git clone for free/unrestricted still.
Require the user to sign in to view html views. Sign in require a valid email or phone where a validation link is sent.
Or: Users signed in won’t see the Anubis. Users not signed in can still see the html views but have to use a very high work level?
Or: Limit unauthenticated requests from an IP to 5/minute. Authenticated requests can do a lot more before hitting the limit.
Barbing | 7 hours ago
Want to render than seven-year old commit via HTML? Sign in. (But I have no idea.)
talkingtab | 7 hours ago
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
hamandcheese | 6 hours ago
iLoveOncall | 6 hours ago
Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
oowa | 6 hours ago
louiskottmann | 6 hours ago
andrewaylett | 6 hours ago
I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.
It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb
This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.
Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.
CqtGLRGcukpy | 6 hours ago
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
justAnotherHero | 5 hours ago
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
vist_orn | 5 hours ago
greatgib | 5 hours ago
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org. What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co. Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. 14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room. Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
zbentley | 2 hours ago
This has nothing to do with that.
Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.
> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.
> it is a shame that such talented people would not be able to have a proper optimization
It's mostly not static content in the sense that you're implying.
Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).
There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.
emsign | 5 hours ago
sgsjchs | 5 hours ago
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
slipknotfan | 5 hours ago
A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.
chr15m | 4 hours ago
afdbcreid | 2 hours ago
dunlin | 5 hours ago
lmz | 5 hours ago
phyzome | 2 hours ago
lmz | 16 minutes ago
gruntled-worker | 5 hours ago
- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.
- Putting in tons of CPU cycles to surpass the Anubis PoC.
- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.
This sounds more like escalating DDoS than AI scraping.
zbentley | 3 hours ago
gruntled-worker | an hour ago
AI scraping might be bad, but if a particular case that's actually a DDoS becomes the cause celebre against AI scraping, it will weaken the argument, not strengthen it.
adangert | 5 hours ago
https://radicle.dev/
ironqcold | 4 hours ago
kristianp | 4 hours ago
UltraSane | 4 hours ago
Wowfunhappy | 4 hours ago
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
strix_varius | 4 hours ago
micah_chatt | 4 hours ago
Wowfunhappy | 3 hours ago
That is super interesting, thank you!
> At that price point, its actually very affordable to many thousands of organizations to get their own copy.
I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
CGamesPlay | 3 hours ago
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
zbentley | 3 hours ago
First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.
Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.
Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.
CGamesPlay | 3 hours ago
2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?
zbentley | 2 hours ago
Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.
I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.
CGamesPlay | an hour ago
But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)
wolttam | 3 hours ago
0xdeadbeefbabe | an hour ago
The horror.
BorisMelnik | an hour ago
I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.
and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.
charcircuit | an hour ago
dzhiurgis | an hour ago
stcg | 16 minutes ago
Wait what? I never heard of that. I call that a botnet
hubraumhugo | 11 minutes ago
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
- what else?
[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...
[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...