The Numbers felt this shift in two distinct waves. The first started around 2024:
We saw a big increase in crawls as AI training joined the search engine crawlers. The AI crawlers are generally less well-behaved than the search engines, which increased the management tasks for us to keep the site running smoothly.
And the second wave was stronger and more damaging:
Around December 2025, we saw another big spike in traffic which I attribute to agentic AI: a combination of AI agents that scrape sites in response to prompts, and people being able to write agents that scrape sites.
Bruce says that only 10% of their traffic is from humans browsing the site, with the rest coming from AI bots and automated traffic.
Around 90% of our time was spent keeping the existing site running while we spent our spare moments working on a new and improved system.
The Internet has always been a hostile place but it's becoming more so. I think we're going to see many websites either going behind a proxy like Cloudflare, or converting into a static website so that the web pages can be served by a CDN.
[OP] Kremor | a day ago
skybrian | a day ago
The Internet has always been a hostile place but it's becoming more so. I think we're going to see many websites either going behind a proxy like Cloudflare, or converting into a static website so that the web pages can be served by a CDN.