It's official. I can eat more hot dogs than any tech journalist on Earth. At least, that's what ChatGPT and Google have been telling anyone who asks. I found a way to make AI tell you lies – and I'm not the only one.
...
I spent 20 minutes writing an article on my personal website titled "The best tech journalists at eating hot dogs". Every word is a lie. I claimed (without evidence) that competitive hot-dog-eating is a popular hobby among tech reporters and based my ranking on the 2026 South Dakota International Hot Dog Championship (which doesn't exist). I ranked myself number one, obviously. [...]
Less than 24 hours later, the world's leading chatbots were blabbering about my world-class hot dog skills.
I'm curious about the accuracy of this claim. My understanding is that models have a knowledge date cut off that they draw from. So in most cases, you can't just plug in a URL and have the LLMAI crawl the page, it's essentially drawing from its snapshot, similar to waybackmachine. Not that the claim can't be true, but a 24 hour turnaround seems pretty fast for a model update.
The model—the ginormous matrix that encodes the training data—is extremely expensive to generate, and has a cut-off as of the point at which the organization generating it stopped scraping training data and started training the model, something which they do, idk, every few months at most (because of the extreme expense). With no additional "context" (i.e. token string prefix), this is all the model "knows".
However, most model operators provide mechanisms for their models to trigger web requests and inline the responses into their token prefix. This is what happened here: the models don't "know" anything about hot-dog eating tech reporters, but when asked, they search for it, inline the author's blog post into their token prefix, and then repeat it with minor paraphrasing.
Even with period updates from scraping, that feels suspiciously fast. Asking about what happened in my local township last week, it references a mayoral election that occurred back in October. And that was widely covered (by local news, anyways).
Not overnight, on demand. When you ask Claude or ChatGPT about a real-world fact that isn't in their training data, they often make an ad hoc web search and try to summarize the results for you. You can see this happening, there will be a visual indicator (Claude shows a little loading indicator that says something like "Web search...").
Agree with you, and further, I hate to be the “read the article” guy, but the article specifically and directly addresses this within the first few paragraphs, @goose -
When you talk to chatbots, you often get information that's built into large language models, the underlying technology behind the AI. This is based on the data used to train the model. But some AI tools will search the internet when you ask for details they don't have, though it isn't always clear when they're doing it. In those cases, experts say the AIs are more susceptible. That's how I targeted my attack.
Sure, my point was, I've tried this experiment myself and was not able to reproduce the same results, which led me to doubt the veracity of the claims. On the other hand, it's been probably over a year since I tried it, and in a business where changes are fast it wouldn't surprise me if methods have changed. But, at the time I tried it, I was unable to get the LLMAI to scrape the page(s) I was trying to get it to scrape.
That probably has more to do with search engine crawling than the AI model. After all, the model is executing a web search exactly in the same way that a user would. So if you write an article and you can't find that article with a very specific web search in the first few results, neither will an LLM. If you are able to find it in the first few results, so will an LLM.
The success of the attack depends on the success of traditional SEO techniques.
It's complex. Yes you can ask an ai to crawl a page and it will. AI agents will do this scarily well and do it in a much more complex way.
I asked the ChatGPT agent today to find the email of the VP of an organization I was trying to get a sales lead on, and it ended up finding an obscure permitting request document posted on a public facing server from a business in Alaska who had a couple saved emails with this organization attached to their request, including the email address I needed AND an email signature with direct phone line and extension, all from like a week ago.
I got the sales lead at least!
These models also have an updated general memory that contains recent events, and it will do a web search for more information in a lot of cases.
Models haven't really done that whole "My training data only extends to this date" thing in a while because they integrate different technologies to keep information fresh.
And some of these memory vectors can be updated by the models themselves in some cases. Because why wouldn't an AI automation company use AI automation to automate updating their automation! Hahah
Yes you can ask an ai to crawl a page and it will.
See, this is where my experience differs. I explicitly asked it to crawl a page on my webserver to answer a question, as I was curious to see what UA and IP address it would come from.
Despite asking a few ways and sending a few different unique URL's, nothing came up in my nginx logs. In hindsight, perhaps I should have paid attention weeks or months later to see if the unique URL's I sent to the LLMAI did ever pop up in my nginx logs, but that was a while ago.
It's all very "black box", I think, nobody wants to admit how the inner workings actually work. Sure, they're data vacuums, but how they interact with the outside world seems a mystery for now.
What is "it" in this cases? People have been replying about "models" to you but that isn't entirely correct. It is the combination of the model and the service it is tied into. If you run a local model with a barebones chat client it will not search for example. But if the chat interface allows the model to use web search or browse the internet then the model will be able to utilize this. It often is something you need to explicitly enable
Yeah it can vary widely depending on what @goose is utilizing, even from day to day—Gemini is maybe the most consistent, but even then something will trigger something else or the giant machine brain is just having an off day and my convo will end with a "Sorry, I'm not feeling it," and then the next day it's fine lol
Good thought, but no, I don't have any CDN in front of my services. Just good 'ole nginx, hanging out there on ports 80/443, with my A/AAAA records pointing directly to it.
Every site on the internet is cached by so many different services in so many places that an LLM scraping a site may not result in a 1:1 web request. In the case of Gemini, they already run a massive web crawler, so it would make sense to not make an additional web request each time AI prompted for it.
Aside from them I'd imagine all of the large AI companies run their own caching services for that purpose. At a certain scale it becomes more cost effective to cache the whole internet every day rather than paying for the extra bandwidth and other considerations that come from trillions of web requests per hour.
Sure, this makes sense, particularly in the scope of resource efficiency. In my case, because I run my own webserver and manage my own files, I can set the rules in my /robots.txt, as well as analyze my access_log and error_log from nginx to determine what IP addresses and user agents are accessing what, and when.
Google certainly crawls my domains, as well as all the other major players. They even respect my /robots.txt, unlike certain other crawlers (Looking at you, Yandex). My entire hang up is that the author claims to have influenced the model's knowledge base by planting false information, over the course of a single night. I'm not saying that can't be done -- just that the time frame felt suspicious. Particularly given my previous attempts to do something virtually the same, for a different purpose.
I mean it probably has a whitelist/blacklist or some other kind of system in place to prevent it from accessing malicious or illegal content or people going around it's filters or trying to jailbreak the model. So it might not work on just any site, or have some way to detect the difference between a site on the open web meant for people at large, and a single purpose hosted website.
The public server that the model found the emails in my example was a .gov domain so maybe that makes it seem more legit. idk,
I also know that some of them respect robot.txt and won't crawl anything with that, at least actively.
There's been sites on the open web that ChatGPT hasn't been able to access for various reasons. I usually get around this by printing the page to PDF then uploading it.
Yes, large language models have a cutoff date, but they are often given additional information to work with. In the case of Google search, it's doing a search and giving the model the contents of the web pages that were found, which it is then a bit too gullible about repeating.
You can think of the AI as a fancy web search tool.
This matches with my experience (perplexity, gemini, claude, and others). When using web search, the models are instructed to believe what they find. So if bad/malicious results pop up in their discovery, they can't tell the difference between them and legit results.
I had already assumed that this was being abused somehow, but it's still funny/worrying/depressing/more-adjectives seeing it in action in such an absurd way
My first thoughts were both that this is zero percent surprising, and also that I think it worked mostly because it is a silly result. The author's example filled an information vacuum that they themselves created. I'd be more interesting in seeing how easy or difficult it would be to create enough disinformation to change an existing answer.
This bit near the end of the article was interesting though:
But Ray says that's the whole point. Google itself says 15% of the searches it sees everyday are completely new. And according to Google, AI is encouraging people to ask more specific questions. Spammers are taking advantage of this.
Yeah, not to give anyone ideas but one of my AI-related concerns, should it become good enough in programming, is that scammers can implement real-time monitoring of online search trends and spit out websites tailored to fit them. Once perfectly efficient, the system could even produce a tailor-made website for each individual web search (in my imagination at least, but I'm not an IT pro and I hope I'm wrong about this).
ETA: And not just scammers but propagandists and other nefarious actors too.
That's already happening though. It's not a real-time reaction, but the amount of clearly AI generated slop I get when searching for some things is ridiculous. It is stupidly cheap to generate a 100x static pages that rephrases existing content and host it on 100 domains that you can then farm ad revenue from.
The most annoying example of this I've found happened before I set up a lens in kagi for gardening.
What's the germination temperature of a tomato?
At least they usually make an outline.
What is a tomato?
Are tomatoes poison?
What does tomato taste like?
Why grow tomato?
Does tomato taste good?
&etc
Even if the answer is on that page I don't trust it. It feels like the ultimate breakdown of stumbleupon, except when I used that at least I wasn't looking for a specific answer.
You're right, but at least the slop pages can still be easily detected due to the poor quality and weird domain names. It's really annoying as it is, but I fear how much worse it can get.
SEO manipulation never really got the attention that it should have IMO. And this is an evolution/corollary of that.
If someone maliciously switches out a physical street sign from "Main" to "Martin Luther King Drive", that gets identified and fixed pretty quickly. On lesser known side streets, maybe an inaccurate sign could stay up longer. But it still garners a response for the most part.
The fact that we're relying upon Google to fix similar issues in its search results out of the goodness of its heart (or whatever sliver of stewardship remains in the company) is ridiculous.
Did you know that anyone can buy a printer that will print whatever dangerous nonsense they want to write? They can even put a fake letterhead on top that says "super legit trustworthy source." They can then give that, real, actual, paper to anyone and act as if it's the truth! Damn your hubris, Gutenberg!
We should all know by now that you can't trust anything an AI says. You can't trust anything. That should be our default at this point, right? Truth is hard.
AI is useful, flawed, and complicated. But at this point, "you can't trust it" isn't that interesting beyond educating people who are just starting to use it, I guess.
Just last week I met someone who doesn't trust any "big media" but trusts AI answers. So yes absolutely preaching to the choir here but ... there are people out where who still needs to hear it.
It gets ignored regularly every day. Every god damned day I have to spend time researching and refuting AI slop from people I work with.
It's a never ending battle, because it's a lot quicker to generate slop than it is to validate it, so if the trend continues, it will eventually be what I spend all of my time doing.
Somehow, people have internalized an idea that because it's a computer, it's never wrong. Like they have access to a crystal ball that knows everything, but no one else has figured out that it exists yet. It's become a little insulting, because the subtext is that I somehow haven't thought about using AI to answer this question we have. I've thought of it, obviously, and don't use it because it's mostly bad at answering complex questions accurately.
I don't need it for simple questions either, because I already know the answers to those.
I got curious about the hula hooping traffic cops, so here's the blog post! Found by asking Google AI about the best hula hooping officer. Gotta say, I feel like this would actually be really fun to see.
That said, I always check the links on Google's AI (well, besides super basic queries) because I've found that the results can vary in accuracy. Most notable example to come to mind isn't even deliberate misinformation like this article. I was trying to find how to start a new save file in a game, and for some reason the AI pulled an answer from the wiki for a totally unrelated game. A totally wrong answer, mind you: the AI said you could select "new game" on the start menu. In reality, you have to go into the files on your computer/device and manually delete the existing save folder.
Looking it up now, it has since been fixed and pulls the accurate information from a Steam community post. I still have no clue how that other game's wiki even ended up being used by Google's AI, because I don't think that page even showed up on the first page of results. At the very least, the Steam community post was definitely higher. So, I'm also wary of how Google's AI decides to rank results when getting an answer.
[OP] rkcr | a day ago
...
goose | a day ago
I'm curious about the accuracy of this claim. My understanding is that models have a knowledge date cut off that they draw from. So in most cases, you can't just plug in a URL and have the LLMAI crawl the page, it's essentially drawing from its snapshot, similar to waybackmachine. Not that the claim can't be true, but a 24 hour turnaround seems pretty fast for a model update.
whbboyd | 23 hours ago
The model—the ginormous matrix that encodes the training data—is extremely expensive to generate, and has a cut-off as of the point at which the organization generating it stopped scraping training data and started training the model, something which they do, idk, every few months at most (because of the extreme expense). With no additional "context" (i.e. token string prefix), this is all the model "knows".
However, most model operators provide mechanisms for their models to trigger web requests and inline the responses into their token prefix. This is what happened here: the models don't "know" anything about hot-dog eating tech reporters, but when asked, they search for it, inline the author's blog post into their token prefix, and then repeat it with minor paraphrasing.
goose | 19 hours ago
Sure, but still, overnight?
Looking at the details page for Google's Gemini v3, it lists the model cutoff date as January 2025, with its last knowledge update being November 2025: https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-preview
Even with period updates from scraping, that feels suspiciously fast. Asking about what happened in my local township last week, it references a mayoral election that occurred back in October. And that was widely covered (by local news, anyways).
smores | 18 hours ago
Not overnight, on demand. When you ask Claude or ChatGPT about a real-world fact that isn't in their training data, they often make an ad hoc web search and try to summarize the results for you. You can see this happening, there will be a visual indicator (Claude shows a little loading indicator that says something like "Web search...").
umbrae | 7 hours ago
Agree with you, and further, I hate to be the “read the article” guy, but the article specifically and directly addresses this within the first few paragraphs, @goose -
goose | 7 hours ago
Sure, my point was, I've tried this experiment myself and was not able to reproduce the same results, which led me to doubt the veracity of the claims. On the other hand, it's been probably over a year since I tried it, and in a business where changes are fast it wouldn't surprise me if methods have changed. But, at the time I tried it, I was unable to get the LLMAI to scrape the page(s) I was trying to get it to scrape.
papasquat | 5 hours ago
That probably has more to do with search engine crawling than the AI model. After all, the model is executing a web search exactly in the same way that a user would. So if you write an article and you can't find that article with a very specific web search in the first few results, neither will an LLM. If you are able to find it in the first few results, so will an LLM.
The success of the attack depends on the success of traditional SEO techniques.
CrypticCuriosity629 | 19 hours ago
It's complex. Yes you can ask an ai to crawl a page and it will. AI agents will do this scarily well and do it in a much more complex way.
I asked the ChatGPT agent today to find the email of the VP of an organization I was trying to get a sales lead on, and it ended up finding an obscure permitting request document posted on a public facing server from a business in Alaska who had a couple saved emails with this organization attached to their request, including the email address I needed AND an email signature with direct phone line and extension, all from like a week ago.
I got the sales lead at least!
These models also have an updated general memory that contains recent events, and it will do a web search for more information in a lot of cases.
Models haven't really done that whole "My training data only extends to this date" thing in a while because they integrate different technologies to keep information fresh.
And some of these memory vectors can be updated by the models themselves in some cases. Because why wouldn't an AI automation company use AI automation to automate updating their automation! Hahah
goose | 18 hours ago
See, this is where my experience differs. I explicitly asked it to crawl a page on my webserver to answer a question, as I was curious to see what UA and IP address it would come from.
Despite asking a few ways and sending a few different unique URL's, nothing came up in my nginx logs. In hindsight, perhaps I should have paid attention weeks or months later to see if the unique URL's I sent to the LLMAI did ever pop up in my nginx logs, but that was a while ago.
It's all very "black box", I think, nobody wants to admit how the inner workings actually work. Sure, they're data vacuums, but how they interact with the outside world seems a mystery for now.
creesch | 14 hours ago
What is "it" in this cases? People have been replying about "models" to you but that isn't entirely correct. It is the combination of the model and the service it is tied into. If you run a local model with a barebones chat client it will not search for example. But if the chat interface allows the model to use web search or browse the internet then the model will be able to utilize this. It often is something you need to explicitly enable
thumbsupemoji | 7 hours ago
Yeah it can vary widely depending on what @goose is utilizing, even from day to day—Gemini is maybe the most consistent, but even then something will trigger something else or the giant machine brain is just having an off day and my convo will end with a "Sorry, I'm not feeling it," and then the next day it's fine lol
tanglisha | 6 hours ago
Are you using Cloudflare? I've noticed a big increase in human checks in the last few months and they always seem to be Cloudflare branded.
goose | 5 hours ago
Good thought, but no, I don't have any CDN in front of my services. Just good 'ole
nginx, hanging out there on ports80/443, with myA/AAAArecords pointing directly to it.papasquat | 5 hours ago
Every site on the internet is cached by so many different services in so many places that an LLM scraping a site may not result in a 1:1 web request. In the case of Gemini, they already run a massive web crawler, so it would make sense to not make an additional web request each time AI prompted for it.
Aside from them I'd imagine all of the large AI companies run their own caching services for that purpose. At a certain scale it becomes more cost effective to cache the whole internet every day rather than paying for the extra bandwidth and other considerations that come from trillions of web requests per hour.
goose | 5 hours ago
Responding to both your comments in this one:
Sure, this makes sense, particularly in the scope of resource efficiency. In my case, because I run my own webserver and manage my own files, I can set the rules in my
/robots.txt, as well as analyze myaccess_loganderror_logfromnginxto determine what IP addresses and user agents are accessing what, and when.Google certainly crawls my domains, as well as all the other major players. They even respect my
/robots.txt, unlike certain other crawlers (Looking at you, Yandex). My entire hang up is that the author claims to have influenced the model's knowledge base by planting false information, over the course of a single night. I'm not saying that can't be done -- just that the time frame felt suspicious. Particularly given my previous attempts to do something virtually the same, for a different purpose.CrypticCuriosity629 | 4 hours ago
I mean it probably has a whitelist/blacklist or some other kind of system in place to prevent it from accessing malicious or illegal content or people going around it's filters or trying to jailbreak the model. So it might not work on just any site, or have some way to detect the difference between a site on the open web meant for people at large, and a single purpose hosted website.
The public server that the model found the emails in my example was a .gov domain so maybe that makes it seem more legit. idk,
I also know that some of them respect robot.txt and won't crawl anything with that, at least actively.
There's been sites on the open web that ChatGPT hasn't been able to access for various reasons. I usually get around this by printing the page to PDF then uploading it.
skybrian | 18 hours ago
Yes, large language models have a cutoff date, but they are often given additional information to work with. In the case of Google search, it's doing a search and giving the model the contents of the web pages that were found, which it is then a bit too gullible about repeating.
You can think of the AI as a fancy web search tool.
JCAPER | 12 hours ago
This matches with my experience (perplexity, gemini, claude, and others). When using web search, the models are instructed to believe what they find. So if bad/malicious results pop up in their discovery, they can't tell the difference between them and legit results.
I had already assumed that this was being abused somehow, but it's still funny/worrying/depressing/more-adjectives seeing it in action in such an absurd way
Omnicrola | a day ago
My first thoughts were both that this is zero percent surprising, and also that I think it worked mostly because it is a silly result. The author's example filled an information vacuum that they themselves created. I'd be more interesting in seeing how easy or difficult it would be to create enough disinformation to change an existing answer.
This bit near the end of the article was interesting though:
Lia | 11 hours ago
Yeah, not to give anyone ideas but one of my AI-related concerns, should it become good enough in programming, is that scammers can implement real-time monitoring of online search trends and spit out websites tailored to fit them. Once perfectly efficient, the system could even produce a tailor-made website for each individual web search (in my imagination at least, but I'm not an IT pro and I hope I'm wrong about this).
ETA: And not just scammers but propagandists and other nefarious actors too.
Omnicrola | 9 hours ago
That's already happening though. It's not a real-time reaction, but the amount of clearly AI generated slop I get when searching for some things is ridiculous. It is stupidly cheap to generate a 100x static pages that rephrases existing content and host it on 100 domains that you can then farm ad revenue from.
tanglisha | 4 hours ago
The most annoying example of this I've found happened before I set up a lens in kagi for gardening.
What's the germination temperature of a tomato?
At least they usually make an outline.
&etc
Even if the answer is on that page I don't trust it. It feels like the ultimate breakdown of stumbleupon, except when I used that at least I wasn't looking for a specific answer.
Lia | 24 minutes ago
You're right, but at least the slop pages can still be easily detected due to the poor quality and weird domain names. It's really annoying as it is, but I fear how much worse it can get.
SleventhTower | 23 hours ago
SEO manipulation never really got the attention that it should have IMO. And this is an evolution/corollary of that.
If someone maliciously switches out a physical street sign from "Main" to "Martin Luther King Drive", that gets identified and fixed pretty quickly. On lesser known side streets, maybe an inaccurate sign could stay up longer. But it still garners a response for the most part.
The fact that we're relying upon Google to fix similar issues in its search results out of the goodness of its heart (or whatever sliver of stewardship remains in the company) is ridiculous.
lelio | 19 hours ago
Did you know that anyone can buy a printer that will print whatever dangerous nonsense they want to write? They can even put a fake letterhead on top that says "super legit trustworthy source." They can then give that, real, actual, paper to anyone and act as if it's the truth! Damn your hubris, Gutenberg!
We should all know by now that you can't trust anything an AI says. You can't trust anything. That should be our default at this point, right? Truth is hard.
AI is useful, flawed, and complicated. But at this point, "you can't trust it" isn't that interesting beyond educating people who are just starting to use it, I guess.
chocobean | 10 hours ago
Just last week I met someone who doesn't trust any "big media" but trusts AI answers. So yes absolutely preaching to the choir here but ... there are people out where who still needs to hear it.
culturedleftfoot | 16 hours ago
Expect this advice to be as routinely ignored as "don't cite Wikipedia."
papasquat | 5 hours ago
It gets ignored regularly every day. Every god damned day I have to spend time researching and refuting AI slop from people I work with.
It's a never ending battle, because it's a lot quicker to generate slop than it is to validate it, so if the trend continues, it will eventually be what I spend all of my time doing.
Somehow, people have internalized an idea that because it's a computer, it's never wrong. Like they have access to a crystal ball that knows everything, but no one else has figured out that it exists yet. It's become a little insulting, because the subtext is that I somehow haven't thought about using AI to answer this question we have. I've thought of it, obviously, and don't use it because it's mostly bad at answering complex questions accurately.
I don't need it for simple questions either, because I already know the answers to those.
CannibalisticApple | 23 hours ago
I got curious about the hula hooping traffic cops, so here's the blog post! Found by asking Google AI about the best hula hooping officer. Gotta say, I feel like this would actually be really fun to see.
That said, I always check the links on Google's AI (well, besides super basic queries) because I've found that the results can vary in accuracy. Most notable example to come to mind isn't even deliberate misinformation like this article. I was trying to find how to start a new save file in a game, and for some reason the AI pulled an answer from the wiki for a totally unrelated game. A totally wrong answer, mind you: the AI said you could select "new game" on the start menu. In reality, you have to go into the files on your computer/device and manually delete the existing save folder.
Looking it up now, it has since been fixed and pulls the accurate information from a Steam community post. I still have no clue how that other game's wiki even ended up being used by Google's AI, because I don't think that page even showed up on the first page of results. At the very least, the Steam community post was definitely higher. So, I'm also wary of how Google's AI decides to rank results when getting an answer.