Hackers influence ChatGPT and Gemini to direct users to scam centers

Think you can trust AI to give you correct answer? Think again.

ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers.

Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.

The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.

This isn’t research or a proof of concept. It’s happening in the wild now, carried out by malicious actors, through automated campaigns operating at a scale traditional takedowns can’t keep up with.

Social engineering is no longer targeting humans. The new targets are AI agents browsing the web.

This is part one of our blog series discussing the implication of AI Disinformation attacks.

Sit back and let’s dive in.

Intro

Over the last year, we got lazy. Yes, even lazier than before. We all have a new best friend that does everything for us and seemingly knows everything.
But did you ever stop and think: How does AI know everything? Where is all this knowledge actually coming from? The answer is obvious — the internet. So, the AI’s answers are entirely shaped by the web.

But wait… anyone can put content on the internet. Who’s guaranteeing that all of it is actually valid? What happens if that content is intentionally poisoned?

That led me to a question: could an attacker craft content so perfectly that they could dictate the behavior of AI agents browsing the web?

A study by Exploding Topics reported:

“92% Of users don’t verify AI answers”

That means that If attackers control AI, and we blindly trust it, they essentially control us.

Content Promotion: the attacker’s version

So, how does AI actually decide which facts to trust and which to ignore?
Some of you might know the term GEO (Generative Engine Optimization), which is basically traditional SEO evolved for the AI era. In short, GEO is how an AI decides which sources to cite.

There are plenty of GEO tricks to make AI favor certain pieces of content over others. For example, throwing in statistics to back up claims or adding quotes from recognized experts to boost an agenda. These techniques simply make the AI feel more confident about specific data points. Many marketers use GEO to make AI cite their brand.

This got me thinking… so from an attacker’s perspective, what happens if I put false information online and use GEO techniques to make an LLM believe it? Can I make it say whatever I want?

AI information supply-chain attack

So can I use the marketing techniques to conduct cyber attacks?

I started to do some experiments to see what’s possible. I was able to conduct some really crazy attacks that we’ll elaborate on in the next chapters.
But something really bugged me — If I could do it, so could the attackers. So how do we find out if these techniques are already in the wrong hands?

The first thing that popped into my mind was phishing. The #1 motivation for attackers is financial, and the most common attack vector is phishing. What if attackers could change the contact details of organizations, such as banks, when people are searching for them using AI?

Detection system

To check if there are real GEO attacks out there, we built a system that analyzes AI answers across Gemini, ChatGPT, and Google AI Overview (Google’s top search result), and the sources they rely on. The system scores the sources and the content that is fetched from them, and looks for any GEO techniques used to push disinformation. When we detected a false detail (phone number, URL, email address, etc) being cited, we marked it as an incident. We ran the system on Fortune 100 companies, banks, airlines, software companies, and more.

Attacks in the wild

The results were astonishing. We found real attack campaigns utilizing posts, documents, websites, images that contain GEO-optimized content that pushes disinformation, like fake contact information, support numbers, email addresses, fake software updates, login pages, and more.

In total, we detected 374 companies that are attacked. Including Fortune 100 organizations, major airlines (Delta, Lufthansa, United Airlines, Emirates, Qatar and more), top banks (Bank of America, Wells Fargo, Chase, Citi, etc), travel platforms (Airbnb, TripAdvisor), various software providers, and more.

The attacks span across ChatGPT, Gemini and Google AI Overview, with tens of thousands of malicious pages detected.

Here are some examples of attacked queries that cite scam phone numbers:

*Important to say: most of the incidents were statistical. As LLM results are inconsistent, we saw that the attack (LLM citing the fake number) did not reproduce every time we ran the query. The better the attacking content, the higher the chance of the poisoning to work.

Poisoning posts

The posts affecting AI were carefully crafted with GEO payloads, and were embedded across innocent websites which allow the creation of User Generated Content (UGC):

1. Social media/forum posts — Instagram, Tumblr, Buzzfeed, LeetCode, YouTube & Vimeo video descriptions, Medium, and more
2. Websites that allow file uploads (mostly PDFs) — Government sites, universities, colleges, and more
3. Website hosting platforms — Google sites, GitHub pages, Wordpress, blogspot.com
4. Fundraisers, Job Search, Event platforms — Posh.vip, onecause.com, bebee.com, raiselysite.com, and more
5. Random websites abused for payload delivery — Yelp reviews, Apple Maps reviews, Mental help foundation (mentalhealth.org.nz), National Cancer Research Foundation (nfcr.org), PC reviews website (pcreview.co.uk), and many many more!

Here are some examples:

Attack Techniques Breakdown: How does it work?

The payload content the attackers used may seem random at first, but it is carefully structured. Here are some of the techniques the attacker uses to make these payloads hijack the LLM answers:

1. GEO-optimized content: The posts repeat the fake number, use question-and-answer sections, and add phrases like “call now,” “24/7,” or “updated 2026.” This makes the number more likely to appear in search results and AI answers.
2. Designed to slip past defenses: They write the same number in many ways: using spaces, dots, emojis, Unicode characters, or words between digits. There are spam/scam filters that remove phone numbers. These filters may see these as different text or fail to detect a phone number, while AI can still understand it, and interpret correctly after tokenization, as it is semantically the same (1️⃣ == 1, ☎️ == phone).
3. Posted everywhere: Attackers copy the same story across social media, forums, websites, and PDFs. They also upload it to trusted .edu, charity, and business sites, making the fake information look widely confirmed and authoritative.
4. Made to look real: The posts mix useful advice and real phone numbers with the fake numbers to cause the LLM to cite both. Official-looking AI-generated images, fake likes, comments, and support labels make the scam appear more trustworthy, and make the LLM cite the fake number alongside real information.
5. Built to create panic: The content focuses on urgent problems like refunds, canceled flights, locked accounts, or money transfers. The goal is to make stressed AI users call quickly instead of checking the company’s official website.

Automated attacks → Bypass of takedown mechanisms

There are many solutions that would automatically take down some of these posts. But that approach won’t work anymore. These attacks were clearly automated: same templates, same phone numbers, same phrasing across many victim companies and websites at a scale too large for manual posting — hundreds of posts per day per platform per company → thousands of daily posts.
Trying to take all of these posts down is a lost cause. While one page is taken down (it can take hours or days at a time), a thousand new ones are created.

In addition, websites like archive.org, which store content of deleted web pages, are still indexed to the search engines. That means that even after takedown, the poison is still effective.

Example: LeetCode attacker posts targeting American Airlines from the last 24 hours take up more than 10 result pages in Google (!), and a PDF that is still up on web archive after takedown:

This problem requires a new approach.
We have developed a mechanism to revert back the disinformation attacks and return the truth to the AI answers automatically, based on each attack detected. We will elaborate on it in the next chapters.

Responsible Disclosure

We reached out to all the targeted companies through their bug bounty, fraud, and security channels. Most of them were not aware of the magnitude of these attacks and their effect. We started working with some of them to mitigate the threat.
Others brushed it off, either claiming their current solutions handle it (obviously they don’t) or saying it’s out of scope for the bug bounty programs since it doesn’t hit their internal systems directly.
Here’s why they should actually care:

• Lost revenue: Scammed customers get hijacked when attempting to purchase, stealing direct revenue.
• Brand Damage: Mad victims blame the real company, hurting trust and driving people away.
• Headaches & Legal Fees: Cleaning up massive customer disputes and potential lawsuits gets expensive fast.

We are expecting more and more security teams to get familiar with this issue soon and attend to it, especially at companies with big brands.

We also contacted Google and OpenAI through their bug bounty channels. Google classified the issue as out of scope, stating that AI-generated misinformation and phishing/social-engineering scenarios are not covered by its VRP. OpenAI closed the report as not reproducible, stating that submissions must demonstrate proven impact to users rather than a theoretical attack, even though the report included real examples of attacks.

So because the attack manipulates what the AI shows users rather than breaching the AI vendors’ own systems, it falls outside how both programs currently define a vulnerability.

Summary

AI Disinformation is a disease that will become a plague.
These attacks will get worse because one fundamental fact is not going to change — AI cannot distinguish between between truth and lies.
Throughout this blog series we will dive into multiple attack use cases that are caused by AI disinformation.
In the next chapter, we’ll discuss how AI disinformation can lead to software supply chain attacks. Stay tuned.

But do not panic — my team and I already have a solution. If you’re interested in hearing more about that, or getting insights as to whether or not your company is under attack, reach out to us at contact@vigilance.security.

Reach out anytime

We are Ariel Simon, Naor Haziz, Dan Lasker, Roi Levy and Ben Silvan— security researchers and entrepreneurs.
For asking any question, collaborating on a research, or just chat, please reach out anytime via LinkedIn :)