Your "AI Visibility Score" Might Be Lying to You — Here's Why

Have you noticed that when you run the same question through ChatGPT twice, back-to-back, you get two different sets of sources cited? Not because anything changed in the world between the two queries — just because that's how these models work. Now imagine building your entire AI-SEO reporting dashboard around a single one of those runs.

That's exactly the problem a new research paper is putting numbers on, and it's worth every Indian marketer's attention before you sink another rupee into "AI visibility tracking" tools.

The Core Problem: One Query Isn't a Measurement, It's a Guess

Most AI visibility dashboards work like this: they ask an AI engine a question once, note which sites got cited, and hand you a tidy percentage — "You appear in 12% of ChatGPT answers for this keyword." It looks scientific. It isn't.

Generative engines are built to inject randomness into their answers. Ask the same question ten times and you may get ten slightly different citation lists. A single snapshot tells you almost nothing about who's "really" ahead — it just tells you what happened on that one roll of the dice.

Researcher Ron Sielinski, co-founder of the visibility-tracking company IQRush, put this to the test in an earlier study on running-gear queries in SearchGPT. Two competing publishers — Tom's Guide and Runner's World — showed up in roughly 9.5% and 6% of citations respectively. Looks like a clear win for Tom's Guide, right? Except the margin of error on those numbers was wide enough that the actual gap could easily be zero. On paper, one site "won." In reality, the data couldn't support that claim at all.

So When Can You Actually Trust a Ranking?

Sielinski's newer paper (not yet published, but reviewed ahead of release) tries to answer a more useful question: how many times do you have to ask before a ranking becomes meaningful instead of noise?

The answer needs two things to be true at once:

1. The order has to stop shuffling. Early on, every new answer can reshuffle who's "on top." Only after enough answers pile up does a real leader start to separate from the pack.
2. The gap between competitors has to be bigger than the error margin. If two sites are neck-and-neck, no amount of extra data will make one a clear winner — because there isn't one.

Across 30 different platform-and-topic combinations tested, the number of citation-bearing answers needed before a ranking became trustworthy ranged from 33 to as many as 94. And in three of those 30 cases — all on SearchGPT — even 125 questions weren't enough to separate the top sites. Sometimes the honest answer really is "we don't know yet," and a good tracker should be willing to say that instead of forcing a confident-looking number.

This Isn't an Isolated Finding

If this sounds familiar, it's because SparkToro flagged something similar back in January: ask an AI assistant for brand recommendations on the same query, and it changes the list more than 99% of the time. That study raised the obvious next question — how many repeats do you need before the answer settles down? Sielinski's work is the clearest attempt yet to answer it.

It also isn't just one lab making this claim. A separate team of researchers at the University of St. Gallen ran their own independent dataset earlier this year and landed on the same conclusion: a single AI query is not a reliable measurement, full stop. When two unrelated research groups arrive at the same answer using different data, that's worth paying attention to.

What This Means If You're Running Campaigns in India

For brands and agencies here that have started reporting "AI visibility" alongside traditional SEO metrics, this changes how you should read your own dashboards:

Ask your tracking vendor one blunt question: does the number they show you come from repeated sampling with a reported range, or from a single pull dressed up as a clean statistic? A suspiciously tidy number is a red flag, not reassurance.

Don't celebrate small movements. If your AI citation share moved from 8% to 11% after a content update, that swing can easily sit inside normal run-to-run noise. Measure before and after multiple times each before you credit your own work for the change.

Different platforms need different sample sizes — and not in the direction you'd expect. Gemini tends to stack several citations onto the same handful of sites within one answer, so extra citations there don't add much new information. SearchGPT spreads citations more widely across an answer, so each response actually tells you more — but a chunk of SearchGPT queries return no citations at all, so you need to ask more questions overall to collect the same amount of usable data.

Trust the top of the list, question everything below it. Even in the cleanest cases, the study found real uncertainty around exact rank — a site sitting at position 6, for instance, could realistically be anywhere within about five spots in either direction, and one case in five had even wider swings than that. Leaders separate from the pack with enough data. The middle of the pack rarely does.

A Reality Check on the Research Itself

To be fair to the data: this is a preprint, not a peer-reviewed publication yet. It covers 30 tests across three AI engines, using AI-generated questions rather than real searcher queries, collected over one window of time. The specific numbers — 33 to 94 answers, five-position error margins — won't transfer exactly to your industry or your keywords. Treat them as a proof that the problem exists, not a formula to plug your numbers into.

The Bigger Shift Happening Here

AI visibility reporting is going through the same growing-up process that web analytics and ad measurement went through years ago — moving from a single confident-looking number to a range with an honest margin of error. The frustrating part is that the basic tools to make this easy aren't fully here yet. Google Search Console, for one, still doesn't break out which clicks originated from AI Overviews or AI Mode.

Until that changes, the responsibility sits with marketers: don't trust a single AI query as proof of anything. Run it more than once. Ask your tools where their numbers actually come from. And if a report can't tell you "we're not confident yet," treat that silence as a bigger warning sign than a bad number would be.

If you're building out AI-search strategy for a client or your own brand, this is exactly the kind of nuance DizitalAdda's SEO & Search AI training covers — reading AI visibility data the way it should be read, not the way a dashboard makes it look.

About the Author

Sapna

Sapna is a Content Writer and Digital Marketing Specialist at DizitalAdda with over 3 years of experience in SEO, content strategy, and writing about AI tools and emerging search trends. She covers topics across digital marketing, search engine optimisation, generative AI, and career guidance for students and professionals looking to build a future in the digital space. Based in New Delhi.