Study finds API-based tools can't reliably mimic ChatGPT citations

Every rank-tracking dashboard that promises to show "how your brand appears in ChatGPT" is, strictly speaking, lying to its customers. It is showing them something else: how a brand appears when queried through OpenAI's API, a different product, running on different retrieval logic, built for developers rather than consumers. Search Engine Journal reports on a new paper that tested whether this substitution holds up. It mostly doesn't.

The researchers compared citations from the actual ChatGPT consumer product against two common proxies: Google search results and OpenAI's API. Both proxies identified the right domains ChatGPT cited with reasonable frequency. Neither reliably identified the right pages. A tool might correctly flag that ChatGPT leans on, say, McKinsey.com for a given query, while missing entirely which McKinsey report, page, or paragraph the model actually pulled from and presented to the user.

Why the gap exists, and why it's structural, not fixable

This is not a bug that a software update quietly resolves. ChatGPT's consumer interface sits on retrieval infrastructure, ranking signals, and conversational memory that the public API does not expose and was never built to expose. Treating the API as a stand-in for the product is like inferring what a chef served a specific table by reading the restaurant's wholesale ingredient invoices. You'll learn the kitchen stocks salmon. You won't learn which diner got it, how it was plated, or whether the chef ever used it that night.

Google-result matching fails for a related but distinct reason. ChatGPT's citation behaviour is not a repackaged SERP. OpenAI has built its own retrieval and ranking logic on top of (and sometimes independent of) web search, which means the correlation between "ranks well on Google" and "gets cited by ChatGPT" is looser than the AI-visibility industry has been pricing in.

This matters more now than it would have eighteen months ago, because an entire category of "generative engine optimisation" tooling has been sold to enterprise marketing teams on the implicit premise that API access equals visibility truth. CMOs at banks, insurers, and multilateral agencies have been buying dashboards that report, with false precision, exactly which pages their organisation's content occupies inside ChatGPT's answers. Per Search Engine Journal's account of the paper, that precision is largely theatre at the page level, however sound it may be at the domain level.

Domain-level signal is still worth something, just not the thing being sold

The practical reframe for brand teams: domain-level matching, which the paper found to be comparatively reliable, is still useful intelligence. If a tool shows that ChatGPT frequently cites a competitor's domain rather than yours on a given topic cluster, that is a real and actionable signal about where authority currently sits. What the tool cannot responsibly claim is that it has identified the specific page ChatGPT is drawing from, or that optimising that exact page will move the needle on the next citation. That is the layer at which current proxy tools, by this paper's own evidence, lose fidelity.

For financial-services communications teams tracking whether ChatGPT cites their research notes over a rival bank's, the distinction is not academic. A procurement or compliance stakeholder asking "are we visible in ChatGPT" deserves an answer with an honest confidence interval, not a page-level heat map built on data the model never generated. The same applies to multilaterals and development institutions, where a single mischaracterised citation, attributed to the wrong page of a policy brief, can misrepresent an organisation's position in exactly the kind of high-stakes, low-volume query where AI search increasingly mediates first impressions.

The sharper implication is for vendors, not just buyers. Any AI-visibility platform that reports page-level ChatGPT citation data sourced primarily from the API, without disclosing that limitation, is selling a measurement it cannot support. Enterprises spending six-figure sums on these tools should ask a blunt diligence question before the next renewal: does this platform's data come from the consumer product, the API, or an inference built on Google rankings? The paper gives the field a benchmark for honesty. Expect the vendors with nothing to hide to start volunteering the answer unprompted, and the rest to keep the methodology section vague.