Cloudflare pilot lets ChatGPT index hundreds of millions more pages

Cloudflare sits in front of roughly one in five of the internet's websites, and it has spent the past year deciding who gets to crawl them. Per Search Engine Journal, a pilot between Cloudflare and OpenAI has let ChatGPT's search function index hundreds of millions of additional pages, using network-level signals, presumably about crawler behaviour, bot verification and site responsiveness, that Cloudflare alone can see. The headline number sounds like a win for the open web. It is actually a quiet admission of how badly broken web indexing for AI has become, and a preview of who will control the toll booth.

The crawl problem nobody fixed properly

ChatGPT's search product has always had a thinner index than Google's. Not because OpenAI lacks ambition, but because crawling the web well is extraordinarily hard: you need to distinguish real pages from spam, respect robots.txt without being trivially blocked by it, verify that a bot claiming to be GPTBot actually is GPTBot, and do all of this at a scale that does not melt smaller publishers' servers. Cloudflare has effectively been running this infrastructure for the entire internet for fifteen years, mostly to stop DDoS attacks and scrapers. It turns out the same plumbing that identifies malicious bots can identify legitimate ones, and that data is valuable enough that OpenAI wanted direct access to it rather than crawling blind.

That is the real story here: AI search engines no longer trust their own crawlers to find the whole web. They are buying (or partnering for) better eyes.

Why a bigger index changes who gets cited

A larger index is not neutral for brand visibility. It changes the competitive set. When ChatGPT's search index was thinner, a narrower band of large, well-optimised, frequently-crawled domains dominated citations almost by default, because the model had fewer alternatives to draw on. Widen the index by hundreds of millions of pages and the competition for any given query answer gets more crowded, not less. A financial services firm with a strong but narrow web presence, think a handful of flagship reports and a well-maintained newsroom, could previously count on being one of the few relevant sources ChatGPT had actually seen. Now it is competing against niche analysts, regional outlets and specialist blogs that were simply invisible to the index before.

For multilaterals and policy institutions this cuts both ways. Bodies like UNDRR or CGAP publish enormous volumes of technical documentation, working papers and country reports that previously sat outside AI crawlers' effective reach, blocked by paywalls-adjacent architecture, buried in PDF repositories, or simply never prioritised by a crawler optimising for popular domains. A wider net means more of that institutional output becomes citable. But it also means the advantage of having been one of the few authoritative sources ChatGPT could find erodes. Authority has to be earned on content quality now, not merely on having been indexed at all.

The access question Cloudflare is quietly answering

The more consequential detail is structural, not numerical. Cloudflare has positioned itself as an intermediary that decides, on commercial terms, which AI companies get privileged access to crawl signals and which do not. That is a different model from the open, permissionless crawl that built Google's index over two decades. If Cloudflare's relationship with OpenAI becomes the template, publishers' visibility in ChatGPT's answers increasingly depends on infrastructure decisions made between two companies, not on the publisher's own robots.txt file or sitemap hygiene.

Industrial groups and large enterprises who sit behind Cloudflare, and a great many do, should treat this as an infrastructure audit item, not a marketing one. The question is no longer just "is our content good enough to be cited." It is "does our CDN configuration, bot-management policy and crawler allowlist actively permit the signals that get us indexed in the first place." A communications team optimising content for AI visibility while an IT team blocks or throttles the crawlers that feed that visibility is now working against itself, and most large organisations have no process that checks for that contradiction.

The index is getting bigger. The gatekeepers are getting fewer and better capitalised. Brands that assume visibility is purely an editorial problem are about to discover it is also a network-configuration one.