Cloudflare said on 1 July 2025 that every new domain it protects will block artificial-intelligence web crawlers unless site owners explicitly grant access or receive payment through a new “Pay per Crawl” marketplace. The change targets bots used to harvest data for training large language models and could alter how AI developers source information online. Branded “Content Independence Day,” the policy reverses decades of open web crawling. Cloudflare, whose infrastructure sits in front of roughly one-fifth of all internet traffic, noted that more than one million of its customers—about 35 % of the world’s 1,000 most-visited websites—have already opted to bar AI bots. Pay per Crawl lets publishers set a fee for each request from registered AI agents. Transactions rely on the long-dormant HTTP 402 “Payment Required” status code, with Cloudflare acting as the merchant of record and using its bot-management technology to verify legitimate crawlers. The programme is being rolled out as a private beta. Chief Executive Officer Matthew Prince said the move is necessary because current practices “strip-mine” content: Google now takes 18 pages for every visitor it sends to a site, while OpenAI’s ratio has grown to about 1,500 to 1. Without new incentives, he warned, creators will have little reason to keep producing material. Media and entertainment companies including Condé Nast, the Associated Press, Time, Universal Music Group, Reddit and Quora have endorsed the shift. The News Media Alliance called automatic blocking an overdue step toward protecting digital journalism and ensuring fair compensation. AI developers that wish to continue training on web data must register, authenticate and pay or risk exclusion. Analysts say the system could curb uncontrolled scraping, although unregistered bots may still attempt work-arounds that Cloudflare will have to police. The initiative arrives amid a wave of copyright lawsuits and licensing negotiations between publishers and AI labs. By converting crawler access into a billable event, Cloudflare positions itself at the centre of a nascent market that could redefine how original content is valued in the generative-AI era.