Cloudflare said on 4 August that it has removed Perplexity AI from its verified-bots programme and is now blocking the company’s traffic after detecting what it calls “stealth crawling” across tens of thousands of websites. According to a technical blog post, Perplexity’s declared bots generate about 20–25 million requests a day, but an additional 3–6 million requests originate from undeclared crawlers that impersonate a Chrome browser, rotate IP addresses and switch autonomous system numbers to bypass robots.txt files and Cloudflare firewall rules. Engineers created brand-new, non-indexed test domains that explicitly disallowed automated access and still found Perplexity could return detailed information from those sites. Cloudflare said the behaviour violates long-standing internet norms that require crawlers to identify themselves and honour site-owner preferences; compliant bots such as OpenAI’s were able to detect the blocks and ceased crawling, the company added. Perplexity disputed the findings, calling Cloudflare’s analysis a “sales pitch” and insisting the cited traffic does not belong to its service. The San Francisco-based start-up, founded in 2022 and valued at about $18 billion after a $100 million fund-raise last month, says its agents fetch content only in response to real-time user queries. Cloudflare has deployed new signature rules to detect the alleged stealth activity and is encouraging publishers to use its pay-per-crawl marketplace or free blocking tools to curb unwanted scraping.
Perplexity AI clarifies that its user-driven agents differ from bots by fetching web content only for real-time user queries without storage, countering Cloudflare's mischaracterisation as malicious scrapers and alleged confusion with BrowserBase traffic. https://t.co/EbP5HxdVmt
Perplexity defends its AI assistant technology against accusations from Cloudflare, which labeled Perplexity as engaging in "stealth crawling." The post clarifies that modern AI agents like Perplexity operate on real-time, user-driven requests rather than traditional web https://t.co/ZpoWpJfXeE
Vercel CEO on Cloudflare blocking Perplexity https://t.co/f9W2VBB98n