Cloudflare plans product where website owners can charge AI bots for scraping
Cloudflare plans to launch a marketplace in 2025 where website owners can sell access to AI model providers for scraping their content. This initiative aims to give publishers more control over how AI bots use their data and compensate them for it. In preparation, Cloudflare has introduced free AI Audit tools, allowing users to track, block, or selectively permit AI bots that crawl their sites.
The marketplace would allow smaller publishers to strike deals with AI companies, similar to agreements already made by larger outlets like Reddit and Quora. This would help balance the current situation where AI models freely scrape websites without compensation, potentially undermining their traffic and business models.
Cloudflare CEO Matthew Prince emphasizes that without paying content creators, AI ecosystems risk becoming unsustainable. While the marketplace concept is still being developed, the goal is to let websites charge AI providers either a monetary fee or request credit for using their content. However, exact payment terms remain unclear. This move comes in response to increasing frustration from website owners who feel their content is being exploited by AI bots without fair compensation.
