Cloudflare Tool Can Prevent AI Bots from Scraping Websites
September 25, 2024
Cloudflare has released AI Audit, a free set of new tools designed to help websites analyze and control how their content is used by artificial intelligence models. Described as “one-click blocking” to prevent unauthorized AI scraping, Cloudflare says it will also make it easier to identify the content bots scan most, so they can wall it off and negotiate payment in exchange for access. Helping its clients toward a sustainable future, Cloudflare is also creating a marketplace for sites to negotiate fees based on AI audits that trace cyber footprints on server files.
AI Audit allows content creators to see how AI model providers are using their content “and then take control of whether and how the models are able to access it,” Cloudflare explains in a news release.
The cloud connectivity firm’s marketplace concept will let rights holders “reliably set a fair price” for allowing AI companies to scrape their sites for model training and retrieval augmented generation (RAG).
The idea is to foster “a transparent exchange between the websites that want greater control over their content, and the AI model providers that are in need of fresh data sources, so that everyone benefits,” the networked distributor says.
The rise of generative AI has made it harder for creators to put a value on their content, Cloudflare writes in a blog post explaining the tools.
“Previously, sites could distinguish between approving access to helpful bots that drive traffic, like search engine crawlers, and denying access to bad bots that try to take down sites or scrape sensitive or competitive data,” Ars Technica reports, explaining that large language models (LLMs) and other generative tools have triggered “a murkier third category” of bots that don’t exactly fit into either category.
AI Audit is not a silver bullet. Ars Technica warns that “for any site operators tempted to toggle off all AI scraping, blocking the Googlebot from scraping and inadvertently causing dips in traffic may be a compelling reason not to use Cloudflare’ s one-click solution.”
“The AI Audit tab will be accessible to existing customers through the Cloudflare dashboard,” writes ZDNet. “The tool integrates with Cloudflare’s global infrastructure, leveraging the company’s scale to provide these auditing features across the Internet.”
Site owners can visit the Cloudflare site to join a waitlist to participate in the beta for price setting capabilities.
No Comments Yet
You can be the first to comment!
Leave a comment
You must be logged in to post a comment.