A new system will enable millions of websites, including prominent platforms like Sky News, The Associated Press, and BuzzFeed, to block artificial intelligence (AI) bots from accessing their content without explicit authorization.
This initiative is being spearheaded by Cloudflare, an internet infrastructure company that hosts approximately one-fifth of the internet.
The long-term vision includes allowing websites to demand financial compensation from AI companies in exchange for granting permission to scrape their content.
Many leading figures in creative fields, such as writers, artists, musicians, and actors, have publicly criticized AI firms for allegedly training their systems using copyrighted material without obtaining proper consent or providing due compensation.
In the UK, this issue has sparked considerable debate between the government and artists, including Sir Elton John, regarding the need to strengthen copyright protections.
Cloudflare’s technology specifically targets AI firm bots, also known as web crawlers, which are programs designed to navigate the internet, indexing and collecting data as they go. These crawlers play a crucial role in the development, training, and operation of AI systems.
Cloudflare reports that its technology is currently active on one million websites.
Roger Lynch, CEO of Condé Nast, whose publications include GQ, Vogue, and The New Yorker, hailed the move as “a game-changer” for publishers.
In a statement, he emphasized that “This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.”
However, other experts suggest that more robust legal safeguards will still be necessary to fully address the issue.
Initially, the system will be applied by default to new Cloudflare users, as well as to websites that participated in a previous initiative aimed at blocking crawlers.
Numerous publishers have accused AI companies of unauthorized use of their content.
Recently, the BBC threatened legal action against Perplexity, a US-based AI firm, demanding an immediate cessation of the use of BBC content and seeking compensation for previously used material.
In general, publishers are amenable to allowing crawlers from search engines like Google to access their sites, as these search engines drive traffic to their content.
Perplexity has countered by accusing the BBC of attempting to preserve “Google’s monopoly.”
Cloudflare contends that AI activities disrupt the implicit agreement between publishers and crawlers. They argue that AI crawlers collect content such as text, articles, and images to generate answers, without directing users to the original source, thereby depriving content creators of revenue.
Matthew Prince, CEO of Cloudflare, stated, “If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone.”
To this end, Cloudflare is developing a “Pay Per Crawl” system that would enable content creators to request payment from AI companies for utilizing their original content.
Cloudflare reports a surge in AI bot activity.
The company stated in March that “AI Crawlers generate more than 50 billion requests to the Cloudflare network every day.”
There is increasing concern that certain AI crawlers are disregarding established protocols for excluding bots.
In an effort to address the most egregious offenders, Cloudflare previously developed a system that would direct them to a “Labyrinth” of web pages filled with AI-generated content.
The new system seeks to leverage technology to safeguard website content and enable AI firms to be charged for access.
In the UK, intense legislative debate is underway between the government, creators, and AI companies regarding the extent to which the creative industries should be protected from AI firms using their works to train systems without permission or compensation.
On both sides of the Atlantic, content creators, licensors, and owners have initiated legal action to prevent what they perceive as AI firms encroaching upon creative rights.
Ed Newton-Rex, founder of Fairly Trained, which certifies that AI companies have trained their systems on properly licensed data, welcomed the development but noted that “there is only so much” one company can do.
He told the BBC that “This is really only a sticking plaster when what’s required is major surgery.”
He added, “It will only offer protection for people on websites they control – it’s like having body armour that stops working when you leave your house.”
“The only real way to protect people’s content from theft by AI companies is through the law.”