Serving technology enthusiasts for more than 25 years.
TechSpot is the place to go for tech advice and analysis.
Context: Cloudflare’s CEO Matthew Prince has recently re-iterated his warning that generative AI summaries and crawlers threaten the foundations for the internet’s business models. The company is developing methods to combat AI scrapers in order to protect publishers from artificial AI traffic which offers no real site visits.
Prince, speaking at an Axios conference in Cannes last week explained that search engines using generative AI for summarizing web content and chatbots have reduced the number human visitors to many sites. The problem has gotten worse compared to just six months ago.
In the past, for every six Google crawls of a website, a person would visit it and possibly view ads. OpenAI’s crawlers had a rate of about 250 to 1, while Anthropic’s was 6,000 to 1. Cloudflare CEO estimates that Google crawl-to visitor rate has decreased to 18:1, OpenAI’s has deteriorated to 1,500:1, and Anthropic is approximately 60,000:1.
The decline in website visits is likely due to the use of chatbots and search engine. Chatbots can retrieve information and summarize it without the user having to leave the chat interface. AI overviews are also available from major search engine providers, which offer answers before users click on the search results.
Google claimed that when it first introduced AI overviews, the technology would boost traffic back to the original sources. In a similar way, large language models like ChatGPT are now citing sources to help direct traffic to the content creators.
The Zero Click Internet: A must read
However, Prince According to users don’t click on footnotes in the majority of cases. As trust in AI has grown over the last six months, many users are accepting AI’s answers at face value. The trend could be dangerous, as AI is known to produce inaccurate or misleading information.
Cloudflare has responded by launching a new tool that offers cybersecurity solutions to websites. AI Labyrinth is also known as. This tool uses generative AI to combat crawlers.
Websites can include instructions on how to block AI crawlers. However, many bots ignore or bypass these directives. When the AI Labyrinth detects this behavior, it leads a bot through a maze created by AI that no human could reasonably follow. This causes the bot to waste computing resources and time.
Prince stresses that Cloudflare’s track record is impressive, despite how intimidating it may seem to take on AI giants such as Google, Microsoft, or OpenAI.
