Publishers told it’s “never too late” to start blocking sites from being crawled by AI bots.
Almost eight in ten of the biggest news websites in the UK and US are now blocking AI training bots.
This week in London publishers were told it is “never too late” to begin blocking the bots due to the frequency with which new crawling needs to take place via retrieval-augmented generation so that AI chatbot responses remain current.
Some 79% of almost 100 top news websites in the UK and US are blocking at least one crawler used for AI training out of OpenAI’s GPTBot, ClaudeBot, Anthropic-ai, CCBot, Applebot-Extended and Google-Extended.
Meanwhile 71% are blocking AI bots from crawling their sites for retrieval or live searches. Those bots are: ChatGPT-User, Claude-Web, Perplexity-User and OAI-SearchBot.
