25 september 2025 Protect your website against AI bots that scrap your content
Major tech companies increasingly train AI models on web content. The texts, images and other content of websites are copied and used to train AI. Maybe you don't want that. Fortunately you can do something about it.
What are AI bots?
AI bots are automatic programs that search the internet for content. Think of bots of companies such as OpenAi, Google, Meta or Amazon. They collect information to make their AI systems smarter. That can be useful for them, but as a site manager you do not always benefit from your content being reused.
How do you block AI bots?
With a robots.txt file you can tell search engines and bots what they can and can not copy from your site. Many AI bots adhere to these rules. If you enable the AI bot protection via our dashboard, we automatically add the correct rules to your robots.txt.
This means that the following bots, among other things, will no longer have access to your site:
# Block AI bots
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
Why is this important?
- Control over your content: You determine whether your texts and media can be used by AI.
- Protection of your brand: you prevent your unique content from disappearing into AI models.
- Transparency: Visitors and customers appreciate that you consciously deal with their (and your own) data.
AI and content scraping are here to stay, but with a simple setting you can keep control. You decide whether AI bots will have access to your site.
-
Sean Blommaert