Should my website block AI crawlers?

By reddysite ·

For most small businesses, no: an AI assistant can only describe a business from pages it is allowed to read. Blocking makes sense when the content itself is what you sell, such as paid articles or original photography, and reddysite.com allows AI crawlers on its own site.

What are AI crawlers?

AI crawlers are bots that fetch web pages for AI companies. They come in two kinds:

Kind Fetches pages to Names in robots.txt
Training crawlers Train AI models GPTBot, ClaudeBot, Google-Extended, CCBot
Answer bots Answer a question someone asked OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot

Each AI company publishes its bots’ names, and the list changes, so check their documentation before you write rules.

When does blocking make sense?

Blocking AI crawlers makes sense when the content itself is the product:

  • Paid articles, courses or research
  • Original photography or artwork you license to others
  • Pages you would rather no model trained on

For a business that wants to be recommended, the pages describing its services are the ones an assistant needs to read.

How do I block or allow them?

A robots.txt file at the root of your domain names each bot and what it may visit:

  • A User-agent: GPTBot line followed by Disallow: / blocks OpenAI’s training crawler.
  • A User-agent: * line followed by Allow: / lets every crawler in.

Blocking a training crawler doesn’t block the same company’s answer bot unless you list that one too. The robots.txt file is a request, not a lock. Well-behaved crawlers follow it, and others may not.

Could my site be blocking them already?

Some hosting and security services add AI-crawler blocks to robots.txt for you. Open yourdomain.com/robots.txt in a browser to see what your site actually says. reddysite.com allows them.

What does reddysite.com do?

reddysite.com allows every crawler, training crawlers included, because a business that sells being found should be readable. reddysite.com’s robots.txt lets everything in apart from one internal folder. How reddysite builds for search and AI lists what every site carries, and what llms.txt is covers the summary file reddysite.com also publishes.

Frequently asked questions

Does blocking AI training crawlers remove my site from AI search?

Not necessarily. Some companies run separate bots for training and for answering questions, and robots.txt treats each name separately. OpenAI's documentation, for example, lists GPTBot and OAI-SearchBot as separate bots.

Will blocking Google-Extended affect Google Search?

No, according to Google's crawler documentation. Google-Extended controls whether content is used for Google's AI models, and crawling for Search is separate.

Can AI crawlers slow my site down?

Heavy crawling can add load to a small server. Static pages served from a global network rarely notice it.

References