Control search and AI crawlers
Understand the distinct roles of indexing, search and training crawlers.
Different crawler purposes
A discovery crawler may help a search product present links, while another may collect material for training. OpenAI documents independent OAI-SearchBot and GPTBot controls. Publishers can make different choices for different purposes.
Understand robots.txt
robots.txt communicates crawl rules to cooperative bots. It is neither authentication nor protection against malicious scraping; sensitive content needs genuine access controls.
Googlebot and Google-Extended
Google-Extended is a control token for certain AI uses, not a separate HTTP crawler. According to Google, it does not affect ordinary Google Search inclusion. Search controls are not identical to preferences for other products.
Document policy decisions
Keep dates, policy versions, selection criteria and affected crawlers. Review logs, CDN rules and published resources; a configuration alone does not guarantee third-party practices.
Three layers of control
An editorial policy distinguishes public pages, access-controlled resources and content for which selected crawling purposes are not desired. Robots.txt communicates preferences to cooperative crawlers but does not enforce a technical barrier. Private files require actual authentication, and contractual or legal controls cannot be reduced to one directive.
Example: search crawling without training crawling
A publisher can distinguish search access from certain training-related collection purposes using documented robots tokens. Record exact agent names, their declared purpose, directives and exceptions. Periodically check logs and discoverability effects; preferences do not stop malicious crawlers.
Core steps
- 01Identify each crawler purpose
- 02Write specific crawl directives
- 03Examine crawls and log entries
- 04Review policy regularly
Three questions to ask
Does robots.txt protect confidential files?
Are search and training bots governed separately?
Is the policy dated and reversible?
Read the source documents
Versions, reuse terms and document status can change. Check them on the original sites.