robots.txt is a text file at yourbusiness.com/robots.txt that tells crawlers what they may read. For a business that wants to be found by assistants, the whole file can be eight lines. Here is who the user agents are and what to write.
| User agent | Who | What it does |
|---|---|---|
| GPTBot | OpenAI | Reads pages to train models. |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results. |
| ChatGPT-User | OpenAI | Fetches a page when a person asks ChatGPT something about it. |
| ClaudeBot | Anthropic | Reads pages to train and to index. |
| Claude-User | Anthropic | Fetches a page when a person asks Claude. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers. |
| Perplexity-User | Perplexity | Fetches a page for a person's question. |
| Google-Extended | Controls use in Gemini training; Google's AI features use the normal Googlebot. |
User-agent: * Allow: / Sitemap: https://yourbusiness.com/sitemap.xml
That is the default when there is no robots.txt at all, so most sites need no change. The problem is usually a rule someone added. Look for these and delete them:
User-agent: GPTBot Disallow: /
If you do not want your pages used for training but do want to be found when someone asks, disallow the training tokens only:
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Allow: /
ClaudeBot does both jobs, so blocking it also removes you from Claude's answers.
A newer, optional line states your preferences in one place, read by Cloudflare and a growing number of crawlers:
Content-Signal: search=yes, ai-input=yes, ai-train=no
ai-input is the one that matters for appearing in answers. The scanner reports the signal when it finds one; it is not scored.
| Where your site is built | Where the setting lives, October 2026 |
|---|---|
| WordPress | Yoast or Rank Math have a robots.txt editor under Tools; otherwise the file sits in the site root. |
| Wix | SEO, Tools and settings, Robots.txt Editor. |
| Squarespace | Not editable; use Settings, Crawlers instead. |
| Shopify | Online Store, Themes, Edit code, template robots.txt.liquid. |
| Webflow | Site settings, SEO, Indexing. |
Dashboards move things around. If a name here does not match, search the dashboard for the word in bold.
Not sure what your file says today? Run the free scanner on your address: the report names each crawler that was refused, what it found for the menu, the structured data and the booking path, and what to fix first. Rescans are free.
Updated 2026-10-14. Dashboard names are as of that date.