---
title: robots.txt for AI agents: who to allow, and how · The Direct Index
url: https://directindex.org/guides/robots-txt-for-ai-agents
---

[The Direct Index](https://directindex.org/)
[Scanner](https://directindex.org/scan)[Cities](https://directindex.org/cities)[Guides](https://directindex.org/guides)[Why](https://directindex.org/why)[Method](https://directindex.org/methodology)[About](https://directindex.org/about)

Guide

# robots.txt for AI agents: who to allow, and how

robots.txt is a text file at `yourbusiness.com/robots.txt` that tells crawlers what they may read. For a business that wants to be found by assistants, the whole file can be eight lines. Here is who the user agents are and what to write.

## The eight that matter

| User agent | Who | What it does |
| --- | --- | --- |
| GPTBot | OpenAI | Reads pages to train models. |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results. |
| ChatGPT-User | OpenAI | Fetches a page when a person asks ChatGPT something about it. |
| ClaudeBot | Anthropic | Reads pages to train and to index. |
| Claude-User | Anthropic | Fetches a page when a person asks Claude. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers. |
| Perplexity-User | Perplexity | Fetches a page for a person's question. |
| Google-Extended | Google | Controls use in Gemini training; Google's AI features use the normal Googlebot. |

## A file that lets them all in

```
User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml
```

That is the default when there is no robots.txt at all, so most sites need no change. The problem is usually a rule someone added. Look for these and delete them:

```
User-agent: GPTBot
Disallow: /
```

## Out of training, in the answers

If you do not want your pages used for training but do want to be found when someone asks, disallow the training tokens only:

```
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
```

ClaudeBot does both jobs, so blocking it also removes you from Claude's answers.

## Content Signals

A newer, optional line states your preferences in one place, read by Cloudflare and a growing number of crawlers:

```
Content-Signal: search=yes, ai-input=yes, ai-train=no
```

`ai-input` is the one that matters for appearing in answers. The scanner reports the signal when it finds one; it is not scored.

## Where to edit it

| Where your site is built | Where the setting lives, October 2026 |
| --- | --- |
| WordPress | Yoast or Rank Math have a robots.txt editor under Tools; otherwise the file sits in the site root. |
| Wix | SEO, Tools and settings, **Robots.txt Editor**. |
| Squarespace | Not editable; use Settings, **Crawlers** instead. |
| Shopify | Online Store, Themes, Edit code, template **robots.txt.liquid**. |
| Webflow | Site settings, SEO, **Indexing**. |

Dashboards move things around. If a name here does not match, search the dashboard for the word in bold.

## Check it

Not sure what your file says today? Run the [free scanner](https://directindex.org/scan) on your address: the report names each crawler that was refused, what it found for the menu, the structured data and the booking path, and what to fix first. Rescans are free.

## More guides

- [ChatGPT doesn't show my restaurant. Why, and what to do](https://directindex.org/guides/chatgpt-does-not-show-my-restaurant)
- [Am I blocking ChatGPT, Claude and Perplexity without knowing?](https://directindex.org/guides/blocking-ai-crawlers-without-knowing)
- [My menu is a PDF. Can AI assistants read it?](https://directindex.org/guides/menu-pdf-ai)
- [Structured data for a restaurant website, copy and paste](https://directindex.org/guides/structured-data-restaurant)
- [All guides](https://directindex.org/guides)

Updated 2026-10-14. Dashboard names are as of that date.

#### The Direct Index

[Why direct](https://directindex.org/why)[About and principles](https://directindex.org/about)[Methodology](https://directindex.org/methodology)[Privacy](https://directindex.org/privacy)

#### Use it

[Free scanner](https://directindex.org/scan)[City indexes](https://directindex.org/cities)[Guides](https://directindex.org/guides)[Common fixes](https://directindex.org/fix)[What agent-direct means](https://directindex.org/agent-direct)

#### For agents

[llms.txt](https://directindex.org/llms.txt)[llms-full.txt](https://directindex.org/llms-full.txt)[OpenAPI](https://directindex.org/openapi.json)[MCP server](https://directindex.org/.well-known/mcp/server-card.json)[A2A agent](https://directindex.org/.well-known/agent-card.json)

#### Data

[City data, JSON and Markdown](https://directindex.org/cities)[Sitemap](https://directindex.org/sitemap.xml)

Business listings from OpenStreetMap contributors, ODbL. Scores under the same licence.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://directindex.org/#org","name":"The Direct Index","url":"https://directindex.org/","logo":"https://directindex.org/favicon.svg","founder":{"@type":"Person","name":"Antonio Curado"},"description":"Independent measurement of whether AI agents can find, read and act on a business's own website."},{"@type":"WebSite","@id":"https://directindex.org/#site","url":"https://directindex.org/","name":"The Direct Index","publisher":{"@id":"https://directindex.org/#org"},"inLanguage":"en","potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://directindex.org/scan?url={url}"},"query-input":"required name=url"}},{"@type":"WebAPI","name":"The Direct Index API","url":"https://directindex.org/openapi.json","documentation":"https://directindex.org/llms-full.txt","provider":{"@id":"https://directindex.org/#org"}}]}
{"@context":"https://schema.org","@graph":[{"@type":"Article","@id":"https://directindex.org/guides/robots-txt-for-ai-agents#article","headline":"robots.txt for AI agents: who to allow, and how","description":"The eight AI user agents that matter for a business website, what each one does, a robots.txt you can paste, and Content Signals.","datePublished":"2026-10-14","dateModified":"2026-10-14","inLanguage":"en","author":{"@id":"https://directindex.org/#org"},"publisher":{"@id":"https://directindex.org/#org"},"mainEntityOfPage":"https://directindex.org/guides/robots-txt-for-ai-agents","isPartOf":{"@id":"https://directindex.org/#site"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Guides","item":"https://directindex.org/guides"},{"@type":"ListItem","position":2,"name":"robots.txt for AI agents: who to allow, and how","item":"https://directindex.org/guides/robots-txt-for-ai-agents"}]}]}
```