--- title: Am I blocking ChatGPT, Claude and Perplexity without knowing? ยท The Direct Index url: https://directindex.org/guides/blocking-ai-crawlers-without-knowing --- [The Direct Index](https://directindex.org/) [Scanner](https://directindex.org/scan)[Cities](https://directindex.org/cities)[Guides](https://directindex.org/guides)[Why](https://directindex.org/why)[Method](https://directindex.org/methodology)[About](https://directindex.org/about) Guide # Am I blocking ChatGPT, Claude and Perplexity without knowing? The most common finding on this site is a website that serves a normal browser fine and refuses GPTBot, ClaudeBot or PerplexityBot with a 403, a challenge page or a timeout. The owner almost never chose it. It comes from a hosting firewall, a security plugin or a "block AI bots" switch that was on by default. ## How to tell Fetch your homepage as each crawler and compare with a normal visit. The [scanner](https://directindex.org/scan) does this for you and names each crawler that was refused and how (403, challenge, timeout). From a terminal: ``` curl -A "GPTBot/1.2" -o /dev/null -s -w "%{http_code}\n" https://yourbusiness.com/ curl -A "ClaudeBot/1.0" -o /dev/null -s -w "%{http_code}\n" https://yourbusiness.com/ curl -A "PerplexityBot/1.0" -o /dev/null -s -w "%{http_code}\n" https://yourbusiness.com/ ``` 200 is fine. 403, 429 or 503 while a browser gets 200 means a rule aimed at bots. ## Where the switch is | Where your site is built | Where the setting lives, October 2026 | | --- | --- | | Cloudflare in front of the site | Security, then Settings, filter by **Bot traffic**: the **Block AI bots** option has three modes; set it to not block, or use **AI Crawl Control** to allow the crawlers you want one by one. Since September 2026 Cloudflare blocks training and agent crawlers by default on pages that carry ads; search crawlers stay allowed. | | WordPress | A security plugin (Wordfence, iThemes, All In One Security) or a "block AI bots" plugin. Look in its firewall or bot rules for GPTBot, ClaudeBot, PerplexityBot and allow them. Check the host's own firewall too. | | Squarespace | Settings, then **Crawlers**: untick **Block known artificial intelligence crawlers**. Leave **Block search engine crawlers** unticked as well. | | Wix | Wix does not block by default. Open SEO, then Tools and settings, **Robots.txt Editor**, and remove any Disallow lines for GPTBot, ChatGPT-User, ClaudeBot or PerplexityBot. | | Shopify | Online Store, Themes, Edit code: the **robots.txt.liquid** template. Keep Shopify's default rules and add Allow rules for the AI user agents. If a security app manages robots.txt, check it too. | | Webflow | Site settings, SEO, **Indexing**: the robots.txt field. Remove Disallow lines for the AI user agents. The same panel has a Content-Signal setting; keep `search=yes` and `ai-input=yes` if you want to appear in answers. | | GoDaddy, Jimdo, Duda, Weebly, Site123 | No bot switch of their own. If the scanner shows a refusal, write to the host's support and ask them to allow GPTBot, ClaudeBot and PerplexityBot. | | Your own server | Look in .htaccess, nginx config or the WAF for user-agent rules. Many "bad bot" lists copied from forums include the AI crawlers. | Dashboards move things around. If a name here does not match, search the dashboard for the word in bold. ## Training, search and agents are different crawlers If you want to stay out of model training but still appear in answers, you can: the token for training (GPTBot, Google-Extended) is not the one that fetches a page for a person's question (ChatGPT-User, OAI-SearchBot, Claude-User, Perplexity-User). Allow the ones that answer questions. [The robots.txt guide](https://directindex.org/guides/robots-txt-for-ai-agents) lists all eight. ## Check it Want the list of who gets in today? Run the [free scanner](https://directindex.org/scan) on your address: the report names each crawler that was refused, what it found for the menu, the structured data and the booking path, and what to fix first. Rescans are free. ## More guides - [ChatGPT doesn't show my restaurant. Why, and what to do](https://directindex.org/guides/chatgpt-does-not-show-my-restaurant) - [robots.txt for AI agents: who to allow, and how](https://directindex.org/guides/robots-txt-for-ai-agents) - [My menu is a PDF. Can AI assistants read it?](https://directindex.org/guides/menu-pdf-ai) - [Structured data for a restaurant website, copy and paste](https://directindex.org/guides/structured-data-restaurant) - [All guides](https://directindex.org/guides) Updated 2026-10-14. Dashboard names are as of that date. #### The Direct Index [Why direct](https://directindex.org/why)[About and principles](https://directindex.org/about)[Methodology](https://directindex.org/methodology)[Privacy](https://directindex.org/privacy) #### Use it [Free scanner](https://directindex.org/scan)[City indexes](https://directindex.org/cities)[Guides](https://directindex.org/guides)[Common fixes](https://directindex.org/fix)[What agent-direct means](https://directindex.org/agent-direct) #### For agents [llms.txt](https://directindex.org/llms.txt)[llms-full.txt](https://directindex.org/llms-full.txt)[OpenAPI](https://directindex.org/openapi.json)[MCP server](https://directindex.org/.well-known/mcp/server-card.json)[A2A agent](https://directindex.org/.well-known/agent-card.json) #### Data [City data, JSON and Markdown](https://directindex.org/cities)[Sitemap](https://directindex.org/sitemap.xml) Business listings from OpenStreetMap contributors, ODbL. Scores under the same licence. ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://directindex.org/#org","name":"The Direct Index","url":"https://directindex.org/","logo":"https://directindex.org/favicon.svg","founder":{"@type":"Person","name":"Antonio Curado"},"description":"Independent measurement of whether AI agents can find, read and act on a business's own website."},{"@type":"WebSite","@id":"https://directindex.org/#site","url":"https://directindex.org/","name":"The Direct Index","publisher":{"@id":"https://directindex.org/#org"},"inLanguage":"en","potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://directindex.org/scan?url={url}"},"query-input":"required name=url"}},{"@type":"WebAPI","name":"The Direct Index API","url":"https://directindex.org/openapi.json","documentation":"https://directindex.org/llms-full.txt","provider":{"@id":"https://directindex.org/#org"}}]} {"@context":"https://schema.org","@graph":[{"@type":"Article","@id":"https://directindex.org/guides/blocking-ai-crawlers-without-knowing#article","headline":"Am I blocking ChatGPT, Claude and Perplexity without knowing?","description":"Hosting firewalls and bot lists refuse AI crawlers on about a quarter of restaurant websites, with no robots.txt rule saying so. How to check in a minute and where the switch is on each builder.","datePublished":"2026-10-14","dateModified":"2026-10-14","inLanguage":"en","author":{"@id":"https://directindex.org/#org"},"publisher":{"@id":"https://directindex.org/#org"},"mainEntityOfPage":"https://directindex.org/guides/blocking-ai-crawlers-without-knowing","isPartOf":{"@id":"https://directindex.org/#site"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Guides","item":"https://directindex.org/guides"},{"@type":"ListItem","position":2,"name":"Am I blocking ChatGPT, Claude and Perplexity without knowing?","item":"https://directindex.org/guides/blocking-ai-crawlers-without-knowing"}]}]} ```