The most common finding on this site is a website that serves a normal browser fine and refuses GPTBot, ClaudeBot or PerplexityBot with a 403, a challenge page or a timeout. The owner almost never chose it. It comes from a hosting firewall, a security plugin or a "block AI bots" switch that was on by default.
Fetch your homepage as each crawler and compare with a normal visit. The scanner does this for you and names each crawler that was refused and how (403, challenge, timeout). From a terminal:
curl -A "GPTBot/1.2" -o /dev/null -s -w "%{http_code}\n" https://yourbusiness.com/
curl -A "ClaudeBot/1.0" -o /dev/null -s -w "%{http_code}\n" https://yourbusiness.com/
curl -A "PerplexityBot/1.0" -o /dev/null -s -w "%{http_code}\n" https://yourbusiness.com/
200 is fine. 403, 429 or 503 while a browser gets 200 means a rule aimed at bots.
| Where your site is built | Where the setting lives, October 2026 |
|---|---|
| Cloudflare in front of the site | Security, then Settings, filter by Bot traffic: the Block AI bots option has three modes; set it to not block, or use AI Crawl Control to allow the crawlers you want one by one. Since September 2026 Cloudflare blocks training and agent crawlers by default on pages that carry ads; search crawlers stay allowed. |
| WordPress | A security plugin (Wordfence, iThemes, All In One Security) or a "block AI bots" plugin. Look in its firewall or bot rules for GPTBot, ClaudeBot, PerplexityBot and allow them. Check the host's own firewall too. |
| Squarespace | Settings, then Crawlers: untick Block known artificial intelligence crawlers. Leave Block search engine crawlers unticked as well. |
| Wix | Wix does not block by default. Open SEO, then Tools and settings, Robots.txt Editor, and remove any Disallow lines for GPTBot, ChatGPT-User, ClaudeBot or PerplexityBot. |
| Shopify | Online Store, Themes, Edit code: the robots.txt.liquid template. Keep Shopify's default rules and add Allow rules for the AI user agents. If a security app manages robots.txt, check it too. |
| Webflow | Site settings, SEO, Indexing: the robots.txt field. Remove Disallow lines for the AI user agents. The same panel has a Content-Signal setting; keep search=yes and ai-input=yes if you want to appear in answers. |
| GoDaddy, Jimdo, Duda, Weebly, Site123 | No bot switch of their own. If the scanner shows a refusal, write to the host's support and ask them to allow GPTBot, ClaudeBot and PerplexityBot. |
| Your own server | Look in .htaccess, nginx config or the WAF for user-agent rules. Many "bad bot" lists copied from forums include the AI crawlers. |
Dashboards move things around. If a name here does not match, search the dashboard for the word in bold.
If you want to stay out of model training but still appear in answers, you can: the token for training (GPTBot, Google-Extended) is not the one that fetches a page for a person's question (ChatGPT-User, OAI-SearchBot, Claude-User, Perplexity-User). Allow the ones that answer questions. The robots.txt guide lists all eight.
Want the list of who gets in today? Run the free scanner on your address: the report names each crawler that was refused, what it found for the menu, the structured data and the booking path, and what to fix first. Rescans are free.
Updated 2026-10-14. Dashboard names are as of that date.