---
title: 31% of business websites turn at least one AI crawler away · The Direct Index
url: https://directindex.org/stats/ai-blocked
---

[The Direct Index](https://directindex.org/)
[Scanner](https://directindex.org/scan)[Cities](https://directindex.org/cities)[Numbers](https://directindex.org/stats)[Rankings](https://directindex.org/rankings)[Search](https://directindex.org/search)[Guides](https://directindex.org/guides)[About](https://directindex.org/about)

 [The numbers](https://directindex.org/stats) / Agents · live, 2026-10-17

# 31% of business websites turn at least one AI crawler away

161,416 reachable business websites in 675 indexes, as scanned by 17 October 2026.

Main points

- That is **31%** of 161,416 reachable business websites.
- By kind of business: highest among hairdressers and beauty salons (34%), lowest among tours, attractions and museums (28%).
- Among 101 restaurant indexes, Houston is highest at 45%; the median is 30%.

Figure 1

## By kind of business

| Kind | Share | Of | |
| --- | --- | --- | --- |
| [restaurants](https://directindex.org/stats/ai-blocked) | 29% | 74,726 |
**
 |
| [cafes and bars](https://directindex.org/stats/ai-blocked?kind=cafes) | 33% | 29,566 |
**
 |
| [hairdressers and beauty salons](https://directindex.org/stats/ai-blocked?kind=salons) | 34% | 17,764 |
**
 |
| [dentists, clinics and vets](https://directindex.org/stats/ai-blocked?kind=clinics) | 30% | 29,450 |
**
 |
| [tours, attractions and museums](https://directindex.org/stats/ai-blocked?kind=tours) | 28% | 9,910 |
**
 |

Figure 2

## Restaurants, city by city

[restaurants](https://directindex.org/stats/ai-blocked)[cafes and bars](https://directindex.org/stats/ai-blocked?kind=cafes)[hairdressers and beauty salons](https://directindex.org/stats/ai-blocked?kind=salons)[dentists, clinics and vets](https://directindex.org/stats/ai-blocked?kind=clinics)[tours, attractions and museums](https://directindex.org/stats/ai-blocked?kind=tours)

Highlight a cityNoneAarhusAmsterdamAntwerpAthensAtlantaAucklandAustinBarcelonaBaselBerlinBernBirminghamBordeauxBostonBremenBrisbaneBrusselsBudapestCalgaryCharlotteChicagoCologneCopenhagenDallasDenverDortmundDresdenDubaiDublinDüsseldorfEdinburghEindhovenEssenFlorenceFrankfurtGenevaGhentGlasgowGothenburgGrazHamburgHanoverHelsinkiHoustonIstanbulLausanneLeipzigLilleLisbonLondonLos AngelesLyonMadridMalmöManchesterMarseilleMelbourneMexico CityMiamiMilanMinneapolisMontrealMunichNantesNew YorkNiceNurembergOrlandoOsakaOsloOttawaParisPhiladelphiaPhoenixPortoPragueRomeRotterdamSan DiegoSan FranciscoSan JoseSeattleSingaporeStockholmStrasbourgStuttgartSydneyThe HagueTokyoTorontoToulouseTurinUtrechtValenciaVancouverVeniceVeronaViennaWarsawWashingtonZurichShow

**another restaurant index**median, 30%

0%25%50%

101 restaurant indexes. The median is 30%; the highest is Houston at 45%.

| # | Index | Share | Of |
| --- | --- | --- | --- |
| 1 | [Houston](https://directindex.org/city/houston) US | 45% | 510 |
| 2 | [Chicago](https://directindex.org/city/chicago) US | 43% | 1,205 |
| 3 | [San Jose](https://directindex.org/city/san-jose) US | 42% | 514 |
| 4 | [Zurich](https://directindex.org/city/zurich) CH | 42% | 710 |
| 5 | [Orlando](https://directindex.org/city/orlando) US | 41% | 239 |
| 6 | [Atlanta](https://directindex.org/city/atlanta) US | 39% | 417 |
| 7 | [Bern](https://directindex.org/city/bern) CH | 39% | 279 |
| 7 | [Los Angeles](https://directindex.org/city/los-angeles) US | 39% | 1,167 |
| 9 | [Charlotte](https://directindex.org/city/charlotte) US | 39% | 301 |
| 10 | [Venice](https://directindex.org/city/venice) IT | 38% | 190 |

 All 101 restaurant indexes

| # | Index | Share | Of |
| --- | --- | --- | --- |
| 1 | [Houston](https://directindex.org/city/houston) US | 45% | 510 |
| 2 | [Chicago](https://directindex.org/city/chicago) US | 43% | 1,205 |
| 3 | [San Jose](https://directindex.org/city/san-jose) US | 42% | 514 |
| 4 | [Zurich](https://directindex.org/city/zurich) CH | 42% | 710 |
| 5 | [Orlando](https://directindex.org/city/orlando) US | 41% | 239 |
| 6 | [Atlanta](https://directindex.org/city/atlanta) US | 39% | 417 |
| 7 | [Bern](https://directindex.org/city/bern) CH | 39% | 279 |
| 7 | [Los Angeles](https://directindex.org/city/los-angeles) US | 39% | 1,167 |
| 9 | [Charlotte](https://directindex.org/city/charlotte) US | 39% | 301 |
| 10 | [Venice](https://directindex.org/city/venice) IT | 38% | 190 |
| 11 | [Tokyo](https://directindex.org/city/tokyo) JP | 38% | 412 |
| 12 | [Denver](https://directindex.org/city/denver) US | 38% | 653 |
| 13 | [Osaka](https://directindex.org/city/osaka) JP | 38% | 212 |
| 14 | [Phoenix](https://directindex.org/city/phoenix) US | 38% | 437 |
| 15 | [San Francisco](https://directindex.org/city/san-francisco) US | 37% | 901 |
| 16 | [Dallas](https://directindex.org/city/dallas) US | 37% | 640 |
| 17 | [Philadelphia](https://directindex.org/city/philadelphia) US | 37% | 567 |
| 18 | [Ottawa](https://directindex.org/city/ottawa) CA | 37% | 487 |
| 19 | [Austin](https://directindex.org/city/austin) US | 36% | 508 |
| 20 | [Warsaw](https://directindex.org/city/warsaw) PL | 35% | 567 |
| 21 | [Montreal](https://directindex.org/city/montreal) CA | 35% | 700 |
| 22 | [Seattle](https://directindex.org/city/seattle) US | 35% | 722 |
| 23 | [San Diego](https://directindex.org/city/san-diego) US | 35% | 720 |
| 24 | [Singapore](https://directindex.org/city/singapore) SG | 35% | 455 |
| 25 | [Basel](https://directindex.org/city/basel) CH | 34% | 220 |
| 25 | [Rotterdam](https://directindex.org/city/rotterdam) NL | 34% | 408 |
| 25 | [Washington](https://directindex.org/city/washington) US | 34% | 1,143 |
| 28 | [Glasgow](https://directindex.org/city/glasgow) GB | 34% | 219 |
| 29 | [Mexico City](https://directindex.org/city/mexico-city) MX | 34% | 190 |
| 29 | [Porto](https://directindex.org/city/porto) PT | 34% | 439 |
| 31 | [Lisbon](https://directindex.org/city/lisbon) PT | 34% | 986 |
| 32 | [Essen](https://directindex.org/city/essen) DE | 34% | 322 |
| 33 | [Hamburg](https://directindex.org/city/hamburg) DE | 33% | 1,873 |
| 34 | [Leipzig](https://directindex.org/city/leipzig) DE | 33% | 377 |
| 35 | [Birmingham](https://directindex.org/city/birmingham) GB | 33% | 275 |
| 36 | [Berlin](https://directindex.org/city/berlin) DE | 33% | 4,294 |
| 36 | [Rome](https://directindex.org/city/rome) IT | 33% | 689 |
| 38 | [The Hague](https://directindex.org/city/the-hague) NL | 33% | 493 |
| 39 | [Munich](https://directindex.org/city/munich) DE | 32% | 1,604 |
| 40 | [Boston](https://directindex.org/city/boston) US | 32% | 683 |
| 40 | [Eindhoven](https://directindex.org/city/eindhoven) NL | 32% | 267 |
| 40 | [Toronto](https://directindex.org/city/toronto) CA | 32% | 957 |
| 43 | [Istanbul](https://directindex.org/city/istanbul) TR | 32% | 259 |
| 44 | [Frankfurt](https://directindex.org/city/frankfurt) DE | 31% | 888 |
| 45 | [Dortmund](https://directindex.org/city/dortmund) DE | 30% | 296 |
| 45 | [Dublin](https://directindex.org/city/dublin) IE | 30% | 375 |
| 45 | [Nuremberg](https://directindex.org/city/nuremberg) DE | 30% | 576 |
| 45 | [Verona](https://directindex.org/city/verona) IT | 30% | 553 |
| 49 | [Amsterdam](https://directindex.org/city/amsterdam) NL | 30% | 1,192 |
| 49 | [London](https://directindex.org/city/london) GB | 30% | 2,637 |
| 51 | [Brisbane](https://directindex.org/city/brisbane) AU | 30% | 247 |
| 51 | [New York](https://directindex.org/city/new-york) US | 30% | 3,524 |
| 53 | [Dubai](https://directindex.org/city/dubai) AE | 30% | 184 |
| 53 | [Stuttgart](https://directindex.org/city/stuttgart) DE | 30% | 743 |
| 53 | [Vancouver](https://directindex.org/city/vancouver) CA | 30% | 849 |
| 56 | [Dresden](https://directindex.org/city/dresden) DE | 30% | 379 |
| 57 | [Geneva](https://directindex.org/city/geneva) CH | 30% | 288 |
| 58 | [Düsseldorf](https://directindex.org/city/dusseldorf) DE | 29% | 558 |
| 59 | [Budapest](https://directindex.org/city/budapest) HU | 29% | 782 |
| 59 | [Madrid](https://directindex.org/city/madrid) ES | 29% | 971 |
| 61 | [Bremen](https://directindex.org/city/bremen) DE | 28% | 422 |
| 61 | [Cologne](https://directindex.org/city/cologne) DE | 28% | 908 |
| 63 | [Manchester](https://directindex.org/city/manchester) GB | 28% | 205 |
| 64 | [Prague](https://directindex.org/city/prague) CZ | 28% | 1,108 |
| 65 | [Calgary](https://directindex.org/city/calgary) CA | 28% | 558 |
| 66 | [Sydney](https://directindex.org/city/sydney) AU | 27% | 709 |
| 67 | [Hanover](https://directindex.org/city/hanover) DE | 27% | 445 |
| 68 | [Copenhagen](https://directindex.org/city/copenhagen) DK | 27% | 999 |
| 68 | [Turin](https://directindex.org/city/turin) IT | 27% | 267 |
| 70 | [Minneapolis](https://directindex.org/city/minneapolis) US | 27% | 389 |
| 71 | [Milan](https://directindex.org/city/milan) IT | 26% | 918 |
| 71 | [Vienna](https://directindex.org/city/vienna) AT | 26% | 2,329 |
| 73 | [Barcelona](https://directindex.org/city/barcelona) ES | 26% | 636 |
| 74 | [Auckland](https://directindex.org/city/auckland) NZ | 25% | 325 |
| 75 | [Florence](https://directindex.org/city/florence) IT | 25% | 187 |
| 75 | [Miami](https://directindex.org/city/miami) US | 25% | 2,802 |
| 77 | [Melbourne](https://directindex.org/city/melbourne) AU | 25% | 927 |
| 78 | [Utrecht](https://directindex.org/city/utrecht) NL | 24% | 536 |
| 79 | [Graz](https://directindex.org/city/graz) AT | 23% | 265 |
| 80 | [Edinburgh](https://directindex.org/city/edinburgh) GB | 23% | 403 |
| 81 | [Oslo](https://directindex.org/city/oslo) NO | 23% | 407 |
| 82 | [Marseille](https://directindex.org/city/marseille) FR | 22% | 189 |
| 83 | [Strasbourg](https://directindex.org/city/strasbourg) FR | 22% | 353 |
| 84 | [Athens](https://directindex.org/city/athens) GR | 21% | 305 |
| 85 | [Nantes](https://directindex.org/city/nantes) FR | 21% | 353 |
| 86 | [Lille](https://directindex.org/city/lille) FR | 21% | 248 |
| 87 | [Bordeaux](https://directindex.org/city/bordeaux) FR | 19% | 336 |
| 88 | [Helsinki](https://directindex.org/city/helsinki) FI | 19% | 751 |
| 88 | [Lausanne](https://directindex.org/city/lausanne) CH | 19% | 281 |
| 90 | [Toulouse](https://directindex.org/city/toulouse) FR | 18% | 413 |
| 91 | [Lyon](https://directindex.org/city/lyon) FR | 18% | 776 |
| 92 | [Valencia](https://directindex.org/city/valencia) ES | 18% | 248 |
| 93 | [Ghent](https://directindex.org/city/ghent) BE | 18% | 411 |
| 94 | [Paris](https://directindex.org/city/paris) FR | 18% | 2,646 |
| 95 | [Aarhus](https://directindex.org/city/aarhus) DK | 18% | 256 |
| 96 | [Gothenburg](https://directindex.org/city/gothenburg) SE | 17% | 406 |
| 97 | [Antwerp](https://directindex.org/city/antwerp) BE | 17% | 392 |
| 98 | [Brussels](https://directindex.org/city/brussels) BE | 17% | 736 |
| 99 | [Nice](https://directindex.org/city/nice) FR | 15% | 346 |
| 100 | [Stockholm](https://directindex.org/city/stockholm) SE | 15% | 1,289 |
| 101 | [Malmö](https://directindex.org/city/malmo) SE | 13% | 214 |

Indexes under 200 scanned websites are left out. The "Of" column is the base each figure rests on.

[Data, CSV](https://directindex.org/stats/ai-blocked.csv)[JSON](https://directindex.org/stats/ai-blocked.json)[Method](https://directindex.org/methodology)[Link to this figure](https://directindex.org/stats/ai-blocked#figure-2)

**About these numbers**
A site turns a crawler away when its robots.txt disallows a known AI crawler or its firewall answers that crawler with an error. Many owners don't know their host or template does it.

Cite this

31% of business websites turn at least one AI crawler away (The Direct Index, 17 October 2026, directindex.org/stats/ai-blocked).

[CSV](https://directindex.org/stats/ai-blocked.csv)[JSON](https://directindex.org/stats/ai-blocked.json)[Markdown](https://directindex.org/stats/ai-blocked.md)[Method](https://directindex.org/methodology)[Every number](https://directindex.org/stats)

Data: ODbL. Business data from OpenStreetMap contributors and Overture Maps; website facts from the Direct Index scanner, which reads each site's public pages.

#### The Direct Index

[Why direct](https://directindex.org/why)[What agent-direct means](https://directindex.org/agent-direct)[About and principles](https://directindex.org/about)[Methodology](https://directindex.org/methodology)[Privacy](https://directindex.org/privacy)[Terms](https://directindex.org/terms)

#### Use it

[Free scanner](https://directindex.org/scan)[City indexes](https://directindex.org/cities)[Search the index](https://directindex.org/search)[Rankings](https://directindex.org/rankings)[What assistants recommend](https://directindex.org/assistants)[Ask the index](https://directindex.org/ask)

#### Vendors and help

[Booking and ordering vendors](https://directindex.org/vendors)[Do their terms allow agents?](https://directindex.org/vendors/terms)[Vendor verification](https://directindex.org/vendors/verification)[Website builders](https://directindex.org/builders)[Who can fix this](https://directindex.org/providers)[For agencies](https://directindex.org/agency)

#### For agents

[llms.txt](https://directindex.org/llms.txt)[llms-full.txt](https://directindex.org/llms-full.txt)[OpenAPI](https://directindex.org/openapi.json)[MCP server](https://directindex.org/.well-known/mcp/server-card.json)[A2A agent](https://directindex.org/.well-known/agent-card.json)[Direct feeds](https://directindex.org/direct.json)

#### Data

[Open data, CSV and JSON](https://directindex.org/data)[The quarterly edition](https://directindex.org/editions)[City data, JSON and Markdown](https://directindex.org/cities)[Sitemap](https://directindex.org/sitemap.xml)

Business listings from OpenStreetMap contributors, ODbL. Scores under the same licence.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://directindex.org/#org","name":"The Direct Index","url":"https://directindex.org/","logo":"https://directindex.org/favicon.svg","founder":{"@type":"Person","name":"Antonio Curado"},"description":"Independent measurement of whether AI agents can find, read and act on a business's own website."},{"@type":"WebSite","@id":"https://directindex.org/#site","url":"https://directindex.org/","name":"The Direct Index","publisher":{"@id":"https://directindex.org/#org"},"inLanguage":"en","potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://directindex.org/scan?url={url}"},"query-input":"required name=url"}},{"@type":"WebAPI","name":"The Direct Index API","url":"https://directindex.org/openapi.json","documentation":"https://directindex.org/llms-full.txt","provider":{"@id":"https://directindex.org/#org"}}]}
{"@context":"https://schema.org","@type":"Dataset","name":"31% of business websites turn at least one AI crawler away","url":"https://directindex.org/stats/ai-blocked","description":"31% of business websites turn at least one AI crawler away. 161,416 reachable business websites in 675 indexes, as scanned by 17 October 2026. A site turns a crawler away when its robots.txt disallows a known AI crawler or its firewall answers that crawler with an error. Many owners don't know their host or template does it.","license":"https://opendatacommons.org/licenses/odbl/","creator":{"@id":"https://directindex.org/#org"},"dateModified":"2026-10-17T20:20:59.298Z","distribution":[{"@type":"DataDownload","encodingFormat":"text/csv","contentUrl":"https://directindex.org/stats/ai-blocked.csv"},{"@type":"DataDownload","encodingFormat":"application/json","contentUrl":"https://directindex.org/stats/ai-blocked.json"}]}
```