31% of business websites turn at least one AI crawler away

161,416 reachable business websites in 675 indexes, as scanned by 17 October 2026.

Main points

That is 31% of 161,416 reachable business websites.
By kind of business: highest among hairdressers and beauty salons (34%), lowest among tours, attractions and museums (28%).
Among 101 restaurant indexes, Houston is highest at 45%; the median is 30%.

Figure 1

By kind of business

Kind	Share	Of
restaurants	29%	74,726
cafes and bars	33%	29,566
hairdressers and beauty salons	34%	17,764
dentists, clinics and vets	30%	29,450
tours, attractions and museums	28%	9,910

Figure 2

Restaurants, city by city

another restaurant indexmedian, 30%

0%25%50%

101 restaurant indexes. The median is 30%; the highest is Houston at 45%.

#	Index	Share	Of
1	Houston US	45%	510
2	Chicago US	43%	1,205
3	San Jose US	42%	514
4	Zurich CH	42%	710
5	Orlando US	41%	239
6	Atlanta US	39%	417
7	Bern CH	39%	279
7	Los Angeles US	39%	1,167
9	Charlotte US	39%	301
10	Venice IT	38%	190

All 101 restaurant indexes

#	Index	Share	Of
1	Houston US	45%	510
2	Chicago US	43%	1,205
3	San Jose US	42%	514
4	Zurich CH	42%	710
5	Orlando US	41%	239
6	Atlanta US	39%	417
7	Bern CH	39%	279
7	Los Angeles US	39%	1,167
9	Charlotte US	39%	301
10	Venice IT	38%	190
11	Tokyo JP	38%	412
12	Denver US	38%	653
13	Osaka JP	38%	212
14	Phoenix US	38%	437
15	San Francisco US	37%	901
16	Dallas US	37%	640
17	Philadelphia US	37%	567
18	Ottawa CA	37%	487
19	Austin US	36%	508
20	Warsaw PL	35%	567
21	Montreal CA	35%	700
22	Seattle US	35%	722
23	San Diego US	35%	720
24	Singapore SG	35%	455
25	Basel CH	34%	220
25	Rotterdam NL	34%	408
25	Washington US	34%	1,143
28	Glasgow GB	34%	219
29	Mexico City MX	34%	190
29	Porto PT	34%	439
31	Lisbon PT	34%	986
32	Essen DE	34%	322
33	Hamburg DE	33%	1,873
34	Leipzig DE	33%	377
35	Birmingham GB	33%	275
36	Berlin DE	33%	4,294
36	Rome IT	33%	689
38	The Hague NL	33%	493
39	Munich DE	32%	1,604
40	Boston US	32%	683
40	Eindhoven NL	32%	267
40	Toronto CA	32%	957
43	Istanbul TR	32%	259
44	Frankfurt DE	31%	888
45	Dortmund DE	30%	296
45	Dublin IE	30%	375
45	Nuremberg DE	30%	576
45	Verona IT	30%	553
49	Amsterdam NL	30%	1,192
49	London GB	30%	2,637
51	Brisbane AU	30%	247
51	New York US	30%	3,524
53	Dubai AE	30%	184
53	Stuttgart DE	30%	743
53	Vancouver CA	30%	849
56	Dresden DE	30%	379
57	Geneva CH	30%	288
58	Düsseldorf DE	29%	558
59	Budapest HU	29%	782
59	Madrid ES	29%	971
61	Bremen DE	28%	422
61	Cologne DE	28%	908
63	Manchester GB	28%	205
64	Prague CZ	28%	1,108
65	Calgary CA	28%	558
66	Sydney AU	27%	709
67	Hanover DE	27%	445
68	Copenhagen DK	27%	999
68	Turin IT	27%	267
70	Minneapolis US	27%	389
71	Milan IT	26%	918
71	Vienna AT	26%	2,329
73	Barcelona ES	26%	636
74	Auckland NZ	25%	325
75	Florence IT	25%	187
75	Miami US	25%	2,802
77	Melbourne AU	25%	927
78	Utrecht NL	24%	536
79	Graz AT	23%	265
80	Edinburgh GB	23%	403
81	Oslo NO	23%	407
82	Marseille FR	22%	189
83	Strasbourg FR	22%	353
84	Athens GR	21%	305
85	Nantes FR	21%	353
86	Lille FR	21%	248
87	Bordeaux FR	19%	336
88	Helsinki FI	19%	751
88	Lausanne CH	19%	281
90	Toulouse FR	18%	413
91	Lyon FR	18%	776
92	Valencia ES	18%	248
93	Ghent BE	18%	411
94	Paris FR	18%	2,646
95	Aarhus DK	18%	256
96	Gothenburg SE	17%	406
97	Antwerp BE	17%	392
98	Brussels BE	17%	736
99	Nice FR	15%	346
100	Stockholm SE	15%	1,289
101	Malmö SE	13%	214

Indexes under 200 scanned websites are left out. The "Of" column is the base each figure rests on.

Data, CSV JSON Method Link to this figure

About these numbers
A site turns a crawler away when its robots.txt disallows a known AI crawler or its firewall answers that crawler with an error. Many owners don't know their host or template does it.

Cite this

31% of business websites turn at least one AI crawler away (The Direct Index, 17 October 2026, directindex.org/stats/ai-blocked).

CSV JSON Markdown Method Every number

Data: ODbL. Business data from OpenStreetMap contributors and Overture Maps; website facts from the Direct Index scanner, which reads each site's public pages.