31% of business websites turn at least one AI crawler away

161,416 reachable business websites in 675 indexes, as scanned by 17 October 2026.

Main points

That is 31% of 161,416 reachable business websites.
By kind of business: highest among hairdressers and beauty salons (34%), lowest among tours, attractions and museums (28%).
Among 43 cafe and bar indexes, Los Angeles is highest at 54%; the median is 34%.

Figure 1

By kind of business

Kind	Share	Of
restaurants	29%	74,726
cafes and bars	33%	29,566
hairdressers and beauty salons	34%	17,764
dentists, clinics and vets	30%	29,450
tours, attractions and museums	28%	9,910

Figure 2

Cafes and bars, city by city

another cafe and bar indexmedian, 34%

0%30%60%

43 cafe and bar indexes. The median is 34%; the highest is Los Angeles at 54%.

#	Index	Share	Of
1	Los Angeles US	54%	373
2	Miami US	46%	717
3	San Diego US	45%	224
4	Philadelphia US	43%	212
5	Zurich CH	41%	307
6	Berlin DE	41%	2,217
6	New York US	41%	1,262
8	Lisbon PT	39%	353
9	Tokyo JP	39%	293
10	Montreal CA	38%	242

All 43 cafe and bar indexes

#	Index	Share	Of
1	Los Angeles US	54%	373
2	Miami US	46%	717
3	San Diego US	45%	224
4	Philadelphia US	43%	212
5	Zurich CH	41%	307
6	Berlin DE	41%	2,217
6	New York US	41%	1,262
8	Lisbon PT	39%	353
9	Tokyo JP	39%	293
10	Montreal CA	38%	242
11	Toronto CA	38%	333
12	Munich DE	37%	447
13	Dublin IE	36%	327
13	Hamburg DE	36%	660
15	Madrid ES	36%	253
16	Cologne DE	36%	345
17	Seattle US	36%	545
18	Chicago US	35%	602
19	Washington, DC US	35%	387
20	London GB	34%	2,079
21	Singapore SG	34%	180
22	Denver US	34%	269
23	Rome IT	33%	179
24	Frankfurt DE	33%	264
25	San Francisco US	33%	335
25	Vienna AT	33%	579
27	Amsterdam NL	32%	469
28	Vancouver CA	32%	335
29	Budapest HU	32%	385
29	Edinburgh GB	32%	331
31	Brisbane AU	32%	250
32	Sydney AU	31%	683
33	Melbourne AU	29%	870
33	Prague CZ	29%	516
35	Boston US	25%	218
35	Milan IT	25%	226
37	Barcelona ES	25%	199
38	Stuttgart DE	24%	243
39	Helsinki FI	24%	263
40	Paris FR	23%	732
41	Stockholm SE	22%	391
42	Brussels BE	18%	240
43	Copenhagen DK	18%	570

Indexes under 200 scanned websites are left out. The "Of" column is the base each figure rests on.

Data, CSV JSON Method Link to this figure

About these numbers
A site turns a crawler away when its robots.txt disallows a known AI crawler or its firewall answers that crawler with an error. Many owners don't know their host or template does it.

Cite this

31% of business websites turn at least one AI crawler away (The Direct Index, 17 October 2026, directindex.org/stats/ai-blocked).

CSV JSON Markdown Method Every number

Data: ODbL. Business data from OpenStreetMap contributors and Overture Maps; website facts from the Direct Index scanner, which reads each site's public pages.