31% of business websites turn at least one AI crawler away

161,416 reachable business websites in 675 indexes, as scanned by 17 October 2026.

Main points

That is 31% of 161,416 reachable business websites.
By kind of business: highest among hairdressers and beauty salons (34%), lowest among tours, attractions and museums (28%).
Among 8 tour and attraction indexes, London is highest at 36%; the median is 31%.

Figure 1

By kind of business

Kind	Share	Of
restaurants	29%	74,726
cafes and bars	33%	29,566
hairdressers and beauty salons	34%	17,764
dentists, clinics and vets	30%	29,450
tours, attractions and museums	28%	9,910

Figure 2

Tours, attractions and museums, city by city

another tour and attraction indexmedian, 31%

0%20%40%

8 tour and attraction indexes. The median is 31%; the highest is London at 36%.

#	Index	Share	Of
1	London GB	36%	304
2	Lisbon PT	36%	324
3	Berlin DE	33%	1,656
4	Miami US	32%	1,306
5	Hamburg DE	30%	278
6	Tokyo JP	28%	243
7	Budapest HU	25%	197
8	Paris FR	19%	189

Indexes under 200 scanned websites are left out. The "Of" column is the base each figure rests on.

Data, CSV JSON Method Link to this figure

About these numbers
A site turns a crawler away when its robots.txt disallows a known AI crawler or its firewall answers that crawler with an error. Many owners don't know their host or template does it.

Cite this

31% of business websites turn at least one AI crawler away (The Direct Index, 17 October 2026, directindex.org/stats/ai-blocked).

CSV JSON Markdown Method Every number

Data: ODbL. Business data from OpenStreetMap contributors and Overture Maps; website facts from the Direct Index scanner, which reads each site's public pages.