What the scanner fetches, how to recognise it, and how to keep it out.
It visits a business website once to measure whether AI agents can find, read and act on it, and publishes the result on this site. One visit is nine to twenty requests: the homepage as a normal browser and once as each of GPTBot, ClaudeBot and PerplexityBot (so the report can say what each one is shown), robots.txt, llms.txt, up to three HEAD requests on menu links, and a handful of well-known files such as /.well-known/api-catalog. Requests to the same host are at least two seconds apart. Businesses in a city index are revisited about once a month with a light check; sites whose owners asked for monitoring, weekly. The action test opens a booking or ordering page in a real browser and looks at the controls; it never fills a form, places an order or logs in.
Every request carries Web Bot Auth headers: Signature-Agent: "https://directindex.org", Signature-Input and Signature, signed with the Ed25519 key published at /.well-known/http-message-signatures-directory. Verify the signature and you know the visit came from us. The browser-view request uses an ordinary Chrome user-agent string so that the comparison with the AI crawlers is fair; the action test's browser identifies as DirectIndexBot/0.2 (+https://directindex.org/bot). Requests come from Cloudflare's network.
Add this to your robots.txt and the next visit stops at robots.txt, the scan is skipped, and the business is dropped from the index:
User-agent: DirectIndexBot Disallow: /
A rule for all crawlers (User-agent: *) is reported as a finding rather than treated as an opt-out, because the check is a single page view, like a visitor's, and the point of the report is to tell an owner what agents are shown. To have a page removed without touching robots.txt, write to hello@directindex.org from the business.
The fetched HTML, for the report. No personal data, no cookies, no forms.