---
title: DirectIndexBot: our crawler · The Direct Index
url: https://directindex.org/bot
---

[The Direct Index](https://directindex.org/)
[Scanner](https://directindex.org/scan)[Cities](https://directindex.org/cities)[Guides](https://directindex.org/guides)[Why](https://directindex.org/why)[Method](https://directindex.org/methodology)[About](https://directindex.org/about)

 Our crawler

# DirectIndexBot

What the scanner fetches, how to recognise it, and how to keep it out.

## What it does

It visits a business website once to measure whether AI agents can find, read and act on it, and publishes the result on this site. One visit is nine to twenty requests: the homepage as a normal browser and once as each of GPTBot, ClaudeBot and PerplexityBot (so the report can say what each one is shown), robots.txt, llms.txt, up to three HEAD requests on menu links, and a handful of well-known files such as `/.well-known/api-catalog`. Requests to the same host are at least two seconds apart. Businesses in a city index are revisited about once a month with a light check; sites whose owners asked for monitoring, weekly. The action test opens a booking or ordering page in a real browser and looks at the controls; it never fills a form, places an order or logs in.

## How to recognise it

Every request carries [Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) headers: `Signature-Agent: "https://directindex.org"`, `Signature-Input` and `Signature`, signed with the Ed25519 key published at [/.well-known/http-message-signatures-directory](https://directindex.org/.well-known/http-message-signatures-directory). Verify the signature and you know the visit came from us. The browser-view request uses an ordinary Chrome user-agent string so that the comparison with the AI crawlers is fair; the action test's browser identifies as `DirectIndexBot/0.2 (+https://directindex.org/bot)`. Requests come from Cloudflare's network.

## How to keep it out

Add this to your robots.txt and the next visit stops at robots.txt, the scan is skipped, and the business is dropped from the index:

```
User-agent: DirectIndexBot
Disallow: /
```

A rule for all crawlers (`User-agent: *`) is reported as a finding rather than treated as an opt-out, because the check is a single page view, like a visitor's, and the point of the report is to tell an owner what agents are shown. To have a page removed without touching robots.txt, write to hello@directindex.org from the business.

## What it keeps

The fetched HTML, for the report. No personal data, no cookies, no forms.

#### The Direct Index

[Why direct](https://directindex.org/why)[About and principles](https://directindex.org/about)[Methodology](https://directindex.org/methodology)[Privacy](https://directindex.org/privacy)

#### Use it

[Free scanner](https://directindex.org/scan)[City indexes](https://directindex.org/cities)[Guides](https://directindex.org/guides)[Common fixes](https://directindex.org/fix)[What agent-direct means](https://directindex.org/agent-direct)

#### For agents

[llms.txt](https://directindex.org/llms.txt)[llms-full.txt](https://directindex.org/llms-full.txt)[OpenAPI](https://directindex.org/openapi.json)[MCP server](https://directindex.org/.well-known/mcp/server-card.json)[A2A agent](https://directindex.org/.well-known/agent-card.json)

#### Data

[City data, JSON and Markdown](https://directindex.org/cities)[Sitemap](https://directindex.org/sitemap.xml)

Business listings from OpenStreetMap contributors, ODbL. Scores under the same licence.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://directindex.org/#org","name":"The Direct Index","url":"https://directindex.org/","logo":"https://directindex.org/favicon.svg","founder":{"@type":"Person","name":"Antonio Curado"},"description":"Independent measurement of whether AI agents can find, read and act on a business's own website."},{"@type":"WebSite","@id":"https://directindex.org/#site","url":"https://directindex.org/","name":"The Direct Index","publisher":{"@id":"https://directindex.org/#org"},"inLanguage":"en","potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://directindex.org/scan?url={url}"},"query-input":"required name=url"}},{"@type":"WebAPI","name":"The Direct Index API","url":"https://directindex.org/openapi.json","documentation":"https://directindex.org/llms-full.txt","provider":{"@id":"https://directindex.org/#org"}}]}
```