The AI Crawler Registry: 51 crawlers, cited to their operators

Key takeaways

• The registry holds 51 records across 26 operators: the token, who runs it, what it is for, whether it honors robots.txt, and where the operator documents it.
• Read it over REST at /api/v1/crawlers or through the MCP tools lookup_crawler and list_crawlers. Free, no key needed.
• Contested behavior is labeled disputed, with the dispute named. The registry records what operators claim and what independent reporting found, and does not pick a winner.

Why a registry, and why cited

Ask a simple question: which AI crawlers can visit my site, and what does each one do with what it takes? The answers exist, scattered across operator help pages, developer docs, and blocklists of uneven quality. The blocklists copy each other. A user-agent directory copied a wrong purpose from another directory last year, and the error is still circulating.

So the registry rule is strict: the citation of record for a crawler is the operator’s own documentation. OpenAI’s bot page for GPTBot, Anthropic’s support article for ClaudeBot, Apple’s support page for Applebot. Every record carries that URL and a verified_at date, the day a human checked the claim against the page. Where an operator publishes no documentation at all (ByteDance, for one), the record says so in a source_note and rests on two independent secondary sources instead. Silence from an operator is itself a fact worth recording.

What a record holds

Each record answers six questions. What is the token, the string a site owner puts in robots.txt? Who operates it? What is it for: training, search indexing, user-triggered fetching, a mix, or something else? Does it honor robots.txt: yes, no, or disputed? Where does the operator document it, and where are its published IP ranges, if it publishes any? And is it still active, or retired, or renamed into a successor?

The purpose split matters more than it looks. GPTBot takes content for training. OAI-SearchBot builds the index ChatGPT search answers from. Blocking the first costs you nothing in visibility; blocking the second removes you from AI search results. Measurements of the top 5,000 websites found that a large share of sites blocking GPTBot also block the search crawlers, apparently by accident, because a blocklist treated them as one thing. They are not one thing, and the registry keeps them apart record by record.

The disputed column

Five records carry disputed, and the word is doing work. Perplexity documents PerplexityBot as a search crawler that respects robots.txt. Cloudflare’s August 2025 report says Perplexity also crawled with undeclared user agents and rotating addresses to get around blocks. Perplexity rejects that account. Both positions are in the record’s notes. GPTBot is disputed on the same pattern: OpenAI states it honors robots.txt, and Cloudflare’s report plus an Atlantic investigation documented fetches of pages that disallowed it.

A registry that quietly resolved those fights would be lying with extra steps. The honest value is disputed, with names attached.

Two fetchers get no, for a different reason. ChatGPT-User and Perplexity-User fetch pages because a person asked. Both operators document that robots.txt does not govern a user-directed fetch, the same category as a browser visit. That is a scope statement, not defiance, and the records keep the two situations separate: no for out-of-scope by design, disputed for claimed compliance that third parties contest.

Read it three ways

REST: GET /api/v1/crawlers lists records with filters for operator, purpose, and status. GET /api/v1/crawlers/GPTBot returns one record. GET /api/v1/crawlers/meta returns counts, the operator list, and a citation string for the dataset.

MCP: lookup_crawler takes a token and returns the record. list_crawlers searches and filters. Ask your assistant “what does the AI Crawler Registry say about ClaudeBot” and it can answer from the record, not from memory.

Web: the dataset page at /developers/crawlers/ renders every record in a table, states the inclusion rules, and shows how to cite the dataset.

Keeping it true

Crawler fleets change monthly. A dataset like this rots unless someone re-verifies it, so the registry is versioned (this release is v1.0.0), carries a changelog, and gets a monthly diff against the major operators’ documentation, with changed records getting a fresh verified_at. Corrections are welcome through the contact page; a correction with a source beats a correction without one.

What the registry does not do: it covers the robots.txt layer, the crawlers that identify themselves and publish rules. It says nothing about undeclared scraping, and network-layer controls at a CDN edge are a separate decision. A record here is the start of a crawler policy, not the whole of it.