Dataset

The AI Crawler Registry

Every AI crawler token a publisher meets in a robots.txt file, identified: who runs it, what it is for, and whether it honors the file it just read. 55 records, 26 operators, dataset version 1.0.1, last verified 2026-11-01.

Browse the API Registry metadata API and MCP docs

Why this exists

Publishers are making one decision and recording another. The pattern in the wild is clear: block the training crawlers, keep the search crawlers that send readers back. Getting that right takes knowing which token does which job, and the tokens keep changing. New AI user-agents appear faster than any hand-kept list survives. This registry is the maintained version of that list, published as data instead of prose.

The split that matters most: 16 records are training crawlers, 14 build search indexes that answer with citations, and 13 fetch a page only when a user asks. Blocking a search or user-triggered agent by mistake costs a site its citations and its on-demand visits, and buys no training protection at all.

How a record gets in

Citation of record. Every record names the operator's own documentation page for that agent. If the operator publishes no documentation, the record says so and names the independent sources it rests on instead.
Dated verification. Every record carries the date it was last checked against its citation. The registry is re-checked monthly against operator documentation and Cloudflare Radar reporting.
Disputes stay disputed. Where credible reporting contradicts an operator's stated robots.txt policy, the record reads disputed and says why. The registry does not resolve disputes by fiat. 5 records are disputed today: Bytespider, DeepSeekBot, MoonshotBot, GPTBot, PerplexityBot.
Nothing is deleted. Retired and renamed agents stay in the dataset with their status marked, because robots.txt files in the wild still name them.
Versioned. Additions bump the patch version, purpose reclassifications bump the minor version and always get a changelog entry. Purpose labels are the sensitive part of this dataset, and they are never edited silently.

Use it

Surface	Address	What you get
REST	`GET /api/v1/crawlers`	All records, filterable by `operator`, `purpose`, `status`
REST	`GET /api/v1/crawlers/GPTBot`	One record by exact token, case-insensitive
REST	`GET /api/v1/crawlers/meta`	Version, counts, coverage, citation
MCP	`lookup_crawler`	One record inside any MCP client, at `https://movahedi.ca/mcp`
MCP	`list_crawlers`	Filtered, paginated browsing of the registry

Free, read-only, no account. Quotas and terms are the same as the rest of the public API.

Cite it

AI Crawler Registry, movahedi.ca (dataset v1.0.1, updated 2026-11-01). https://movahedi.ca/developers/crawlers/

The records

All 55 records in dataset v1.0.1. The documentation link on each row is that record's citation of record.

Token	Operator	Purpose	Honors robots.txt	Status	Docs
`AI2Bot`	Allen Institute for AI	Training	Yes	active	source
`SemanticScholarBot`	Allen Institute for AI	Search index	Yes	active	source
`Amazonbot`	Amazon	Search and training	Yes	active	source
`Amzn-SearchBot`	Amazon	Search index	Yes	active	source
`Amzn-User`	Amazon	User-triggered	No	active	source
`anthropic-ai`	Anthropic	Training	Yes	renamed (ClaudeBot)	source
`Claude-SearchBot`	Anthropic	Search index	Yes	active	source
`Claude-User`	Anthropic	User-triggered	Yes	active	source
`Claude-Web`	Anthropic	User-triggered	Yes	retired	source
`ClaudeBot`	Anthropic	Training	Yes	active	source
`Applebot`	Apple	Search and training	Yes	active	source
`Applebot-Extended`	Apple	Training	Yes	active	source
`Bytespider`	ByteDance	Training	Disputed	active	source
`cohere-ai`	Cohere	Training	Yes	active	source
`CCBot`	Common Crawl	Training	Yes	active	source
`DeepSeekBot`	DeepSeek	Training	Disputed	active	source
`Diffbot`	Diffbot	Other	Yes	active	source
`DuckAssistBot`	DuckDuckGo	User-triggered	Yes	active	source
`FriendlyCrawler`	FriendlyCrawler project	Training	Yes	active	source
`Google-Agent`	Google	User-triggered	No	active	source
`Google-CloudVertexBot`	Google	User-triggered	Yes	active	source
`Google-Extended`	Google	Training	Yes	active	source
`Google-GeminiNotebook`	Google	User-triggered	No	active	source
`Google-InspectionTool`	Google	User-triggered	Yes	active	source
`Google-Read-Aloud`	Google	User-triggered	Yes	active	source
`Googlebot`	Google	Search index	Yes	active	source
`Googlebot-Image`	Google	Search index	Yes	active	source
`Googlebot-News`	Google	Search index	Yes	active	source
`Googlebot-Video`	Google	Search index	Yes	active	source
`GoogleOther`	Google	Other	Yes	active	source
`GoogleOther-Image`	Google	Other	Yes	active	source
`GoogleOther-Video`	Google	Other	Yes	active	source
`Storebot-Google`	Google	Other	Yes	active	source
`ImagesiftBot`	Hive	Training	Yes	active	source
`archive.org_bot`	Internet Archive	Other	Yes	active	source
`facebookexternalhit`	Meta	Other	Yes	active	source
`Meta-ExternalAgent`	Meta	Training	Yes	active	source
`Meta-ExternalFetcher`	Meta	User-triggered	Yes	active	source
`AdIdxBot`	Microsoft	Other	Yes	active	source
`bingbot`	Microsoft	Search index	Yes	active	source
`MSNBot`	Microsoft	Search index	Yes	renamed (bingbot)	source
`MistralAI-User`	Mistral AI	User-triggered	Yes	active	source
`MoonshotBot`	Moonshot AI	Training	Disputed	active	source
`NeevaBot`	Neeva	Search index	Yes	retired	source
`ChatGPT-User`	OpenAI	User-triggered	No	active	source
`GPTBot`	OpenAI	Training	Disputed	active	source
`OAI-AdsBot`	OpenAI	Other	Yes	active	source
`OAI-SearchBot`	OpenAI	Search index	Yes	active	source
`Perplexity-User`	Perplexity	User-triggered	No	active	source
`PerplexityBot`	Perplexity	Search index	Disputed	active	source
`Timpibot`	Timpi	Search index	Yes	active	source
`VelenPublicWebCrawler`	Velen	Training	Yes	active	source
`Omgilibot`	Webz.io	Training	Yes	active	source
`xAI-Bot`	xAI	Search and training	Yes	active	source
`YouBot`	You.com	Search index	Yes	active	source

What this registry does not claim

It records what operators document and what credible reporting shows. It does not test crawler behavior, and a "yes" on robots.txt is a statement of the operator's policy, not a measurement.
A site's real exposure also depends on its CDN and firewall settings, which can block crawlers at the edge no matter what robots.txt says. This dataset covers the robots.txt layer only.
First-documented dates are the earliest confident public documentation dates, not launch claims.

Corrections are welcome and get a changelog entry: contact.