Identifies Amazon-operated automated web collection. Amazon has multiple crawler identities. Do not transfer the purpose or controls of an advertising or search bot to this token without checking Amazon…
Identifies a ByteDance-associated web crawler family. Cloudflare documents Bytespider strings under Toutiao as well. The supplied AI Crawler label alone does not establish the downstream use of a particular…
Collects web pages for Common Crawl web datasets. Common Crawl supplies web data; it is not itself a conversational model or a promise that a page will appear in…
Collects web content that can support development of Anthropic models. Do not combine its traffic statistics with Claude-User or Claude-SearchBot; they represent different documented purposes.
Names a Meta-associated crawler identity listed in the supplied Cloudflare dashboard. Public documentation coverage for this exact legacy label is limited. Do not silently equate FacebookBot with facebookexternalhit link…
Supports site-owner-requested crawling associated with Google Cloud Vertex AI. This is separate from Googlebot general Search crawling and Google-Extended content-use controls. A visit does not imply a Google Search…
Crawls content that may support training OpenAI foundation models. Its robots.txt setting is independent of OAI-SearchBot. Training collection and search visibility should be evaluated separately.
Collects public web content for Meta AI and product-related indexing purposes. Separate this automated collection identity from the user-initiated Meta-ExternalFetcher.
Provides agent-oriented crawling through a Novellum MCP-related tool. The display label differs from the documented Novellum request token. Cloudflare also links a published signature-agent directory for this identity.
Names the ProRata.ai crawler identity shown in the supplied Cloudflare list. Detailed public documentation for this exact crawler token was not established in this review. Operator product descriptions do…
Names a ByteDance-associated crawler label from the supplied Cloudflare list. An operator-authored specification for this exact token was not established in this review. Do not assume a crawl rate,…
AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
No agents found
Try a different search, category or interface, or reset the filters.
This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.