xpu liveBETA

AGENTS & WEB AUTOMATION

AI Crawlers agents & crawlers.

Explore coding agents, AI assistants, search crawlers and web archives.
Compare their roles, operators and documented capabilities.

11agents & crawlersOfficial sources. Clear specifications.

11 agents

Categories can overlap · Listed A–Z
Documented identity

Amazonbot

Amazon

Identifies Amazon-operated automated web collection. Amazon has multiple crawler identities. Do not transfer the purpose or controls of an advertising or search bot to this token without checking Amazon…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Bytespider

ByteDance

Identifies a ByteDance-associated web crawler family. Cloudflare documents Bytespider strings under Toutiao as well. The supplied AI Crawler label alone does not establish the downstream use of a particular…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

CCBot

Common Crawl

Collects web pages for Common Crawl web datasets. Common Crawl supplies web data; it is not itself a conversational model or a promise that a page will appear in…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

ClaudeBot

Anthropic

Collects web content that can support development of Anthropic models. Do not combine its traffic statistics with Claude-User or Claude-SearchBot; they represent different documented purposes.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

FacebookBot

Meta

Names a Meta-associated crawler identity listed in the supplied Cloudflare dashboard. Public documentation coverage for this exact legacy label is limited. Do not silently equate FacebookBot with facebookexternalhit link…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Google-CloudVertexBot

Google

Supports site-owner-requested crawling associated with Google Cloud Vertex AI. This is separate from Googlebot general Search crawling and Google-Extended content-use controls. A visit does not imply a Google Search…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

GPTBot

OpenAI

Crawls content that may support training OpenAI foundation models. Its robots.txt setting is independent of OAI-SearchBot. Training collection and search visibility should be evaluated separately.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Meta-ExternalAgent

Meta

Collects public web content for Meta AI and product-related indexing purposes. Separate this automated collection identity from the user-initiated Meta-ExternalFetcher.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Novellum AI Crawl

Novellum

Provides agent-oriented crawling through a Novellum MCP-related tool. The display label differs from the documented Novellum request token. Cloudflare also links a published signature-agent directory for this identity.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

ProRataInc

ProRata.ai

Names the ProRata.ai crawler identity shown in the supplied Cloudflare list. Detailed public documentation for this exact crawler token was not established in this review. Operator product descriptions do…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

TikTok Spider

ByteDance

Names a ByteDance-associated crawler label from the supplied Cloudflare list. An operator-authored specification for this exact token was not established in this review. Do not assume a crawl rate,…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified

This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.