Collects web content for search experiences across Apple products. Applebot and the Applebot-Extended control token have different roles; do not treat a search crawler request as proof of training…
Supports web search discovery for Claude. Its search purpose is distinct from ClaudeBot training collection and from a Claude-User request triggered by an individual.
Discovers content for ChatGPT search results. This search crawler has a separate role from GPTBot training collection and ChatGPT-User user-triggered visits. Access does not guarantee a citation.
Collects web information for Perplexity search discovery. This crawler is separate from the user-initiated Perplexity-User fetcher. A successful visit does not guarantee that an answer will cite the page.
Builds web indexes used by Petal Search and related Huawei discovery services. Cloudflare may group the identity under an AI collection category, while the documented purpose also includes search.…
AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
No agents found
Try a different search, category or interface, or reset the filters.
This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.