xpu liveBETA

AGENTS & WEB AUTOMATION

AI Search agents & crawlers.

Explore coding agents, AI assistants, search crawlers and web archives.
Compare their roles, operators and documented capabilities.

5agents & crawlersOfficial sources. Clear specifications.

5 agents

Categories can overlap · Listed A–Z
Documented identity

Applebot

Apple

Collects web content for search experiences across Apple products. Applebot and the Applebot-Extended control token have different roles; do not treat a search crawler request as proof of training…

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Claude-SearchBot

Anthropic

Supports web search discovery for Claude. Its search purpose is distinct from ClaudeBot training collection and from a Claude-User request triggered by an individual.

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

OAI-SearchBot

OpenAI

Discovers content for ChatGPT search results. This search crawler has a separate role from GPTBot training collection and ChatGPT-User user-triggered visits. Access does not guarantee a citation.

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

PerplexityBot

Perplexity

Collects web information for Perplexity search discovery. This crawler is separate from the user-initiated Perplexity-User fetcher. A successful visit does not guarantee that an answer will cite the page.

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

PetalBot

Huawei

Builds web indexes used by Petal Search and related Huawei discovery services. Cloudflare may group the identity under an AI collection category, while the documented purpose also includes search.…

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified

This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.