xpu liveBETA

AGENTS & WEB AUTOMATION

AI Assistants agents & crawlers.

Explore coding agents, AI assistants, search crawlers and web archives.
Compare their roles, operators and documented capabilities.

7agents & crawlersOfficial sources. Clear specifications.

7 agents

Categories can overlap · Listed A–Z
Documented identity

ChatGPT-User

OpenAI

Fetches pages for certain user-initiated ChatGPT and custom GPT actions. This is not OpenAI automatic search indexing. User-initiated access and automatic robots.txt crawling have different documented handling.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Claude-User

Anthropic

Retrieves web pages in response to Claude user requests. A user-triggered fetcher, distinct from ClaudeBot training collection and Claude-SearchBot search indexing.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

DuckAssistBot

DuckDuckGo

Retrieves web content for DuckDuckGo assisted-answer experiences. This identity should be distinguished from DuckDuckBot general search crawling. Check DuckDuckGo documentation for the current purpose and verification details.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Manus Bot

Manus

Identifies web activity associated with Manus agent workflows. A browser agent identity is not a separate downloadable model. The initiating task and applicable service terms determine what the workflow…

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Meta-ExternalFetcher

Meta

Retrieves individual links for user-initiated Meta product features. Keep user-driven fetches distinct from Meta-ExternalAgent automated collection when reading traffic and access rules.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

MistralAI-User

Mistral

Identifies user-facing web retrieval associated with Mistral services. The fetcher is not an independently selectable language model. Do not infer a specific model version, context limit or API tariff…

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Perplexity-User

Perplexity

Fetches pages in connection with Perplexity user requests. Separate user-driven retrieval from PerplexityBot indexing. Review the operator documentation for the different access and robots.txt behavior.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified

This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.