xpu liveBETA

AGENTS & WEB AUTOMATION

Explore agents & crawlers.

Explore coding agents, AI assistants, search crawlers and web archives.
Compare their roles, operators and documented capabilities.

42agents & crawlersOfficial sources. Clear specifications.

42 agents

Categories can overlap · Listed A–Z
Available

Aider

Paul Gauthier and Aider contributors

A terminal-based AI pair programmer for creating projects and modifying existing repositories. Aider maps relevant code, applies edits and integrates with Git so changes remain easy to inspect.

Coding
Interface
CLI
License
Apache-2.0
Documented identity

Amazonbot

Amazon

Identifies Amazon-operated automated web collection. Amazon has multiple crawler identities. Do not transfer the purpose or controls of an advertising or search bot to this token without checking Amazon…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Anchor Browser

Anchor

Provides browser infrastructure that AI agents can use to interact with websites. This is a browser service rather than a downloadable foundation model. Cloudflare lists a signature-agent directory for…

Browser Automation
Interface
Browser API
License
Operator-managed / not specified
Documented identity

Applebot

Apple

Collects web content for search experiences across Apple products. Applebot and the Applebot-Extended control token have different roles; do not treat a search crawler request as proof of training…

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

archive.org_bot

Internet Archive

Collects public web pages for preservation through the Internet Archive. Archival capture is different from ordinary search indexing or an interactive AI answer. A capture does not guarantee that…

Web Archiving
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Arquivo Web Crawler

Arquivo

Harvests web content for the Portuguese web archive Arquivo.pt. The published crawler examples include Heritrix, Brozzler and Browsertrix identifiers. Keep the exact request string when investigating an archive visit.

Web Archiving
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Baidu

Baidu

Crawls pages for the Baidu search engine under the Baiduspider identity. Baidu is the display name in the supplied dashboard; Baiduspider is the relevant crawler family. Search discovery does…

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

BingBot

Microsoft

Discovers and revisits web content for Microsoft Bing search. Bing operates several automated clients. Verify bingbot with the current webmaster guidance rather than trusting the request name alone.

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Bytespider

ByteDance

Identifies a ByteDance-associated web crawler family. Cloudflare documents Bytespider strings under Toutiao as well. The supplied AI Crawler label alone does not establish the downstream use of a particular…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

CCBot

Common Crawl

Collects web pages for Common Crawl web datasets. Common Crawl supplies web data; it is not itself a conversational model or a promise that a page will appear in…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

ChatGPT-User

OpenAI

Fetches pages for certain user-initiated ChatGPT and custom GPT actions. This is not OpenAI automatic search indexing. User-initiated access and automatic robots.txt crawling have different documented handling.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Claude-SearchBot

Anthropic

Supports web search discovery for Claude. Its search purpose is distinct from ClaudeBot training collection and from a Claude-User request triggered by an individual.

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Claude-User

Anthropic

Retrieves web pages in response to Claude user requests. A user-triggered fetcher, distinct from ClaudeBot training collection and Claude-SearchBot search indexing.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

ClaudeBot

Anthropic

Collects web content that can support development of Anthropic models. Do not combine its traffic statistics with Claude-User or Claude-SearchBot; they represent different documented purposes.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Available

Cline

Cline Bot Inc.

A coding agent that works with your project files and development tools. Cline is available in an IDE, terminal and desktop, with an SDK for programmatic integration.

CodingWorkflow Automation
Interface
CLI · Desktop · IDE · SDK
License
Apache-2.0
Documented identity

Cloudflare Crawler

Cloudflare

Retrieves web content through the Cloudflare Browser Run crawl service. The crawl identity is separate from other Cloudflare browser and scanning services. Its documented behavior does not bypass CAPTCHAs…

Browser Automation
Interface
Browser API
License
Operator-managed / not specified
Not maintained

Continue

Continue Dev, Inc.

An open-source coding agent delivered as a VS Code extension, JetBrains plugin and CLI. Its public code remains available for use and adaptation, but the official repository is no…

Coding
Interface
CLI · IDE
License
Apache-2.0
Documented identity

DuckAssistBot

DuckDuckGo

Retrieves web content for DuckDuckGo assisted-answer experiences. This identity should be distinguished from DuckDuckBot general search crawling. Check DuckDuckGo documentation for the current purpose and verification details.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

FacebookBot

Meta

Names a Meta-associated crawler identity listed in the supplied Cloudflare dashboard. Public documentation coverage for this exact legacy label is limited. Do not silently equate FacebookBot with facebookexternalhit link…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Available

Gemini CLI

Google

A terminal-based agent from Google for understanding code, editing files and using tools. It combines Gemini access with shell, file-system and web tools, and can be extended with MCP…

CodingResearchWorkflow Automation
Interface
CLI · IDE
License
Apache-2.0
Documented identity

Google-CloudVertexBot

Google

Supports site-owner-requested crawling associated with Google Cloud Vertex AI. This is separate from Googlebot general Search crawling and Google-Extended content-use controls. A visit does not imply a Google Search…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Googlebot

Google

Discovers web content for Google Search. Google publishes several crawler types and verification methods. A request is not proof of indexing, ranking or use by every Google AI product.

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Available

Goose

Block (original creator); AAIF community

A general-purpose agent for development, research and everyday workflows. Goose runs on your machine and connects language models to tools through extensions. Originally built at Block, it is now…

CodingGeneral-purposeResearchWorkflow Automation
Interface
API · CLI · Desktop
License
Apache-2.0
Documented identity

GPTBot

OpenAI

Crawls content that may support training OpenAI foundation models. Its robots.txt setting is independent of OAI-SearchBot. Training collection and search visibility should be evaluated separately.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Available

Hermes Agent

Nous Research

A persistent personal agent with memory, reusable skills and scheduled tasks. Hermes can work through a terminal or messaging gateway and retain useful context across sessions.

CodingGeneral-purposeResearchWorkflow Automation
Interface
CLI · Desktop · Messaging
License
MIT
Documented identity

Manus Bot

Manus

Identifies web activity associated with Manus agent workflows. A browser agent identity is not a separate downloadable model. The initiating task and applicable service terms determine what the workflow…

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Meta-ExternalAgent

Meta

Collects public web content for Meta AI and product-related indexing purposes. Separate this automated collection identity from the user-initiated Meta-ExternalFetcher.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Meta-ExternalFetcher

Meta

Retrieves individual links for user-initiated Meta product features. Keep user-driven fetches distinct from Meta-ExternalAgent automated collection when reading traffic and access rules.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

MistralAI-User

Mistral

Identifies user-facing web retrieval associated with Mistral services. The fetcher is not an independently selectable language model. Do not infer a specific model version, context limit or API tariff…

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Novellum AI Crawl

Novellum

Provides agent-oriented crawling through a Novellum MCP-related tool. The display label differs from the documented Novellum request token. Cloudflare also links a published signature-agent directory for this identity.

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

OAI-SearchBot

OpenAI

Discovers content for ChatGPT search results. This search crawler has a separate role from GPTBot training collection and ChatGPT-User user-triggered visits. Access does not guarantee a citation.

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Available

Open Interpreter

Open Interpreter team

A coding agent focused on efficient use of lower-cost models. The current official project is a Rust-based agent with configurable model harnesses, terminal workflows and computer-use testing tools.

CodingComputer Use
Interface
CLI · IDE
License
Apache-2.0
Beta

OpenHands

OpenHands (All Hands AI)

An open-source software-development agent ecosystem. Agent Canvas provides a self-hosted interface for running agents and automations, while the OpenHands agent can edit code and execute development tasks.

CodingWorkflow Automation
Interface
CLI · SDK · Web UI
License
MIT
Documented identity

Perplexity-User

Perplexity

Fetches pages in connection with Perplexity user requests. Separate user-driven retrieval from PerplexityBot indexing. Review the operator documentation for the different access and robots.txt behavior.

AI Assistants
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

PerplexityBot

Perplexity

Collects web information for Perplexity search discovery. This crawler is separate from the user-initiated Perplexity-User fetcher. A successful visit does not guarantee that an answer will cite the page.

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

PetalBot

Huawei

Builds web indexes used by Petal Search and related Huawei discovery services. Cloudflare may group the identity under an AI collection category, while the documented purpose also includes search.…

AI Search
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

ProRataInc

ProRata.ai

Names the ProRata.ai crawler identity shown in the supplied Cloudflare list. Detailed public documentation for this exact crawler token was not established in this review. Operator product descriptions do…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Available

Qwen Code

Alibaba — Qwen team

An open-source coding agent from the Qwen team. It provides terminal and editor workflows, with additional desktop, browser and messaging interfaces, and supports multiple model protocols.

CodingWorkflow Automation
Interface
CLI · Desktop · IDE · Messaging · SDK · Web UI
License
Apache-2.0
Superseded

SWE-agent

Princeton and Stanford researchers

A research-oriented software-engineering agent that uses tools to investigate and fix repository issues. Its configuration and execution traces support experimentation with autonomous coding systems.

CodingResearch
Interface
CLI
License
MIT
Documented identity

Terracotta Bot

Ceramic

Builds indexes for Ceramic search. Cloudflare lists Terracotta and Terracotta-News request identifiers. These are web retrieval identities, not general-purpose coding assistants.

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

TikTok Spider

ByteDance

Names a ByteDance-associated crawler label from the supplied Cloudflare list. An operator-authored specification for this exact token was not established in this review. Do not assume a crawl rate,…

AI Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Timpibot

Timpi

Identifies Timpi-related web collection in the supplied Cloudflare list. Timpi describes a web-indexing ecosystem. Exact request authentication and crawl-control details for this token should be confirmed with current operator…

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified

This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.