Available
Paul Gauthier and Aider contributors
A terminal-based AI pair programmer for creating projects and modifying existing repositories. Aider maps relevant code, applies edits and integrates with Git so changes remain easy to inspect.
Coding
- Interface
- CLI
- License
- Apache-2.0
Documented identity
Amazon
Identifies Amazon-operated automated web collection. Amazon has multiple crawler identities. Do not transfer the purpose or controls of an advertising or search bot to this token without checking Amazon…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Anchor
Provides browser infrastructure that AI agents can use to interact with websites. This is a browser service rather than a downloadable foundation model. Cloudflare lists a signature-agent directory for…
Browser Automation
- Interface
- Browser API
- License
- Operator-managed / not specified
Documented identity
Apple
Collects web content for search experiences across Apple products. Applebot and the Applebot-Extended control token have different roles; do not treat a search crawler request as proof of training…
AI Search
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Internet Archive
Collects public web pages for preservation through the Internet Archive. Archival capture is different from ordinary search indexing or an interactive AI answer. A capture does not guarantee that…
Web Archiving
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Arquivo
Harvests web content for the Portuguese web archive Arquivo.pt. The published crawler examples include Heritrix, Brozzler and Browsertrix identifiers. Keep the exact request string when investigating an archive visit.
Web Archiving
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Baidu
Crawls pages for the Baidu search engine under the Baiduspider identity. Baidu is the display name in the supplied dashboard; Baiduspider is the relevant crawler family. Search discovery does…
Search Engine Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Microsoft
Discovers and revisits web content for Microsoft Bing search. Bing operates several automated clients. Verify bingbot with the current webmaster guidance rather than trusting the request name alone.
Search Engine Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
ByteDance
Identifies a ByteDance-associated web crawler family. Cloudflare documents Bytespider strings under Toutiao as well. The supplied AI Crawler label alone does not establish the downstream use of a particular…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Common Crawl
Collects web pages for Common Crawl web datasets. Common Crawl supplies web data; it is not itself a conversational model or a promise that a page will appear in…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
OpenAI
Fetches pages for certain user-initiated ChatGPT and custom GPT actions. This is not OpenAI automatic search indexing. User-initiated access and automatic robots.txt crawling have different documented handling.
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Anthropic
Supports web search discovery for Claude. Its search purpose is distinct from ClaudeBot training collection and from a Claude-User request triggered by an individual.
AI Search
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Anthropic
Retrieves web pages in response to Claude user requests. A user-triggered fetcher, distinct from ClaudeBot training collection and Claude-SearchBot search indexing.
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Anthropic
Collects web content that can support development of Anthropic models. Do not combine its traffic statistics with Claude-User or Claude-SearchBot; they represent different documented purposes.
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Available
Cline Bot Inc.
A coding agent that works with your project files and development tools. Cline is available in an IDE, terminal and desktop, with an SDK for programmatic integration.
CodingWorkflow Automation
- Interface
- CLI · Desktop · IDE · SDK
- License
- Apache-2.0
Documented identity
Cloudflare
Retrieves web content through the Cloudflare Browser Run crawl service. The crawl identity is separate from other Cloudflare browser and scanning services. Its documented behavior does not bypass CAPTCHAs…
Browser Automation
- Interface
- Browser API
- License
- Operator-managed / not specified
Not maintained
Continue Dev, Inc.
An open-source coding agent delivered as a VS Code extension, JetBrains plugin and CLI. Its public code remains available for use and adaptation, but the official repository is no…
Coding
- Interface
- CLI · IDE
- License
- Apache-2.0
Documented identity
DuckDuckGo
Retrieves web content for DuckDuckGo assisted-answer experiences. This identity should be distinguished from DuckDuckBot general search crawling. Check DuckDuckGo documentation for the current purpose and verification details.
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Meta
Names a Meta-associated crawler identity listed in the supplied Cloudflare dashboard. Public documentation coverage for this exact legacy label is limited. Do not silently equate FacebookBot with facebookexternalhit link…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Available
Google
A terminal-based agent from Google for understanding code, editing files and using tools. It combines Gemini access with shell, file-system and web tools, and can be extended with MCP…
CodingResearchWorkflow Automation
- Interface
- CLI · IDE
- License
- Apache-2.0
Documented identity
Google
Supports site-owner-requested crawling associated with Google Cloud Vertex AI. This is separate from Googlebot general Search crawling and Google-Extended content-use controls. A visit does not imply a Google Search…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Google
Discovers web content for Google Search. Google publishes several crawler types and verification methods. A request is not proof of indexing, ranking or use by every Google AI product.
Search Engine Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Available
Block (original creator); AAIF community
A general-purpose agent for development, research and everyday workflows. Goose runs on your machine and connects language models to tools through extensions. Originally built at Block, it is now…
CodingGeneral-purposeResearchWorkflow Automation
- Interface
- API · CLI · Desktop
- License
- Apache-2.0
Documented identity
OpenAI
Crawls content that may support training OpenAI foundation models. Its robots.txt setting is independent of OAI-SearchBot. Training collection and search visibility should be evaluated separately.
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Available
Nous Research
A persistent personal agent with memory, reusable skills and scheduled tasks. Hermes can work through a terminal or messaging gateway and retain useful context across sessions.
CodingGeneral-purposeResearchWorkflow Automation
- Interface
- CLI · Desktop · Messaging
- License
- MIT
Documented identity
Manus
Identifies web activity associated with Manus agent workflows. A browser agent identity is not a separate downloadable model. The initiating task and applicable service terms determine what the workflow…
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Meta
Collects public web content for Meta AI and product-related indexing purposes. Separate this automated collection identity from the user-initiated Meta-ExternalFetcher.
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Meta
Retrieves individual links for user-initiated Meta product features. Keep user-driven fetches distinct from Meta-ExternalAgent automated collection when reading traffic and access rules.
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Mistral
Identifies user-facing web retrieval associated with Mistral services. The fetcher is not an independently selectable language model. Do not infer a specific model version, context limit or API tariff…
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
NDocumented identity
Novellum
Provides agent-oriented crawling through a Novellum MCP-related tool. The display label differs from the documented Novellum request token. Cloudflare also links a published signature-agent directory for this identity.
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
OpenAI
Discovers content for ChatGPT search results. This search crawler has a separate role from GPTBot training collection and ChatGPT-User user-triggered visits. Access does not guarantee a citation.
AI Search
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Available
Open Interpreter team
A coding agent focused on efficient use of lower-cost models. The current official project is a Rust-based agent with configurable model harnesses, terminal workflows and computer-use testing tools.
CodingComputer Use
- Interface
- CLI · IDE
- License
- Apache-2.0
Beta
OpenHands (All Hands AI)
An open-source software-development agent ecosystem. Agent Canvas provides a self-hosted interface for running agents and automations, while the OpenHands agent can edit code and execute development tasks.
CodingWorkflow Automation
- Interface
- CLI · SDK · Web UI
- License
- MIT
Documented identity
Perplexity
Fetches pages in connection with Perplexity user requests. Separate user-driven retrieval from PerplexityBot indexing. Review the operator documentation for the different access and robots.txt behavior.
AI Assistants
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Perplexity
Collects web information for Perplexity search discovery. This crawler is separate from the user-initiated Perplexity-User fetcher. A successful visit does not guarantee that an answer will cite the page.
AI Search
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Huawei
Builds web indexes used by Petal Search and related Huawei discovery services. Cloudflare may group the identity under an AI collection category, while the documented purpose also includes search.…
AI Search
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
ProRata.ai
Names the ProRata.ai crawler identity shown in the supplied Cloudflare list. Detailed public documentation for this exact crawler token was not established in this review. Operator product descriptions do…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Available
Alibaba — Qwen team
An open-source coding agent from the Qwen team. It provides terminal and editor workflows, with additional desktop, browser and messaging interfaces, and supports multiple model protocols.
CodingWorkflow Automation
- Interface
- CLI · Desktop · IDE · Messaging · SDK · Web UI
- License
- Apache-2.0
Superseded
Princeton and Stanford researchers
A research-oriented software-engineering agent that uses tools to investigate and fix repository issues. Its configuration and execution traces support experimentation with autonomous coding systems.
CodingResearch
- Interface
- CLI
- License
- MIT
Documented identity
Ceramic
Builds indexes for Ceramic search. Cloudflare lists Terracotta and Terracotta-News request identifiers. These are web retrieval identities, not general-purpose coding assistants.
Search Engine Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
ByteDance
Names a ByteDance-associated crawler label from the supplied Cloudflare list. An operator-authored specification for this exact token was not established in this review. Do not assume a crawl rate,…
AI Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified
Documented identity
Timpi
Identifies Timpi-related web collection in the supplied Cloudflare list. Timpi describes a web-indexing ecosystem. Exact request authentication and crawl-control details for this token should be confirmed with current operator…
Search Engine Crawlers
- Interface
- Web crawler / fetcher
- License
- Operator-managed / not specified