xpu liveBETA

AGENTS & WEB AUTOMATION

Search Engine Crawlers agents & crawlers.

Explore coding agents, AI assistants, search crawlers and web archives.
Compare their roles, operators and documented capabilities.

5agents & crawlersOfficial sources. Clear specifications.

5 agents

Categories can overlap · Listed A–Z
Documented identity

Baidu

Baidu

Crawls pages for the Baidu search engine under the Baiduspider identity. Baidu is the display name in the supplied dashboard; Baiduspider is the relevant crawler family. Search discovery does…

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

BingBot

Microsoft

Discovers and revisits web content for Microsoft Bing search. Bing operates several automated clients. Verify bingbot with the current webmaster guidance rather than trusting the request name alone.

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Googlebot

Google

Discovers web content for Google Search. Google publishes several crawler types and verification methods. A request is not proof of indexing, ranking or use by every Google AI product.

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Terracotta Bot

Ceramic

Builds indexes for Ceramic search. Cloudflare lists Terracotta and Terracotta-News request identifiers. These are web retrieval identities, not general-purpose coding assistants.

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Timpibot

Timpi

Identifies Timpi-related web collection in the supplied Cloudflare list. Timpi describes a web-indexing ecosystem. Exact request authentication and crawl-control details for this token should be confirmed with current operator…

Search Engine Crawlers
Interface
Web crawler / fetcher
License
Operator-managed / not specified

This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.