xpu liveBETA

AGENTS & WEB AUTOMATION

Web Archiving agents & crawlers.

Explore coding agents, AI assistants, search crawlers and web archives.
Compare their roles, operators and documented capabilities.

2agents & crawlersOfficial sources. Clear specifications.

2 agents

Categories can overlap · Listed A–Z
Documented identity

archive.org_bot

Internet Archive

Collects public web pages for preservation through the Internet Archive. Archival capture is different from ordinary search indexing or an interactive AI answer. A capture does not guarantee that…

Web Archiving
Interface
Web crawler / fetcher
License
Operator-managed / not specified
Documented identity

Arquivo Web Crawler

Arquivo

Harvests web content for the Portuguese web archive Arquivo.pt. The published crawler examples include Heritrix, Brozzler and Browsertrix identifiers. Keep the exact request string when investigating an archive visit.

Web Archiving
Interface
Web crawler / fetcher
License
Operator-managed / not specified

This directory includes open-source tools and operator-managed web identities. A crawler is not necessarily an installable AI agent. Profiles distinguish purpose, source limits, licenses and costs.