Agents Index

AI Crawler Index

Online

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

Visit publisherConnectJSON

What it can do: 9 tools

  • whoami read-only{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consen
  • example read-only{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no conse
  • classify_user_agent read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to ver
  • lookup_crawler read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: operator, category, robots.txt token and stance, user-agent sub
  • list_crawlers read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the re
  • generate_robots_txt read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit t
  • is_verified_crawler_ip read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored
  • whats_changed read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index was built, when each operator source was last fetched, whic
  • changes_since read-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, reco

Connect it

Claude Code
claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/c/mcp-registry-official/mcp
Claude.ai / ChatGPT
Settings → Connectors → Add custom connector
URL: https://www.pathwren.workers.dev/c/mcp-registry-official/mcp
Cursor (~/.cursor/mcp.json)
{
  "mcpServers": {
    "ai-crawler-index": {
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}
VS Code (.vscode/mcp.json)
{
  "servers": {
    "ai-crawler-index": {
      "type": "http",
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}

Questions

What does AI Crawler Index do?

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key It exposes 9 tools to AI assistants.

What tools does AI Crawler Index provide?

9 tools, including whoami, example, classify_user_agent, lookup_crawler, list_crawlers, generate_robots_txt, is_verified_crawler_ip, whats_changed, changes_since.

How do I connect AI Crawler Index to Claude, ChatGPT or Cursor?

Add the remote MCP endpoint https://www.pathwren.workers.dev/c/mcp-registry-official/mcp as a custom connector or MCP server. Copy-paste setup is on this page.

Does AI Crawler Index require an account or API key?

No sign-in was needed to list its tools. Individual tools may still need an account or API key.

Is AI Crawler Index online?

As of 2026-11-19 our probe found it online. It answered 100% of our last 3 checks.

Who publishes AI Crawler Index?

www.pathwren.workers.dev. The official MCP Registry verified that the publisher controls this domain before accepting the listing.

Similar in Web Search & Scraping