AI Crawler Index
Online{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
What it can do: 9 tools
whoamiread-only{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consenexampleread-only{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no conseclassify_user_agentread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verlookup_crawlerread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: operator, category, robots.txt token and stance, user-agent sublist_crawlersread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the regenerate_robots_txtread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit tis_verified_crawler_ipread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirroredwhats_changedread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index was built, when each operator source was last fetched, whicchanges_sinceread-onlyFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, reco
Connect it
claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/c/mcp-registry-official/mcpSettings → Connectors → Add custom connector
URL: https://www.pathwren.workers.dev/c/mcp-registry-official/mcp{
"mcpServers": {
"ai-crawler-index": {
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
}
}
}{
"servers": {
"ai-crawler-index": {
"type": "http",
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
}
}
}Questions
What does AI Crawler Index do?
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key It exposes 9 tools to AI assistants.
What tools does AI Crawler Index provide?
9 tools, including whoami, example, classify_user_agent, lookup_crawler, list_crawlers, generate_robots_txt, is_verified_crawler_ip, whats_changed, changes_since.
How do I connect AI Crawler Index to Claude, ChatGPT or Cursor?
Add the remote MCP endpoint https://www.pathwren.workers.dev/c/mcp-registry-official/mcp as a custom connector or MCP server. Copy-paste setup is on this page.
Does AI Crawler Index require an account or API key?
No sign-in was needed to list its tools. Individual tools may still need an account or API key.
Is AI Crawler Index online?
As of 2026-11-19 our probe found it online. It answered 100% of our last 3 checks.
Who publishes AI Crawler Index?
www.pathwren.workers.dev. The official MCP Registry verified that the publisher controls this domain before accepting the listing.
Similar in Web Search & Scraping
-
HugeDomains: premium domain names for saleOnlineOfficialPopularNew
Search and price premium aftermarket domain names for sale on HugeDomains.com.
-
OxylabsOnlineOfficialPopular
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
-
PadletOnline · sign-in requiredOfficialPopular
Read, search, create and edit Padlet boards: posts, sections, comments, settings and sharing.
-
Atom.com — Premium DomainsOnlineOfficialPopular
Search, appraise, trademark-check, and buy premium brandable domain names from Atom.com.
-
Dm Drogerie MarktOnlineOfficialPopular
Real-time product data, semantic search and more from dm-drogerie markt, Europe's leading drugstore.
-
ZolaOnlineOfficialNew
Find US wedding vendors, registry gifts and invitations, and Zola's wedding planning answers.