# MapleMarkets Robots & AI Crawling Policy # Institutional Machine Discovery Surface — FarmDash AEO/GEO Compliant # Production Canonical Host: https://maplemarketssite.vercel.app # ============================================================================== # 1. DEFAULT SEARCH CRAWLERS & GENERAL WEB AGENTS # ============================================================================== User-agent: * Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # Standard Web Search Engines User-agent: Googlebot Allow: / Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks User-agent: Bingbot Allow: / Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks User-agent: Applebot Allow: / Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks User-agent: DuckDuckBot Allow: / Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # ============================================================================== # 2. DEDICATED WELCOME BLOCKS FOR 14 AI SEARCH & ANSWER BOTS # ============================================================================== # 1. OpenAI Training & Knowledge Crawler User-agent: GPTBot Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 2. OpenAI Real-Time Conversational Web Fetcher User-agent: ChatGPT-User Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 3. Anthropic Claude Web Fetcher User-agent: ClaudeBot Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 4. Anthropic System Fetcher User-agent: anthropic-ai Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 5. Perplexity Answer Indexing Crawler User-agent: PerplexityBot Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 6. Perplexity Real-Time Search Agent User-agent: Perplexity-Search Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 7. OpenAI SearchBot (SearchGPT) User-agent: OAI-SearchBot Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 8. Common Crawl Foundation Bot User-agent: CCBot Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 9. Google Extended AI Training Crawler User-agent: Google-Extended Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 10. Applebot Extended AI Ingest User-agent: Applebot-Extended Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 11. Meta External AI Agent User-agent: Meta-ExternalAgent Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 12. Cohere AI Engine User-agent: cohere-ai Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 13. Amazon Web AI Bot User-agent: Amazonbot Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # 14. ByteDance AI Crawler User-agent: Bytespider Allow: / Allow: /meso-exchange Allow: /gear-bazaar Allow: /leveling-guide Allow: /anti-ban-protocol Allow: /trust-ledger Allow: /freelance-escrow Allow: /partner Allow: /faq Allow: /directory Allow: /methodology Allow: /guides/ Allow: /compare/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /agent-report.md Allow: /.well-known/* Disallow: /staff Disallow: /operator Disallow: /api/crypto/webhooks # ============================================================================== # 3. CANONICAL SITEMAPS (DUAL SITEMAP ARCHITECTURE) # ============================================================================== Sitemap: https://maplemarketssite.vercel.app/sitemap.xml Sitemap: https://maplemarketssite.vercel.app/sitemap-agents.xml