# searchshop.ai — crawl policy

# Disallow lines come BEFORE `Allow: /` on purpose: real crawlers (Google/OpenAI) use
# longest-match so order is moot for them, but some parsers (incl. older urllib.robotparser,
# the CI guard) use FIRST-match — with `Allow: /` first they wrongly ALLOW /audit. Disallow-first
# is correct under both algorithms and never affects /blog. (2026-07-16)
User-agent: *
# Retired A/B homepage draft — block the specific file ONLY.
# NEVER add a bare `Disallow: /b` — robots.txt Disallow is a PREFIX match, so `/b`
# also blocks /blog and every /blog/* post (it did, silently, until 2026-07-16).
Disallow: /index-b.html
# auth-gated internal tooling
Disallow: /audit
# Client + prospect deliverables. /reports is Basic-Auth gated EXCEPT the paths listed in
# middleware.ts PUBLIC_REPORT_PREFIXES, which are served as bare shareable links.
#
# DELIBERATE TENSION, do not "fix" this by deleting the Disallow. Disallow and noindex pull in
# opposite directions: a crawler that obeys Disallow never fetches the page, so it never sees the
# noindex meta or the X-Robots-Tag header — which means a URL discovered elsewhere could still be
# listed URL-only. Google's own advice is therefore "allow crawling if you want noindex honoured".
# We keep the Disallow anyway, because the PRIMARY risk here is not a search listing: it is a
# teardown naming a real company being ingested as crawlable content ABOUT that company, which is
# exactly the mechanism these reports warn clients about. Allowing the fetch to make noindex work
# would hand the corpus the thing we are trying to withhold.
# Defence in depth for crawlers that fetch anyway: X-Robots-Tag on /reports/(.*) in vercel.json,
# plus the per-page meta injected by the portal builders.
# Trailing slash is deliberate — a bare `/reports` would also prefix-match /reports-anything.
Disallow: /reports/
Disallow: /brand-layer
Allow: /

# ── AI crawlers — explicitly allowed everywhere EXCEPT the draft + internal tooling below ──
# (OpenAI, Anthropic, Perplexity, Google AI, Apple, Meta, Amazon — training, live browsing, citations)
# NOTE: a named user-agent group does NOT inherit the `User-agent: *` rules — a crawler obeys only its
# most-specific matching group. So the draft/internal Disallows MUST be repeated here, or these bots
# would crawl /index-b.html and /audit despite the * block above.
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /index-b.html
Disallow: /audit
Disallow: /reports/
Disallow: /brand-layer
Allow: /

Sitemap: https://www.searchshop.ai/sitemap.xml