robots.txt for AI Crawlers: The 10-Minute Fix That Unblocks ChatGPT Citations

Somewhere along the way — a security plugin update, a well-meaning blog post about AI scraping, a Cloudflare default — your site may have told every AI answer engine on earth to go away. If so, no content strategy on the planet can get you cited. This is the highest-leverage 10 minutes in GEO.

The bots that decide whether AI can cite you

• OAI-SearchBot — powers ChatGPT's search answers
• ChatGPT-User — fetches your page live when a user's chat needs it (blocking this one kills real-time citation even if everything else is open)
• GPTBot — OpenAI's training crawler
• PerplexityBot — Perplexity's answer engine
• ClaudeBot — Anthropic
• Google-Extended — Gemini training control (does NOT affect AI Overviews, which rides on normal Google Search)
• Bingbot — feeds Copilot and much of ChatGPT search

Check your robots.txt in 30 seconds

Open yourdomain.com/robots.txt. Look for any of the user agents above followed by Disallow: / . If you see them, you're blocking. Also look for a blanket rule like this, which catches everything:

User-agent: *
Disallow: /   # ← if this exists with no bot-specific exceptions, you're invisible

The baseline robots.txt that allows AI answers

Most brands' actual preference is: allow answer/search bots, keep training-only choices for later. Here's the file we ship in the Technical Fix Pack:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart
Disallow: /checkout

Sitemap: https://yourdomain.com/sitemap.xml

The plot twist: robots.txt is only half the story

robots.txt is a request. Your firewall is a bouncer. Cloudflare's bot rules, AWS WAF, and security plugins can block AI user agents at the network edge regardless of what robots.txt says. Test from a server, not your browser:

for UA in "GPTBot" "OAI-SearchBot" "ChatGPT-User" "PerplexityBot" "ClaudeBot" "bingbot"; do
  echo "== $UA =="
  curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" https://yourdomain.com/
done

Any 403 or 503 means a WAF rule is blocking. Check Cloudflare's 'Bot Fight Mode' and managed AI-bot rules, then whitelist verified AI crawlers. Re-run this monthly — platform defaults change.

The third silent killer: JavaScript-only pages

Most AI crawlers don't execute JavaScript. If your product descriptions, pricing or articles only appear after client-side rendering, the bots see an empty shell. The test: curl your key page and grep for a sentence from the visible content. If it's not in the raw HTML, move those pages to static generation or server-side rendering.

Do this now

1. Audit robots.txt for AI bot blocks (2 min)
2. Replace with the baseline above (2 min)
3. Run the curl test; fix any WAF blocks (5 min)
4. curl-grep your three most important pages for real content (1 min)
5. Put a monthly reminder to re-test — rules drift

Ten minutes. For a surprising number of sites, this single fix is the difference between 'invisible to AI' and 'citable by AI.' It's checkpoint #1 of the 120 in the full CiteSight audit for a reason.