Is your site blocking AI crawlers?

You can write the best comparison content in your category and still be invisible in AI answers — because a single line in your robots.txt quietly tells the AI crawlers to stay out. If a model can’t read your site, it can’t recommend you.

The invisible blocker

Most teams set up robots.txt years ago, often with a blanket Disallow or a CMS default, and never look at it again. That was fine when the only thing reading it was Googlebot. But AI assistants — ChatGPT, Claude, Perplexity, Gemini — send their own crawlers, and each one honours robots.txt under a different user-agent name. Block those names, intentionally or not, and you remove yourself from the data those models use to answer buyer questions.

The painful part is that nothing breaks. Your pages still rank, your analytics look normal, and there’s no error to investigate. You only find out you’re excluded when a prospect tells you a competitor came up in ChatGPT and you didn’t — if they tell you at all.

The bots that matter

These are the user-agents worth knowing. Each powers a different surface, so blocking one is not the same as blocking another:

• GPTBot — OpenAI’s training crawler. Feeds the underlying model’s knowledge of your brand.
• OAI-SearchBot — powers ChatGPT Search results and citations.
• ChatGPT-User — fetches a page live when a ChatGPT user (or plugin) browses to it during a conversation.
• ClaudeBot — Anthropic’s crawler for Claude.
• PerplexityBot — indexes the web for Perplexity’s answer engine and citations.
• Google-Extended — the toggle that controls whether your content trains Gemini and feeds AI Overviews. Blocking it does not affect normal Search.
• Googlebot — classic Search, and the same crawl that underpins Google’s AI Overviews and AI Mode.
• Bingbot — Bing Search, which also supplies grounding data to Copilot and, historically, ChatGPT.
• CCBot — Common Crawl, a public dataset many models train on indirectly.

Why blocking makes you invisible

AI assistants build recommendations from what they can read about you — your pages, plus what trusted third parties say. Your own site is usually the highest-authority source on your own product: pricing, features, positioning, the exact wording you want quoted. Block the crawlers and you hand that narrative entirely to review sites, competitors, and stale training data. The model still answers the question; it just answers it without you.

This is the cheapest fix in all of AI visibility. It’s not a content strategy or a backlink campaign — it’s deleting one wrong line. Yet a surprising number of sites ship with AI bots disallowed by default, so it’s the first thing worth checking.

The JavaScript problem

Even with robots.txt wide open, there’s a second, subtler way to be unreadable: rendering your content in JavaScript. Most AI crawlers do not execute JS. They fetch the raw HTML and read whatever is there — nothing more. If your headline, copy, and key facts only appear after a client-side React or Vue bundle runs, an AI crawler sees an almost-empty shell.

Googlebot can render JavaScript, so single-page apps often look fine in Search and feel safe. But GPTBot, ClaudeBot, PerplexityBot and friends typically don’t, which means content that’s perfectly visible to Google can be completely invisible to ChatGPT. The fix is server-side rendering, static generation, or pre-rendering — anything that puts your real content in the initial HTML response.

# Allow AI assistants to read the site
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

How to check and fix it

Checking both problems takes a couple of minutes. For the robots.txt side, run your domain through the AI Bot Access Checker — it fetches yourrobots.txt and tells you exactly which AI user-agents are allowed or blocked. For the rendering side, the AI Crawlability Checker fetches your page the way a crawler does (no JavaScript) and shows you how much real content survives.

To fix what they surface:

• Remove the blocks. Replace any Disallow: / aimed at AI user-agents with the allow rules above.
• Render server-side. Make sure your headline and core copy appear in the raw HTML, not just after JS hydration.
• Keep a sitemap. Point crawlers at your important URLs explicitly.
• Re-check after deploys. CMS upgrades and framework changes love to silently reintroduce blocks.

FAQ

Does blocking GPTBot also block ChatGPT Search? No — ChatGPT Search uses OAI-SearchBot and live browsing uses ChatGPT-User. You need all three allowed to be fully visible in ChatGPT.

If my site ranks on Google, am I safe? Not necessarily. Googlebot renders JavaScript and may be allowed, but most AI crawlers don’t run JS and may be blocked separately, so a page that ranks fine can still be invisible to ChatGPT or Perplexity.

Should I block AI crawlers to protect my content? That’s a real trade-off, but for most brands the upside — being named in AI answers — far outweighs the downside. If discoverability matters to you, allow them.

How do I know it worked? Re-run the free checkers after deploying, then track your mentions over time with Visiblee to confirm the change translates into more AI recommendations.