We use essential cookies to make our site work. With your consent, we may also use non-essential cookies to improve user experience and analyze website traffic. By clicking “Accept,” you agree to our website's cookie use as described in our Cookie Policy. You can change your cookie settings at any time by clicking “Preferences.”
Visiblee MCP is live — connect your AI agents
visiblee

Free AI crawler tool

AI Bot Access Checker

See whether your robots.txt allows the AI crawlers behind ChatGPT, Claude, Perplexity and more. Free, instant, no signup.

Is your site blocking AI crawlers?

Before an AI assistant can mention you, its crawler has to be allowed to read your site. A single line in robots.txt — often added by a plugin or a cautious developer — can block GPTBot, ClaudeBot or PerplexityBot and quietly erase you from that engine. This free tool shows, bot by bot, who can and can't read your site.

The AI bots that matter

  • GPTBot, OAI-SearchBot, ChatGPT-User — OpenAI: GPTBot crawls for model training, OAI-SearchBot indexes for ChatGPT search, ChatGPT-User fetches a page when a user asks about it.
  • ClaudeBot, Claude-SearchBot, Claude-User — Anthropic: training crawl, search indexing, and user-initiated fetches for Claude.
  • PerplexityBot, Perplexity-User — Perplexity's search index and user-initiated fetches.
  • Googlebot & Bingbot — the classic search crawlers also feed Google's AI Overviews / AI Mode and Microsoft Copilot. Blocking them removes you from normal search too, so almost nobody should.
  • Google-Extended & Applebot-Extended — not crawlers but robots.txt control tokens: they tell Google and Apple whether your content may be used for AI training.
  • Bytespider (ByteDance), CCBot (Common Crawl), meta-externalagent (Meta) and Amazonbot (Amazon) — training and index crawlers that feed many models.

Access is necessary, not sufficient

Allowing the crawlers gets you in the door — but being recommended depends on much more: structured data, reviews, comparison content and how often you're cited. Visiblee tracks whether AI assistants actually mention you across ChatGPT, Perplexity and Google's AI surfaces, and tells you what to fix. Unblock your bots above, then run a free scan.

Further readingIs your site blocking AI crawlers?

Frequently asked questions

What does the AI Bot Access Checker do?

It reads your robots.txt and shows, for each major AI bot — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — whether your site allows or blocks it. Blocking these bots is a common, invisible reason brands don't appear in AI answers.

Is it free?

Yes — enter your website for an instant per-bot report, no signup.

Why does AI bot access matter?

If your robots.txt blocks GPTBot or PerplexityBot, those engines can't read your site — so they can't cite or recommend you. Many sites block AI bots by accident (or via a plugin default) and lose AI visibility without realising.

Should I allow every AI bot?

If you want AI visibility, yes — allow the bots behind the assistants your buyers use (OpenAI, Anthropic, Perplexity, Google). Some publishers block training bots like GPTBot or CCBot for content-rights reasons; just know that blocking them reduces your presence in those tools.

What's the difference between these bots?

They do different jobs. Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent) gather content that trains models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Amazonbot) index your site so search and AI answers can cite it — Googlebot and Bingbot also power classic Google and Bing search, so blocking them cuts you out of both. User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page when a user asks about it. Google-Extended and Applebot-Extended aren't crawlers at all — they're robots.txt tokens that control whether Google and Apple may use your content for AI training.

Explore more free AI tools

Part of Visiblee's free toolkit for winning in AI search.

GEO Audit Tool

Score your site's AI-answer readiness: entities, schema, comparison pages and citations.

llms.txt Generator

Generate and validate an llms.txt file so AI crawlers understand your site.

Schema Markup Analyzer

Extract and validate your structured data, and find the AI-critical schema types you're missing.

AI Crawlability Checker

See how much of your content AI crawlers can read without JavaScript.

ChatGPT Brand Visibility Checker

See whether ChatGPT recommends your brand — and which competitors it names instead.

AI Visibility Checker

See whether your brand appears in Google's AI Overview answers — and which sources they cite instead.