Skip to content

Tools is made by Valley.

See how ↗

Tools is made by Valley.

See how ↗

Tools is made by Valley.

See how ↗

Free tool · 09 / GROW

AI crawler checker.Which AI bots can read your site, and which can't, no signup.

Shows which AI crawlers, GPTBot, ClaudeBot and others, can read a page, and why not.

We look this up and don't keep it.

Try: · ·

Checked https://yourcompany.example/ (an example: a made-up site that blocks training crawlers and lets AI search in)

  • AI search: 3 of 3 can read this page.
  • Model training: 1 of 5 can read this page.
  • Fetches a user asked for: 4 of 4 can read this page.
  • Open web dataset: 0 of 1 can read this page.
Each AI crawler: what robots.txt says, what a live request returned, and whether it can read the page
Crawlerrobots.txtLive requestCan it read this page?
OAI-SearchBotOpenAI · AI searchAllowedServed (HTTP 200)Can read this page
Claude-SearchBotAnthropic · AI searchAllowedServed (HTTP 200)Can read this page
PerplexityBotPerplexity · AI searchAllowedServed (HTTP 200)Can read this page
GPTBotOpenAI · Model trainingBlocked (line 5)Served (HTTP 200)Blocked in robots.txt
ClaudeBotAnthropic · Model trainingBlocked (line 5)Served (HTTP 200)Blocked in robots.txt
Google-ExtendedGoogle · Model trainingBlocked (line 5)Not requested: a token, not a crawlerBlocked in robots.txt
Applebot-ExtendedApple · Model trainingAllowedNot requested: a token, not a crawlerCan read this page
Meta-ExternalAgentMeta · Model trainingAllowedRefused (HTTP 403)Allowed in robots.txt, but the server turns it away
CCBotCommon Crawl · Open web datasetBlocked (line 5)Served (HTTP 200)Blocked in robots.txt
ChatGPT-UserOpenAI · Fetches a user asked forAllowedServed (HTTP 200)Can read this page
Claude-UserAnthropic · Fetches a user asked forAllowedServed (HTTP 200)Can read this page
Perplexity-UserPerplexity · Fetches a user asked forAllowedServed (HTTP 200)Can read this page
Meta-ExternalFetcherMeta · Fetches a user asked forAllowedServed (HTTP 200)Can read this page
  • No llms.txt. It's a 2024 proposal, not a standard; we only check whether it's there.

The live request carries each crawler's user agent from our servers. Firewalls that also check a crawler's IP addresses can treat the real one differently. Anthropic doesn't publish full user agent strings, so ours carry its tokens.

Test or build the robots.txt itself →

Being found is half of it. Valley finds the people engaging with your posts or your profile, researches each one and writes to them. See how Valley does it ↗

Built by Valley ↗

NEXT STEP

Fed up with being found by bots and not by buyers.Valley messages the people already circling you.

Book a demo ↗

$500 a month for your first seat, $200 for each seat after that.

The crawlers

Which AI crawlers are there?Each one, what it's for, and whether it follows robots.txt.

Most AI companies run separate crawlers for search, for model training and for pages a user asks about. OpenAI's are OAI-SearchBot, GPTBot and ChatGPT-User; Anthropic's Claude-SearchBot, ClaudeBot and Claude-User; Perplexity's PerplexityBot and Perplexity-User. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers.

AI crawlers and robots.txt tokens: company, purpose and robots.txt behaviour, from each company's documentation
robots.txt tokenCompanyWhat it's forrobots.txt
OAI-SearchBotOpenAISurfaces sites in ChatGPT's search featuresSays it follows robots.txt
Claude-SearchBotAnthropicIndexes the web to improve Claude's search resultsSays it follows robots.txt
PerplexityBotPerplexitySurfaces and links sites in Perplexity's answers; not used for trainingSays it follows robots.txt
GPTBotOpenAICrawls content that may be used to train OpenAI's modelsSays it follows robots.txt
ClaudeBotAnthropicCollects web content that could contribute to training ClaudeSays it follows robots.txt
Google-ExtendedGoogleA token that controls Gemini training and grounding; doesn't affect Google SearchRead from robots.txt only
Applebot-ExtendedAppleA token that controls whether Applebot's crawl trains Apple's models; doesn't crawlRead from robots.txt only
Meta-ExternalAgentMetaCrawls for training AI models or indexing content directlySays it follows robots.txt
CCBotCommon CrawlBuilds Common Crawl's open repository of web crawl dataSays it follows robots.txt
ChatGPT-UserOpenAIVisits a page when someone asks ChatGPT about itIts company says robots.txt rules may not apply
Claude-UserAnthropicFetches a page when someone asks Claude about itSays it follows robots.txt
Perplexity-UserPerplexityVisits a page to answer a Perplexity user's questionIts company says it generally ignores robots.txt
Meta-ExternalFetcherMetaFetches a link a user shared or asked aboutIts company says robots.txt rules may not apply

From each company's own documentation, read 11 Nov 2026. Crawlers change; check the sources below before you rely on one.

From OpenAI: Overview of OpenAI crawlers ↗ · Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗ · Perplexity: Perplexity crawlers ↗ · Google Search Central: List of Google's common crawlers (Google-Extended) ↗ · Apple Support: About Applebot ↗ · Meta for Developers: Meta web crawlers ↗ · Common Crawl: CCBot ↗, read 11 Nov 2026

How it works

How this tool works.And what it can't see.

Type a page or a domain. Our server fetches the site's robots.txt and reads it by RFC 9309 for each AI crawler, using the path you typed. It then requests the page once as each crawler that has a user agent of its own, four at a time, and reads only the status. It also reads the page's robots tags and checks for /llms.txt. We don't keep the address or anything we fetched.

Each crawler gets three readings: whether robots.txt allows it, what the live request returned (served, refused, rate-limited or challenged), and an overall answer. A crawler that robots.txt allows but the server refuses is being blocked by a firewall, CDN or server rule, which robots.txt checkers can't see.

Google-Extended and Applebot-Extended aren't crawlers: they're tokens Google and Apple read from robots.txt to decide how their main crawlers' data may be used, so there's nothing to request as them. ChatGPT-User, Perplexity-User and Meta's fetcher act for a person, and their companies say robots.txt may not apply to them.

What it can't see is the crawler's side. Our live request comes from our servers, not the crawler's, so a firewall that checks IP addresses can treat the real crawler differently, in either direction.

Built by Valley.

Sources

AI crawlers

Decide which AI bots get in.Search, training and user fetches are different bots.

Blocking AI isn't one switch. Here's what each company says its crawlers do.

  1. 01

    Search and training are separate

    OpenAI says GPTBot crawls content that may train its models and OAI-SearchBot is for search; Anthropic splits ClaudeBot for training from Claude-SearchBot for search; Perplexity says PerplexityBot surfaces sites in its answers and isn't used to train foundation models. You can block training and stay visible in AI search.

    From OpenAI: Overview of OpenAI crawlers ↗ · Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗ · Perplexity: Perplexity crawlers ↗, read 11 Nov 2026

  2. 02

    Tokens that don't touch search

    Google says Google-Extended controls whether its crawl may train and ground Gemini, has no user agent of its own, and doesn't affect inclusion or ranking in Google Search. Apple says Applebot-Extended doesn't crawl, and that pages disallowing it can still appear in its search results.

    From Google Search Central: List of Google's common crawlers (Google-Extended) ↗ · Apple Support: About Applebot ↗, read 11 Nov 2026

  3. 03

    User-requested fetches play by different rules

    When a person asks an assistant about a page, the fetch is theirs. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores them; Meta says its fetcher may bypass them. Anthropic says its bots honour robots.txt.

    From OpenAI: Overview of OpenAI crawlers ↗ · Perplexity: Perplexity crawlers ↗ · Meta for Developers: Meta web crawlers ↗ · Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗, read 11 Nov 2026

  4. 04

    robots.txt isn't the only gate

    A firewall, CDN or server rule can refuse a crawler that robots.txt allows, which is why this page also requests the page as each one. llms.txt is a separate idea: a 2024 proposal for a Markdown file that helps AI agents use a site, whose only required part is an H1. It isn't a standard, so we only report whether it's there.

    Being found by AI is one way in. Reaching the buyers already looking at you is another. See how Valley does it ↗

    From llmstxt.org: The /llms.txt file (Jeremy Howard) ↗, read 11 Nov 2026

Questions

Questions people ask.

How do I check if AI crawlers can access my site?
Type your site here. You'll see, for each AI crawler, what robots.txt says, what a live request returned and whether it can read the page.
How do I block GPTBot?
Add User-agent: GPTBot and Disallow: / to robots.txt. OpenAI says disallowing GPTBot signals your content shouldn't be used to train its models.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses OAI-SearchBot for search and GPTBot for training; blocking one doesn't block the other.
What is Google-Extended?
A robots.txt token, not a crawler, that controls whether Google may use your content to train and ground Gemini models.
Does blocking Google-Extended affect Google Search?
No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal.
Do AI crawlers follow robots.txt?
It depends on the bot. Anthropic says its bots honour robots.txt; OpenAI says it may not apply to ChatGPT-User, Meta that its fetcher may bypass it, and Perplexity that Perplexity-User generally ignores it.
Why can't AI crawlers reach my site when robots.txt allows them?
A firewall, CDN or server rule is refusing them. The live request column here shows a refused, rate-limited or challenged response.
What is llms.txt?
A 2024 proposal for a Markdown file at /llms.txt that helps AI agents use a site. It's not a standard; its only required part is an H1.
Is this AI crawler checker free?
Yes. It's free and needs no signup. We request the page you ask for and don't keep it.