Free tool · 09 / GROW
AI crawler checker.Which AI bots can read your site, and which can't, no signup.
Shows which AI crawlers, GPTBot, ClaudeBot and others, can read a page, and why not.
We look this up and don't keep it.
Try: · ·
Checked https://yourcompany.example/ (an example: a made-up site that blocks training crawlers and lets AI search in)
- AI search: 3 of 3 can read this page.
- Model training: 1 of 5 can read this page.
- Fetches a user asked for: 4 of 4 can read this page.
- Open web dataset: 0 of 1 can read this page.
| Crawler | robots.txt | Live request | Can it read this page? |
|---|---|---|---|
| OAI-SearchBotOpenAI · AI search | Allowed | Served (HTTP 200) | Can read this page |
| Claude-SearchBotAnthropic · AI search | Allowed | Served (HTTP 200) | Can read this page |
| PerplexityBotPerplexity · AI search | Allowed | Served (HTTP 200) | Can read this page |
| GPTBotOpenAI · Model training | Blocked (line 5) | Served (HTTP 200) | Blocked in robots.txt |
| ClaudeBotAnthropic · Model training | Blocked (line 5) | Served (HTTP 200) | Blocked in robots.txt |
| Google-ExtendedGoogle · Model training | Blocked (line 5) | Not requested: a token, not a crawler | Blocked in robots.txt |
| Applebot-ExtendedApple · Model training | Allowed | Not requested: a token, not a crawler | Can read this page |
| Meta-ExternalAgentMeta · Model training | Allowed | Refused (HTTP 403) | Allowed in robots.txt, but the server turns it away |
| CCBotCommon Crawl · Open web dataset | Blocked (line 5) | Served (HTTP 200) | Blocked in robots.txt |
| ChatGPT-UserOpenAI · Fetches a user asked for | Allowed | Served (HTTP 200) | Can read this page |
| Claude-UserAnthropic · Fetches a user asked for | Allowed | Served (HTTP 200) | Can read this page |
| Perplexity-UserPerplexity · Fetches a user asked for | Allowed | Served (HTTP 200) | Can read this page |
| Meta-ExternalFetcherMeta · Fetches a user asked for | Allowed | Served (HTTP 200) | Can read this page |
- No llms.txt. It's a 2024 proposal, not a standard; we only check whether it's there.
The live request carries each crawler's user agent from our servers. Firewalls that also check a crawler's IP addresses can treat the real one differently. Anthropic doesn't publish full user agent strings, so ours carry its tokens.
Test or build the robots.txt itself →
Being found is half of it. Valley finds the people engaging with your posts or your profile, researches each one and writes to them. See how Valley does it ↗
NEXT STEP
Fed up with being found by bots and not by buyers.Valley messages the people already circling you.
Book a demo ↗$500 a month for your first seat, $200 for each seat after that.
The crawlers
Which AI crawlers are there?Each one, what it's for, and whether it follows robots.txt.
Most AI companies run separate crawlers for search, for model training and for pages a user asks about. OpenAI's are OAI-SearchBot, GPTBot and ChatGPT-User; Anthropic's Claude-SearchBot, ClaudeBot and Claude-User; Perplexity's PerplexityBot and Perplexity-User. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers.
| robots.txt token | Company | What it's for | robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | Says it follows robots.txt |
| Claude-SearchBot | Anthropic | Indexes the web to improve Claude's search results | Says it follows robots.txt |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's answers; not used for training | Says it follows robots.txt |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models | Says it follows robots.txt |
| ClaudeBot | Anthropic | Collects web content that could contribute to training Claude | Says it follows robots.txt |
| Google-Extended | A token that controls Gemini training and grounding; doesn't affect Google Search | Read from robots.txt only | |
| Applebot-Extended | Apple | A token that controls whether Applebot's crawl trains Apple's models; doesn't crawl | Read from robots.txt only |
| Meta-ExternalAgent | Meta | Crawls for training AI models or indexing content directly | Says it follows robots.txt |
| CCBot | Common Crawl | Builds Common Crawl's open repository of web crawl data | Says it follows robots.txt |
| ChatGPT-User | OpenAI | Visits a page when someone asks ChatGPT about it | Its company says robots.txt rules may not apply |
| Claude-User | Anthropic | Fetches a page when someone asks Claude about it | Says it follows robots.txt |
| Perplexity-User | Perplexity | Visits a page to answer a Perplexity user's question | Its company says it generally ignores robots.txt |
| Meta-ExternalFetcher | Meta | Fetches a link a user shared or asked about | Its company says robots.txt rules may not apply |
From each company's own documentation, read 11 Nov 2026. Crawlers change; check the sources below before you rely on one.
From OpenAI: Overview of OpenAI crawlers ↗ · Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗ · Perplexity: Perplexity crawlers ↗ · Google Search Central: List of Google's common crawlers (Google-Extended) ↗ · Apple Support: About Applebot ↗ · Meta for Developers: Meta web crawlers ↗ · Common Crawl: CCBot ↗, read 11 Nov 2026
How it works
How this tool works.And what it can't see.
Type a page or a domain. Our server fetches the site's robots.txt and reads it by RFC 9309 for each AI crawler, using the path you typed. It then requests the page once as each crawler that has a user agent of its own, four at a time, and reads only the status. It also reads the page's robots tags and checks for /llms.txt. We don't keep the address or anything we fetched.
Each crawler gets three readings: whether robots.txt allows it, what the live request returned (served, refused, rate-limited or challenged), and an overall answer. A crawler that robots.txt allows but the server refuses is being blocked by a firewall, CDN or server rule, which robots.txt checkers can't see.
Google-Extended and Applebot-Extended aren't crawlers: they're tokens Google and Apple read from robots.txt to decide how their main crawlers' data may be used, so there's nothing to request as them. ChatGPT-User, Perplexity-User and Meta's fetcher act for a person, and their companies say robots.txt may not apply to them.
What it can't see is the crawler's side. Our live request comes from our servers, not the crawler's, so a firewall that checks IP addresses can treat the real crawler differently, in either direction.
Built by Valley.
Sources
- OpenAI: Overview of OpenAI crawlers ↗read 11 Nov 2026
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗read 11 Nov 2026
- Perplexity: Perplexity crawlers ↗read 11 Nov 2026
- Google Search Central: List of Google's common crawlers (Google-Extended) ↗read 11 Nov 2026
- Apple Support: About Applebot ↗read 11 Nov 2026
- Meta for Developers: Meta web crawlers ↗read 11 Nov 2026
- Common Crawl: CCBot ↗read 11 Nov 2026
- RFC 9309: Robots Exclusion Protocol ↗read 11 Nov 2026
- llmstxt.org: The /llms.txt file (Jeremy Howard) ↗read 11 Nov 2026
AI crawlers
Decide which AI bots get in.Search, training and user fetches are different bots.
Blocking AI isn't one switch. Here's what each company says its crawlers do.
- 01
Search and training are separate
OpenAI says GPTBot crawls content that may train its models and OAI-SearchBot is for search; Anthropic splits ClaudeBot for training from Claude-SearchBot for search; Perplexity says PerplexityBot surfaces sites in its answers and isn't used to train foundation models. You can block training and stay visible in AI search.
From OpenAI: Overview of OpenAI crawlers ↗ · Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗ · Perplexity: Perplexity crawlers ↗, read 11 Nov 2026
- 02
Tokens that don't touch search
Google says Google-Extended controls whether its crawl may train and ground Gemini, has no user agent of its own, and doesn't affect inclusion or ranking in Google Search. Apple says Applebot-Extended doesn't crawl, and that pages disallowing it can still appear in its search results.
From Google Search Central: List of Google's common crawlers (Google-Extended) ↗ · Apple Support: About Applebot ↗, read 11 Nov 2026
- 03
User-requested fetches play by different rules
When a person asks an assistant about a page, the fetch is theirs. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores them; Meta says its fetcher may bypass them. Anthropic says its bots honour robots.txt.
From OpenAI: Overview of OpenAI crawlers ↗ · Perplexity: Perplexity crawlers ↗ · Meta for Developers: Meta web crawlers ↗ · Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? ↗, read 11 Nov 2026
- 04
robots.txt isn't the only gate
A firewall, CDN or server rule can refuse a crawler that robots.txt allows, which is why this page also requests the page as each one. llms.txt is a separate idea: a 2024 proposal for a Markdown file that helps AI agents use a site, whose only required part is an H1. It isn't a standard, so we only report whether it's there.
Being found by AI is one way in. Reaching the buyers already looking at you is another. See how Valley does it ↗
From llmstxt.org: The /llms.txt file (Jeremy Howard) ↗, read 11 Nov 2026
Questions
Questions people ask.
- How do I check if AI crawlers can access my site?
- Type your site here. You'll see, for each AI crawler, what robots.txt says, what a live request returned and whether it can read the page.
- How do I block GPTBot?
- Add User-agent: GPTBot and Disallow: / to robots.txt. OpenAI says disallowing GPTBot signals your content shouldn't be used to train its models.
- Does blocking GPTBot remove my site from ChatGPT search?
- No. OpenAI uses OAI-SearchBot for search and GPTBot for training; blocking one doesn't block the other.
- What is Google-Extended?
- A robots.txt token, not a crawler, that controls whether Google may use your content to train and ground Gemini models.
- Does blocking Google-Extended affect Google Search?
- No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal.
- Do AI crawlers follow robots.txt?
- It depends on the bot. Anthropic says its bots honour robots.txt; OpenAI says it may not apply to ChatGPT-User, Meta that its fetcher may bypass it, and Perplexity that Perplexity-User generally ignores it.
- Why can't AI crawlers reach my site when robots.txt allows them?
- A firewall, CDN or server rule is refusing them. The live request column here shows a refused, rate-limited or challenged response.
- What is llms.txt?
- A 2024 proposal for a Markdown file at /llms.txt that helps AI agents use a site. It's not a standard; its only required part is an H1.
- Is this AI crawler checker free?
- Yes. It's free and needs no signup. We request the page you ask for and don't keep it.
Keep going
Try one of these next.
- Robots.txt checker and generatorInbound, content and searchTests any URL against any crawler's robots.txt rules, or builds a new file.Open the tool
- Readability checkerMessagingScores any text's Flesch Reading Ease and grade level, and marks the long sentences.Open the tool
- Domain age checkerEmail outreachShows when a domain was registered, its age, expiry and registrar, read from the registry.Open the tool
- UTM builderGTM opsBuilds UTM links one at a time or in bulk, and shows the GA4 channel each lands in.Open the tool
- Link preview checkerInbound, content and searchShows how a link previews on LinkedIn, X and Slack, and checks its Open Graph tags.Open the tool