# AI crawler checker

> Shows which AI crawlers, GPTBot, ClaudeBot and others, can read a page, and why not. Free, no signup. We look this up and don't keep it.

**AI crawler checker. Which AI bots can read your site, and which can't, no signup.** Use it at https://tools.joinvalley.co/ai-crawler-checker

## Which AI crawlers are there? Each one, what it's for, and whether it follows robots.txt.

Most AI companies run separate crawlers for search, for model training and for pages a user asks about. OpenAI's are OAI-SearchBot, GPTBot and ChatGPT-User; Anthropic's Claude-SearchBot, ClaudeBot and Claude-User; Perplexity's PerplexityBot and Perplexity-User. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers.

| robots.txt token | Company | What it's for | robots.txt |
| --- | --- | --- | --- |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | Says it follows robots.txt |
| Claude-SearchBot | Anthropic | Indexes the web to improve Claude's search results | Says it follows robots.txt |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's answers; not used for training | Says it follows robots.txt |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models | Says it follows robots.txt |
| ClaudeBot | Anthropic | Collects web content that could contribute to training Claude | Says it follows robots.txt |
| Google-Extended | Google | A token that controls Gemini training and grounding; doesn't affect Google Search | Read from robots.txt only |
| Applebot-Extended | Apple | A token that controls whether Applebot's crawl trains Apple's models; doesn't crawl | Read from robots.txt only |
| Meta-ExternalAgent | Meta | Crawls for training AI models or indexing content directly | Says it follows robots.txt |
| CCBot | Common Crawl | Builds Common Crawl's open repository of web crawl data | Says it follows robots.txt |
| ChatGPT-User | OpenAI | Visits a page when someone asks ChatGPT about it | Its company says robots.txt rules may not apply |
| Claude-User | Anthropic | Fetches a page when someone asks Claude about it | Says it follows robots.txt |
| Perplexity-User | Perplexity | Visits a page to answer a Perplexity user's question | Its company says it generally ignores robots.txt |
| Meta-ExternalFetcher | Meta | Fetches a link a user shared or asked about | Its company says robots.txt rules may not apply |

From each company's own documentation, read 11 Nov 2026. Crawlers change; check the sources below before you rely on one.

Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (read 2026-11-11), [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (read 2026-11-11), [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots) (read 2026-11-11), [Google Search Central: List of Google's common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (read 2026-11-11), [Apple Support: About Applebot](https://support.apple.com/en-us/119829) (read 2026-11-11), [Meta for Developers: Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (read 2026-11-11), [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (read 2026-11-11)

## How it works

Type a page or a domain. Our server fetches the site's robots.txt and reads it by RFC 9309 for each AI crawler, using the path you typed. It then requests the page once as each crawler that has a user agent of its own, four at a time, and reads only the status. It also reads the page's robots tags and checks for /llms.txt. We don't keep the address or anything we fetched.

Each crawler gets three readings: whether robots.txt allows it, what the live request returned (served, refused, rate-limited or challenged), and an overall answer. A crawler that robots.txt allows but the server refuses is being blocked by a firewall, CDN or server rule, which robots.txt checkers can't see.

Google-Extended and Applebot-Extended aren't crawlers: they're tokens Google and Apple read from robots.txt to decide how their main crawlers' data may be used, so there's nothing to request as them. ChatGPT-User, Perplexity-User and Meta's fetcher act for a person, and their companies say robots.txt may not apply to them.

What it can't see is the crawler's side. Our live request comes from our servers, not the crawler's, so a firewall that checks IP addresses can treat the real crawler differently, in either direction.

## Decide which AI bots get in. Search, training and user fetches are different bots.

Blocking AI isn't one switch. Here's what each company says its crawlers do.

### Search and training are separate

OpenAI says GPTBot crawls content that may train its models and OAI-SearchBot is for search; Anthropic splits ClaudeBot for training from Claude-SearchBot for search; Perplexity says PerplexityBot surfaces sites in its answers and isn't used to train foundation models. You can block training and stay visible in AI search.

Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (read 2026-11-11), [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (read 2026-11-11), [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots) (read 2026-11-11)

### Tokens that don't touch search

Google says Google-Extended controls whether its crawl may train and ground Gemini, has no user agent of its own, and doesn't affect inclusion or ranking in Google Search. Apple says Applebot-Extended doesn't crawl, and that pages disallowing it can still appear in its search results.

Sources: [Google Search Central: List of Google's common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (read 2026-11-11), [Apple Support: About Applebot](https://support.apple.com/en-us/119829) (read 2026-11-11)

### User-requested fetches play by different rules

When a person asks an assistant about a page, the fetch is theirs. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores them; Meta says its fetcher may bypass them. Anthropic says its bots honour robots.txt.

Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (read 2026-11-11), [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots) (read 2026-11-11), [Meta for Developers: Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (read 2026-11-11), [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (read 2026-11-11)

### robots.txt isn't the only gate

A firewall, CDN or server rule can refuse a crawler that robots.txt allows, which is why this page also requests the page as each one. llms.txt is a separate idea: a 2024 proposal for a Markdown file that helps AI agents use a site, whose only required part is an H1. It isn't a standard, so we only report whether it's there.

Being found by AI is one way in. Reaching the buyers already looking at you is another. [Valley](https://www.joinvalley.co/?utm_source=tools_suite)

Sources: [llmstxt.org: The /llms.txt file (Jeremy Howard)](https://llmstxt.org/) (read 2026-11-11)

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (read 2026-11-11)
- [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (read 2026-11-11)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots) (read 2026-11-11)
- [Google Search Central: List of Google's common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (read 2026-11-11)
- [Apple Support: About Applebot](https://support.apple.com/en-us/119829) (read 2026-11-11)
- [Meta for Developers: Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (read 2026-11-11)
- [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (read 2026-11-11)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (read 2026-11-11)
- [llmstxt.org: The /llms.txt file (Jeremy Howard)](https://llmstxt.org/) (read 2026-11-11)

## Questions people ask

### How do I check if AI crawlers can access my site?

Type your site here. You'll see, for each AI crawler, what robots.txt says, what a live request returned and whether it can read the page.

### How do I block GPTBot?

Add User-agent: GPTBot and Disallow: / to robots.txt. OpenAI says disallowing GPTBot signals your content shouldn't be used to train its models.

### Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for search and GPTBot for training; blocking one doesn't block the other.

### What is Google-Extended?

A robots.txt token, not a crawler, that controls whether Google may use your content to train and ground Gemini models.

### Does blocking Google-Extended affect Google Search?

No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal.

### Do AI crawlers follow robots.txt?

It depends on the bot. Anthropic says its bots honour robots.txt; OpenAI says it may not apply to ChatGPT-User, Meta that its fetcher may bypass it, and Perplexity that Perplexity-User generally ignores it.

### Why can't AI crawlers reach my site when robots.txt allows them?

A firewall, CDN or server rule is refusing them. The live request column here shows a refused, rate-limited or challenged response.

### What is llms.txt?

A 2024 proposal for a Markdown file at /llms.txt that helps AI agents use a site. It's not a standard; its only required part is an H1.

### Is this AI crawler checker free?

Yes. It's free and needs no signup. We request the page you ask for and don't keep it.

## Related tools

- [Robots.txt checker and generator](https://tools.joinvalley.co/robots-txt-checker): Tests any URL against any crawler's robots.txt rules, or builds a new file.
- [Readability checker](https://tools.joinvalley.co/readability-checker): Scores any text's Flesch Reading Ease and grade level, and marks the long sentences.
- [Domain age checker](https://tools.joinvalley.co/domain-age-checker): Shows when a domain was registered, its age, expiry and registrar, read from the registry.
- [UTM builder](https://tools.joinvalley.co/utm-builder): Builds UTM links one at a time or in bulk, and shows the GA4 channel each lands in.
- [Link preview checker](https://tools.joinvalley.co/link-preview-checker): Shows how a link previews on LinkedIn, X and Slack, and checks its Open Graph tags.

## Next step

Want replies, not just clean copy? Valley finds buyers showing real buying signals and messages them across email and LinkedIn. [Valley](https://www.joinvalley.co/?utm_source=tools_suite)

Shelf: [09 Inbound, content and search](https://tools.joinvalley.co/inbound-and-content)

---

Tools by Valley, made by [Valley](https://www.joinvalley.co/?utm_source=tools_suite). HTML version: https://tools.joinvalley.co/ai-crawler-checker
