# Robots.txt checker and generator

> Tests any URL against any crawler's robots.txt rules, or builds a new file. Free, no signup. We look this up and don't keep it.

**Robots.txt checker and generator. Test any URL against any crawler, or build a file, no signup.** Use it at https://tools.joinvalley.co/robots-txt-checker

## How does robots.txt matching work? RFC 9309's rules, and where Google adds its own.

A crawler follows the group that names its user agent, or the * group if none does. Within that group, the rule with the longest matching path wins, and an allow wins a tie with an equally long disallow. If no rule matches, the URL is allowed.

| Rule | What happens | Source |
| --- | --- | --- |
| Which group | The group naming the crawler, case-insensitively; groups with the same name merge; else the * group | RFC 9309 §2.2.1 |
| Which rule | The most specific match, the one with the most characters | RFC 9309 §2.2.2 |
| A tie | Allow wins over an equally long disallow | RFC 9309 §2.2.2 |
| * and $ | * matches any characters; $ ends the pattern | RFC 9309 §2.2.3 |
| /robots.txt | Always allowed, whatever the rules say | RFC 9309 §2.2.2 |
| File returns 4xx | Crawlers may access everything (Google: except 429) | RFC 9309 §2.3.1.3, Google |
| File returns 5xx | Crawlers must assume everything is disallowed | RFC 9309 §2.3.1.4, Google |
| File size | Crawlers must read at least 500 KiB; Google stops there | RFC 9309 §2.5, Google |
| Fields | Google reads user-agent, allow, disallow and sitemap; it ignores crawl-delay | Google |

Google also caches robots.txt for up to 24 hours, so a change can take a day to apply.

Sources: [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (read 2026-11-11), [Google Search Central: How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) (read 2026-11-11)

## How it works

Type a page and fetch its site's robots.txt, or paste a file of your own; pick a crawler and the test runs as you type. Our server fetches the file and doesn't keep it; the test itself runs in your browser.

It follows RFC 9309, the standard Google co-wrote: the crawler's own group if one names it, or the * group; within it, the longest matching rule, with allow winning a tie. The result names the line and the rule that decided it, and says whether the crawler used its own group or the * group.

It flags what crawlers ignore: rules before any User-agent line, fields Google doesn't read, such as crawl-delay and noindex, sitemap lines without a full URL, and files over the 500 KiB Google reads.

The generator builds a file from a few choices, with presets that block each AI company's training or search crawler by its documented token. What robots.txt can't do is keep a page out of search: Google says a blocked URL can still be indexed if other pages link to it.

## Get robots.txt right the first time. Four things that trip sites up.

A short file with outsized effects. Here's what the standard and Google's documentation say.

### It manages crawling, not indexing

Google says robots.txt is mainly for managing crawler traffic, and is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed if it's linked from elsewhere, shown without a description. To keep a page out, use noindex or put it behind a password.

Sources: [Google Search Central: Robots.txt introduction and guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro) (read 2026-11-11)

### A named crawler ignores the * group

A crawler follows only the group that names it, if there is one; the * group applies only when none does. So a group for GPTBot with one Disallow line drops every rule you wrote for *. Repeat the rules that should still apply.

Sources: [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (read 2026-11-11)

### The longest rule wins, not the first

Order doesn't matter. The rule with the longest matching path decides, and an allow beats a disallow of the same length. Allow: /admin/help/ beats Disallow: /admin/ for /admin/help/faq.

Sources: [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (read 2026-11-11)

### Errors mean different things

A missing file, a 404, means crawlers may go anywhere. A server error means the opposite: under RFC 9309 crawlers must assume everything is disallowed, and Google stops crawling for the first 12 hours while it retries. Google makes one exception to the 4xx rule: a 429 doesn't count as a missing file.

Crawlers decide who can find you. Buyers who already have are worth a message. [Valley](https://www.joinvalley.co/?utm_source=tools_suite)

Sources: [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (read 2026-11-11), [Google Search Central: How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) (read 2026-11-11)

## Sources

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (read 2026-11-11)
- [Google Search Central: How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) (read 2026-11-11)
- [Google Search Central: Robots.txt introduction and guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro) (read 2026-11-11)
- [Search Console Help: robots.txt report](https://support.google.com/webmasters/answer/6062598) (read 2026-11-11)

## Questions people ask

### What is a robots.txt file?

A text file at the root of a site, /robots.txt, that tells crawlers which paths they may and may not fetch, group by group, under RFC 9309.

### How do I check my robots.txt?

Type your site here and fetch it. You'll see its groups, sitemaps and any syntax problems, and can test any URL against any crawler.

### How do I test if a URL is blocked by robots.txt?

Enter the URL, pick the crawler and fetch the file. The result says allowed or blocked and names the line that decided it.

### Does robots.txt keep a page out of Google?

No. Google says a disallowed URL can still be indexed if other pages link to it. Use noindex or password protection instead.

### Does Google support crawl-delay?

No. Google reads only user-agent, allow, disallow and sitemap, and ignores crawl-delay.

### What happens if robots.txt returns a 404?

Crawlers may access everything, under RFC 9309. Google treats every 4xx except 429 as if there were no file.

### What happens if robots.txt returns a server error?

Crawlers must assume the whole site is disallowed. Google stops crawling for the first 12 hours and keeps retrying.

### How do I block GPTBot with robots.txt?

Add a group with User-agent: GPTBot and Disallow: /. The generator here does it for each AI company's training crawler.

### What is the maximum size of a robots.txt file?

Google reads the first 500 KiB and ignores the rest. RFC 9309 requires crawlers to read at least that much.

### Is this robots.txt checker free?

Yes. It's free and needs no signup. We fetch the file you ask for and don't keep it.

## Related tools

- [AI crawler checker](https://tools.joinvalley.co/ai-crawler-checker): Shows which AI crawlers, GPTBot, ClaudeBot and others, can read a page, and why not.
- [UTM builder](https://tools.joinvalley.co/utm-builder): Builds UTM links one at a time or in bulk, and shows the GA4 channel each lands in.
- [Domain age checker](https://tools.joinvalley.co/domain-age-checker): Shows when a domain was registered, its age, expiry and registrar, read from the registry.
- [Readability checker](https://tools.joinvalley.co/readability-checker): Scores any text's Flesch Reading Ease and grade level, and marks the long sentences.
- [Link preview checker](https://tools.joinvalley.co/link-preview-checker): Shows how a link previews on LinkedIn, X and Slack, and checks its Open Graph tags.

## Next step

Want replies, not just clean copy? Valley finds buyers showing real buying signals and messages them across email and LinkedIn. [Valley](https://www.joinvalley.co/?utm_source=tools_suite)

Shelf: [09 Inbound, content and search](https://tools.joinvalley.co/inbound-and-content)

---

Tools by Valley, made by [Valley](https://www.joinvalley.co/?utm_source=tools_suite). HTML version: https://tools.joinvalley.co/robots-txt-checker
