Skip to content

Tools is made by Valley.

See how ↗

Tools is made by Valley.

See how ↗

Tools is made by Valley.

See how ↗

Free tool · 09 / GROW

Robots.txt checker and generator.Test any URL against any crawler, or build a file, no signup.

Tests any URL against any crawler's robots.txt rules, or builds a new file.

We look this up and don't keep it.

Test a URL

Blocked

/admin/settings Blocked by line 3, Disallow: /admin/, the most specific match in the * group.

Example file for a made-up site. Fetch a real one above, or paste your own.

Groups

  • Line 2: *, 3 rules
  • Line 7: GPTBot, CCBot, 1 rule

Sitemaps

  • https://yourcompany.example/sitemap.xml

Problems

  • No syntax problems found.

Build a robots.txt

Everyone else (*)
AI crawlers
User-agent: *
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://yourcompany.example/sitemap.xml

A crawler named in its own group follows only that group, not the * group. The presets name each company's own token.

See which AI crawlers can actually reach a page →

Valley finds the people already paying attention, like those who viewed your profile or company page, and writes to them first. See how Valley does it ↗

Built by Valley ↗

NEXT STEP

Fed up with tuning crawlers instead of talking to buyers.Valley messages the people already circling you.

Book a demo ↗

$500 a month for your first seat, $200 for each seat after that.

The rules

How does robots.txt matching work?RFC 9309's rules, and where Google adds its own.

A crawler follows the group that names its user agent, or the * group if none does. Within that group, the rule with the longest matching path wins, and an allow wins a tie with an equally long disallow. If no rule matches, the URL is allowed.

How crawlers read robots.txt under RFC 9309, with Google's documented differences
RuleWhat happensSource
Which groupThe group naming the crawler, case-insensitively; groups with the same name merge; else the * groupRFC 9309 §2.2.1
Which ruleThe most specific match, the one with the most charactersRFC 9309 §2.2.2
A tieAllow wins over an equally long disallowRFC 9309 §2.2.2
* and $* matches any characters; $ ends the patternRFC 9309 §2.2.3
/robots.txtAlways allowed, whatever the rules sayRFC 9309 §2.2.2
File returns 4xxCrawlers may access everything (Google: except 429)RFC 9309 §2.3.1.3, Google
File returns 5xxCrawlers must assume everything is disallowedRFC 9309 §2.3.1.4, Google
File sizeCrawlers must read at least 500 KiB; Google stops thereRFC 9309 §2.5, Google
FieldsGoogle reads user-agent, allow, disallow and sitemap; it ignores crawl-delayGoogle

Google also caches robots.txt for up to 24 hours, so a change can take a day to apply.

From RFC 9309: Robots Exclusion Protocol ↗ · Google Search Central: How Google interprets the robots.txt specification ↗, read 11 Nov 2026

How it works

How this tool works.And what it can't see.

Type a page and fetch its site's robots.txt, or paste a file of your own; pick a crawler and the test runs as you type. Our server fetches the file and doesn't keep it; the test itself runs in your browser.

It follows RFC 9309, the standard Google co-wrote: the crawler's own group if one names it, or the * group; within it, the longest matching rule, with allow winning a tie. The result names the line and the rule that decided it, and says whether the crawler used its own group or the * group.

It flags what crawlers ignore: rules before any User-agent line, fields Google doesn't read, such as crawl-delay and noindex, sitemap lines without a full URL, and files over the 500 KiB Google reads.

The generator builds a file from a few choices, with presets that block each AI company's training or search crawler by its documented token. What robots.txt can't do is keep a page out of search: Google says a blocked URL can still be indexed if other pages link to it.

Built by Valley.

Sources

Robots.txt

Get robots.txt right the first time.Four things that trip sites up.

A short file with outsized effects. Here's what the standard and Google's documentation say.

  1. 01

    It manages crawling, not indexing

    Google says robots.txt is mainly for managing crawler traffic, and is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed if it's linked from elsewhere, shown without a description. To keep a page out, use noindex or put it behind a password.

    From Google Search Central: Robots.txt introduction and guide ↗, read 11 Nov 2026

  2. 02

    A named crawler ignores the * group

    A crawler follows only the group that names it, if there is one; the * group applies only when none does. So a group for GPTBot with one Disallow line drops every rule you wrote for *. Repeat the rules that should still apply.

    From RFC 9309: Robots Exclusion Protocol ↗, read 11 Nov 2026

  3. 03

    The longest rule wins, not the first

    Order doesn't matter. The rule with the longest matching path decides, and an allow beats a disallow of the same length. Allow: /admin/help/ beats Disallow: /admin/ for /admin/help/faq.

    From RFC 9309: Robots Exclusion Protocol ↗, read 11 Nov 2026

  4. 04

    Errors mean different things

    A missing file, a 404, means crawlers may go anywhere. A server error means the opposite: under RFC 9309 crawlers must assume everything is disallowed, and Google stops crawling for the first 12 hours while it retries. Google makes one exception to the 4xx rule: a 429 doesn't count as a missing file.

    Crawlers decide who can find you. Buyers who already have are worth a message. See how Valley does it ↗

    From RFC 9309: Robots Exclusion Protocol ↗ · Google Search Central: How Google interprets the robots.txt specification ↗, read 11 Nov 2026

Questions

Questions people ask.

What is a robots.txt file?
A text file at the root of a site, /robots.txt, that tells crawlers which paths they may and may not fetch, group by group, under RFC 9309.
How do I check my robots.txt?
Type your site here and fetch it. You'll see its groups, sitemaps and any syntax problems, and can test any URL against any crawler.
How do I test if a URL is blocked by robots.txt?
Enter the URL, pick the crawler and fetch the file. The result says allowed or blocked and names the line that decided it.
Does robots.txt keep a page out of Google?
No. Google says a disallowed URL can still be indexed if other pages link to it. Use noindex or password protection instead.
Does Google support crawl-delay?
No. Google reads only user-agent, allow, disallow and sitemap, and ignores crawl-delay.
What happens if robots.txt returns a 404?
Crawlers may access everything, under RFC 9309. Google treats every 4xx except 429 as if there were no file.
What happens if robots.txt returns a server error?
Crawlers must assume the whole site is disallowed. Google stops crawling for the first 12 hours and keeps retrying.
How do I block GPTBot with robots.txt?
Add a group with User-agent: GPTBot and Disallow: /. The generator here does it for each AI company's training crawler.
What is the maximum size of a robots.txt file?
Google reads the first 500 KiB and ignores the rest. RFC 9309 requires crawlers to read at least that much.
Is this robots.txt checker free?
Yes. It's free and needs no signup. We fetch the file you ask for and don't keep it.