Free tool · 09 / GROW
Robots.txt checker and generator.Test any URL against any crawler, or build a file, no signup.
Tests any URL against any crawler's robots.txt rules, or builds a new file.
We look this up and don't keep it.
Test a URL
Blocked
/admin/settings Blocked by line 3, Disallow: /admin/, the most specific match in the * group.
Example file for a made-up site. Fetch a real one above, or paste your own.
Groups
- Line 2: *, 3 rules
- Line 7: GPTBot, CCBot, 1 rule
Sitemaps
- https://yourcompany.example/sitemap.xml
Problems
- No syntax problems found.
Build a robots.txt
User-agent: * Disallow: /admin/ User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: / Sitemap: https://yourcompany.example/sitemap.xml
A crawler named in its own group follows only that group, not the * group. The presets name each company's own token.
See which AI crawlers can actually reach a page →
Valley finds the people already paying attention, like those who viewed your profile or company page, and writes to them first. See how Valley does it ↗
NEXT STEP
Fed up with tuning crawlers instead of talking to buyers.Valley messages the people already circling you.
Book a demo ↗$500 a month for your first seat, $200 for each seat after that.
The rules
How does robots.txt matching work?RFC 9309's rules, and where Google adds its own.
A crawler follows the group that names its user agent, or the * group if none does. Within that group, the rule with the longest matching path wins, and an allow wins a tie with an equally long disallow. If no rule matches, the URL is allowed.
| Rule | What happens | Source |
|---|---|---|
| Which group | The group naming the crawler, case-insensitively; groups with the same name merge; else the * group | RFC 9309 §2.2.1 |
| Which rule | The most specific match, the one with the most characters | RFC 9309 §2.2.2 |
| A tie | Allow wins over an equally long disallow | RFC 9309 §2.2.2 |
| * and $ | * matches any characters; $ ends the pattern | RFC 9309 §2.2.3 |
| /robots.txt | Always allowed, whatever the rules say | RFC 9309 §2.2.2 |
| File returns 4xx | Crawlers may access everything (Google: except 429) | RFC 9309 §2.3.1.3, Google |
| File returns 5xx | Crawlers must assume everything is disallowed | RFC 9309 §2.3.1.4, Google |
| File size | Crawlers must read at least 500 KiB; Google stops there | RFC 9309 §2.5, Google |
| Fields | Google reads user-agent, allow, disallow and sitemap; it ignores crawl-delay |
Google also caches robots.txt for up to 24 hours, so a change can take a day to apply.
From RFC 9309: Robots Exclusion Protocol ↗ · Google Search Central: How Google interprets the robots.txt specification ↗, read 11 Nov 2026
How it works
How this tool works.And what it can't see.
Type a page and fetch its site's robots.txt, or paste a file of your own; pick a crawler and the test runs as you type. Our server fetches the file and doesn't keep it; the test itself runs in your browser.
It follows RFC 9309, the standard Google co-wrote: the crawler's own group if one names it, or the * group; within it, the longest matching rule, with allow winning a tie. The result names the line and the rule that decided it, and says whether the crawler used its own group or the * group.
It flags what crawlers ignore: rules before any User-agent line, fields Google doesn't read, such as crawl-delay and noindex, sitemap lines without a full URL, and files over the 500 KiB Google reads.
The generator builds a file from a few choices, with presets that block each AI company's training or search crawler by its documented token. What robots.txt can't do is keep a page out of search: Google says a blocked URL can still be indexed if other pages link to it.
Built by Valley.
Sources
- RFC 9309: Robots Exclusion Protocol ↗read 11 Nov 2026
- Google Search Central: How Google interprets the robots.txt specification ↗read 11 Nov 2026
- Google Search Central: Robots.txt introduction and guide ↗read 11 Nov 2026
- Search Console Help: robots.txt report ↗read 11 Nov 2026
Robots.txt
Get robots.txt right the first time.Four things that trip sites up.
A short file with outsized effects. Here's what the standard and Google's documentation say.
- 01
It manages crawling, not indexing
Google says robots.txt is mainly for managing crawler traffic, and is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed if it's linked from elsewhere, shown without a description. To keep a page out, use noindex or put it behind a password.
From Google Search Central: Robots.txt introduction and guide ↗, read 11 Nov 2026
- 02
A named crawler ignores the * group
A crawler follows only the group that names it, if there is one; the * group applies only when none does. So a group for GPTBot with one Disallow line drops every rule you wrote for *. Repeat the rules that should still apply.
From RFC 9309: Robots Exclusion Protocol ↗, read 11 Nov 2026
- 03
The longest rule wins, not the first
Order doesn't matter. The rule with the longest matching path decides, and an allow beats a disallow of the same length. Allow: /admin/help/ beats Disallow: /admin/ for /admin/help/faq.
From RFC 9309: Robots Exclusion Protocol ↗, read 11 Nov 2026
- 04
Errors mean different things
A missing file, a 404, means crawlers may go anywhere. A server error means the opposite: under RFC 9309 crawlers must assume everything is disallowed, and Google stops crawling for the first 12 hours while it retries. Google makes one exception to the 4xx rule: a 429 doesn't count as a missing file.
Crawlers decide who can find you. Buyers who already have are worth a message. See how Valley does it ↗
From RFC 9309: Robots Exclusion Protocol ↗ · Google Search Central: How Google interprets the robots.txt specification ↗, read 11 Nov 2026
Questions
Questions people ask.
- What is a robots.txt file?
- A text file at the root of a site, /robots.txt, that tells crawlers which paths they may and may not fetch, group by group, under RFC 9309.
- How do I check my robots.txt?
- Type your site here and fetch it. You'll see its groups, sitemaps and any syntax problems, and can test any URL against any crawler.
- How do I test if a URL is blocked by robots.txt?
- Enter the URL, pick the crawler and fetch the file. The result says allowed or blocked and names the line that decided it.
- Does robots.txt keep a page out of Google?
- No. Google says a disallowed URL can still be indexed if other pages link to it. Use noindex or password protection instead.
- Does Google support crawl-delay?
- No. Google reads only user-agent, allow, disallow and sitemap, and ignores crawl-delay.
- What happens if robots.txt returns a 404?
- Crawlers may access everything, under RFC 9309. Google treats every 4xx except 429 as if there were no file.
- What happens if robots.txt returns a server error?
- Crawlers must assume the whole site is disallowed. Google stops crawling for the first 12 hours and keeps retrying.
- How do I block GPTBot with robots.txt?
- Add a group with User-agent: GPTBot and Disallow: /. The generator here does it for each AI company's training crawler.
- What is the maximum size of a robots.txt file?
- Google reads the first 500 KiB and ignores the rest. RFC 9309 requires crawlers to read at least that much.
- Is this robots.txt checker free?
- Yes. It's free and needs no signup. We fetch the file you ask for and don't keep it.
Keep going
Try one of these next.
- AI crawler checkerInbound, content and searchShows which AI crawlers, GPTBot, ClaudeBot and others, can read a page, and why not.Open the tool
- UTM builderGTM opsBuilds UTM links one at a time or in bulk, and shows the GA4 channel each lands in.Open the tool
- Domain age checkerEmail outreachShows when a domain was registered, its age, expiry and registrar, read from the registry.Open the tool
- Readability checkerMessagingScores any text's Flesch Reading Ease and grade level, and marks the long sentences.Open the tool
- Link preview checkerInbound, content and searchShows how a link previews on LinkedIn, X and Slack, and checks its Open Graph tags.Open the tool