I wrote the robots.txt audit skill after one too many templates that blocked Google while trying to block AI training. The skill scores your live file against a crawl policy you pick. This page is the plain-language companion: what each check means, and why the skill cares. If you followed a Why this matters link, the # anchor in the URL should match a section below.
What robots.txt actually does
robots.txt is a text file at your domain root. Compliant crawlers read it before they request paths (RFC 9309).
| robots.txt does | robots.txt does not |
|---|---|
| Allow or block crawling per user-agent | Guarantee indexing (use noindex on cart, checkout, and other thin URLs) |
| Separate training bots from search and answer bots | Secure private content |
Point crawlers to XML sitemaps with Sitemap: | Replace llms.txt or fix a bad sitemap by itself |
| Keep admin paths out of routine crawls | Remove content already in a corpus |
Crawl is not the same as index. A bot must be allowed to fetch a URL before it can see a noindex tag. Blocking /cart/ in robots.txt can hide the very signal you need (Google: block indexing with noindex).
If https://example.com/robots.txt fails, try https://www.example.com/robots.txt.
Crawl policies
Tell the skill which posture you want. These names match the skill input crawl_policy.
| Policy | What you are aiming for |
|---|---|
max_discovery | Allow Google, Bing, and the main AI answer crawlers. Keep admin paths tidy. Declare a valid sitemap. Training blocks are optional, not required. |
block_training_allow_answers | Everything in max_discovery, plus explicit blocks on common training tokens (GPTBot, Google-Extended, CCBot, ClaudeBot). |
restrictive | Block AI answer crawlers and training bots; keep traditional search bots. You trade GEO and citation reach for a tighter fence. |
audit_only | Report what is live. No compliance gate against a preset. |
Training versus search and answer bots
Optimize per user-agent, not one blunt “block AI” rule.
| You care about | Typical tokens | Skill expectation |
|---|---|---|
| Google Search | Googlebot | Allow public pages |
| Google training reuse | Google-Extended | Block only if you opted out of training |
| Bing | bingbot | Allow public pages |
| ChatGPT search / citations | OAI-SearchBot | Allow when you want discovery |
| OpenAI training | GPTBot | Block only when policy calls for it |
| Claude search | Claude-SearchBot | Allow when you want discovery |
| Anthropic training | ClaudeBot | Block only when policy calls for it |
| Perplexity discovery | PerplexityBot | Allow when you want discovery |
Pairing mistakes hurt. Do not block Googlebot when you only meant to limit generative training; use Google-Extended. The same pattern applies to OAI-SearchBot versus GPTBot, and Claude-SearchBot versus ClaudeBot. The skill flags reversed pairings as gaps.
User-agent groups replace the * group for that token; they do not merge. If OAI-SearchBot has its own group, a Disallow: / on * does not govern it.
Path hygiene
For User-agent: *, restrict low-value paths such as /wp-admin/ or /admin/ when your stack uses them. That is crawl hygiene, not a security control.
For cart and checkout, prefer noindex (meta or header) on the live URL. The skill warns when those paths are crawlable without noindex. It does not require a robots Disallow there when noindex is already doing the job.
Sitemap declaration
Search crawlers use Sitemap: lines in robots.txt to find your XML sitemaps.
- Declare at least one
Sitemap:with a fullhttps://URL on your host (not a relative path). - Prefer
https://example.com/sitemap.xmlor a sitemap index on the same site. - Fix typos such as
Schemamap:(seen in some SEO plugins). Crawlers ignore unknown directive names. - Do not list
/llms.txtor/llms-full.txtas a sitemap; those are not XML sitemap endpoints.
Syntax validation
Broken syntax wastes the file. The skill runs a light lint (similar in spirit to Search Console robots reports): unknown directives, malformed lines, and bad sitemap spellings. Treat warnings as “fix before you forget,” not as vanity polish.
Anthropic and OpenAI pairing
I see the same pairing mistakes on almost every site I audit, so here is the cheat sheet without rereading the full table.
- Blocking
ClaudeBotwhile allowingClaude-SearchBotis valid when you want answers but not training. - Blocking
Claude-SearchBotwhile allowingClaudeBotis almost never what you want. - The OpenAI pair (
OAI-SearchBot/GPTBot) follows the same logic.
Content-Signal alignment
Content-Signals declare intent (search, ai-input, ai-train) alongside crawl rules. They do not replace Allow / Disallow.
On Cloudflare managed robots.txt, the edge block often sets search=yes and ai-train=no on *. Your origin append can align answer bots with search_and_ai_input when you use max_discovery or block_training_allow_answers. When the declaration and the real allows disagree, the skill may flag an optional gap. That is an alignment nudge, not a failed audit.
Cloudflare detection
When Cloudflare managed robots.txt is on, the response you fetch from the public URL is merged:
1. Managed block between # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content (training bots, baseline signals).
2. Your origin file appended after that marker (Sitemap:, path rules, answer-bot allows).
The skill assesses cloudflare, origin, and effective layers separately. Edit the origin file below the END marker; do not duplicate training blocks Cloudflare already maintains. Keep the dashboard toggle on unless you have a deliberate reason to turn it off.
File accessibility and CDN reality
Always audit the URL visitors and bots hit, not only the file on your server. CDNs, proxies, and managed robots can serve different text than SFTP shows.
After deploy, fetch https://{domain}/robots.txt from the public internet, spot-check cornerstone paths, and keep a rollback copy. Google Search Console’s robots report is still the authoritative sanity check when you need proof.
llms.txt companion
llms.txt is a curated Markdown guide for agents. It is not an Allow/Disallow file and Google Search does not use it for rankings.
The skill’s companion check is narrow:
- If discovery crawlers may fetch
/but cannot fetch/llms.txt, that is an optional gap. - Putting
llms.txtin aSitemap:line is misuse. - A missing
llms.txtis not a failure. Do not publish an empty file just to clear a checkbox.
Robots.txt only controls whether compliant bots may request llms.txt. It does not write or grade the file.
Enforcement beyond robots.txt
Some aggressive crawlers ignore robots.txt. Optional skill gaps (for example certain scrapers) may point you toward WAF rules, rate limits, or provider-specific blocks. Treat those as enforcement layers, not as replacements for a coherent robots file.
After you publish
Run the skill again in recheck mode so the comparison uses the live URL. Fix syntax and sitemap issues first; then revisit policy presets if the effective file still disagrees with your intent.
Get the skill: robots-txt-audit on GitHub. Want help on a live site? Roots & Fruit does crawl policy and GEO work for WordPress and SaaS teams.