--- title: "robots.txt audit reference guide" date: 2026-10-16 author: "Matt Cromwell" --- # robots.txt audit reference guide I wrote the [robots.txt audit skill](https://github.com/Roots-and-Fruit/skills/tree/master/marketing/skills/robots-txt-audit) after one too many templates that blocked Google while trying to block AI training. The skill scores your live file against a crawl policy you pick. This page is the plain-language companion: what each check means, and why the skill cares. If you followed a **Why this matters** link, the `#` anchor in the URL should match a section below. ## What robots.txt actually does `robots.txt` is a text file at your domain root. Compliant crawlers read it before they request paths ([RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309)). robots.txt doesrobots.txt does notAllow or block **crawling** per user-agentGuarantee **indexing** (use `noindex` on cart, checkout, and other thin URLs)Separate **training** bots from **search and answer** botsSecure private contentPoint crawlers to XML sitemaps with `Sitemap:`Replace `llms.txt` or fix a bad sitemap by itselfKeep admin paths out of routine crawlsRemove content already in a corpus**Crawl is not the same as index.** A bot must be allowed to fetch a URL before it can see a `noindex` tag. Blocking `/cart/` in robots.txt can hide the very signal you need ([Google: block indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing)). If `https://example.com/robots.txt` fails, try `https://www.example.com/robots.txt`. ## Crawl policies Tell the skill which posture you want. These names match the skill input `crawl_policy`. PolicyWhat you are aiming for`max_discovery`Allow Google, Bing, and the main AI **answer** crawlers. Keep admin paths tidy. Declare a valid sitemap. Training blocks are optional, not required.`block_training_allow_answers`Everything in `max_discovery`, plus explicit blocks on common **training** tokens (`GPTBot`, `Google-Extended`, `CCBot`, `ClaudeBot`).`restrictive`Block AI answer crawlers and training bots; keep traditional search bots. You trade GEO and citation reach for a tighter fence.`audit_only`Report what is live. No compliance gate against a preset.## Training versus search and answer bots Optimize **per user-agent**, not one blunt “block AI” rule. You care aboutTypical tokensSkill expectationGoogle Search`Googlebot`Allow public pagesGoogle training reuse`Google-Extended`Block only if you opted out of trainingBing`bingbot`Allow public pagesChatGPT search / citations`OAI-SearchBot`Allow when you want discoveryOpenAI training`GPTBot`Block only when policy calls for itClaude search`Claude-SearchBot`Allow when you want discoveryAnthropic training`ClaudeBot`Block only when policy calls for itPerplexity discovery`PerplexityBot`Allow when you want discovery**Pairing mistakes hurt.** Do not block `Googlebot` when you only meant to limit generative training; use `Google-Extended`. The same pattern applies to `OAI-SearchBot` versus `GPTBot`, and `Claude-SearchBot` versus `ClaudeBot`. The skill flags reversed pairings as gaps. User-agent groups **replace** the `*` group for that token; they do not merge. If `OAI-SearchBot` has its own group, a `Disallow: /` on `*` does not govern it. ## Path hygiene For `User-agent: *`, restrict low-value paths such as `/wp-admin/` or `/admin/` when your stack uses them. That is crawl hygiene, not a security control. For cart and checkout, prefer **`noindex`** (meta or header) on the live URL. The skill warns when those paths are crawlable without `noindex`. It does **not** require a robots `Disallow` there when `noindex` is already doing the job. ## Sitemap declaration Search crawlers use `Sitemap:` lines in robots.txt to find your XML sitemaps. - Declare at least one `Sitemap:` with a full `https://` URL on your host (not a relative path). - Prefer `https://example.com/sitemap.xml` or a sitemap index on the same site. - Fix typos such as `Schemamap:` (seen in some SEO plugins). Crawlers ignore unknown directive names. - Do not list `/llms.txt` or `/llms-full.txt` as a sitemap; those are not XML sitemap endpoints. ## Syntax validation Broken syntax wastes the file. The skill runs a light lint (similar in spirit to Search Console robots reports): unknown directives, malformed lines, and bad sitemap spellings. Treat warnings as “fix before you forget,” not as vanity polish. ## Anthropic and OpenAI pairing I see the same pairing mistakes on almost every site I audit, so here is the cheat sheet without rereading the full table. - Blocking `ClaudeBot` while allowing `Claude-SearchBot` is valid when you want answers but not training. - Blocking `Claude-SearchBot` while allowing `ClaudeBot` is almost never what you want. - The OpenAI pair (`OAI-SearchBot` / `GPTBot`) follows the same logic. ## Content-Signal alignment [Content-Signals](https://contentsignals.org/) declare intent (`search`, `ai-input`, `ai-train`) alongside crawl rules. They do not replace `Allow` / `Disallow`. On Cloudflare managed robots.txt, the edge block often sets `search=yes` and `ai-train=no` on `*`. Your origin append can align answer bots with `search_and_ai_input` when you use `max_discovery` or `block_training_allow_answers`. When the declaration and the real allows disagree, the skill may flag an optional gap. That is an alignment nudge, not a failed audit. ## Cloudflare detection When Cloudflare **managed robots.txt** is on, the response you fetch from the public URL is merged: 1\. **Managed block** between `# BEGIN Cloudflare Managed content` and `# END Cloudflare Managed Content` (training bots, baseline signals). 2\. **Your origin file** appended after that marker (`Sitemap:`, path rules, answer-bot allows). The skill assesses **cloudflare**, **origin**, and **effective** layers separately. Edit the origin file below the END marker; do not duplicate training blocks Cloudflare already maintains. Keep the dashboard toggle on unless you have a deliberate reason to turn it off. ## File accessibility and CDN reality Always audit the URL visitors and bots hit, not only the file on your server. CDNs, proxies, and managed robots can serve different text than SFTP shows. After deploy, fetch `https://{domain}/robots.txt` from the public internet, spot-check cornerstone paths, and keep a rollback copy. Google Search Console’s robots report is still the authoritative sanity check when you need proof. ## llms.txt companion [`llms.txt`](https://llmstxt.org/) is a curated Markdown guide for agents. It is not an Allow/Disallow file and Google Search does not use it for rankings. The skill’s companion check is narrow: - If discovery crawlers may fetch `/` but cannot fetch `/llms.txt`, that is an optional gap. - Putting `llms.txt` in a `Sitemap:` line is misuse. - A missing `llms.txt` is **not** a failure. Do not publish an empty file just to clear a checkbox. Robots.txt only controls whether compliant bots may **request** `llms.txt`. It does not write or grade the file. ## Enforcement beyond robots.txt Some aggressive crawlers ignore robots.txt. Optional skill gaps (for example certain scrapers) may point you toward WAF rules, rate limits, or provider-specific blocks. Treat those as enforcement layers, not as replacements for a coherent robots file. ## After you publish Run the skill again in **recheck** mode so the comparison uses the live URL. Fix syntax and sitemap issues first; then revisit policy presets if the effective file still disagrees with your intent. **Get the skill:** [robots-txt-audit on GitHub](https://github.com/Roots-and-Fruit/skills/tree/master/marketing/skills/robots-txt-audit). **Want help on a live site?** [Roots & Fruit](https://rootsandfruit.com) does crawl policy and GEO work for WordPress and SaaS teams.