Help Centre

The tool runs on Cloudflare Workers and reaches domains over the public internet. An "unreachable" result means the Worker could not fetch the domain's robots.txt or sitemaps. Common causes:

• The domain is itself behind Cloudflare. Cloudflare Workers cannot reach a host that is itself behind Cloudflare — this is a well-known platform limitation. See the Cloudflare limitation below.
• The domain does not have a robots.txt file (it returns a 4xx status).
• The domain is down, has DNS issues, or rejects connections from the Worker's IP range.

Check the error message in the tool output — it often identifies the specific problem.

A contradiction means the same URL appears in both your sitemap and a Disallow rule in your robots.txt. Your sitemap tells Google "index this page" while your robots.txt says "don't read it." Google will typically obey the robots.txt block and not crawl the page, but it might still index the URL from other signals (links, redirects, etc.) — leading to the "Indexed, though blocked by robots.txt" warning in Search Console.

Neither file is wrong on its own; the contradiction is only visible when someone reads both at the same time. The tool reports the conflict — you decide which file to fix.

Not necessarily. Some Disallow rules are intentional (e.g., blocking admin pages or staging environments), and including those in a sitemap can serve internal publishing workflows. The tool does not judge whether a page should be indexed.

For public-facing content, a good rule of thumb:

• If the page should be indexed: either remove the Disallow rule from robots.txt, or remove the URL from the sitemap — but not both. Decide which file is correct and bring the other into line.
• If the page should not be indexed: remove it from the sitemap and keep the Disallow rule. Optionally add a <meta name="robots" content="noindex"> tag to the page itself as a belt-and-braces measure.

This is a known platform limitation. Cloudflare Workers cannot connect to domains that are themselves proxied through Cloudflare. The tool reports this as unreachable even though your site is up and serving requests.

Options to work around this:

• Check your robots.txt and sitemap files manually in a browser — the tool's logic is documented so you can replicate it.
• Use a different diagnostic tool that runs outside the Cloudflare network.
• Temporarily set your domain's Cloudflare proxy status to "DNS only" (grey cloud) for a quick test, then re-enable proxying. Note: this exposes your origin IP briefly.

Robotsdiff uses prefix matching and simple glob matching for Disallow patterns, roughly following RFC 9309 conventions. It supports:

• * — matches any sequence of characters (zero or more)
• $ — matches end of path only when it appears at the end of a pattern

Complex patterns combining * with query parameters (*?parameter=value*) are approximated and may produce incomplete results. This is explicitly listed in the known limitations below.

A Pro tier is in development at $9/month. Planned features include bulk domain comparison (upload a list), CSV and JSON export, and priority processing. The Pro tier is not yet available for purchase — no payment processor is connected to this business.

See the Pricing page for current details and to leave your interest. In the meantime, the Free tier handles any single domain you paste, with no limits on how many domains you can check — they just have to be checked one at a time.

No. Robotsdiff is fully stateless. There is no database, no user accounts, and no persistent storage of any kind. Each check is a standalone request: your browser sends a domain name to the Cloudflare Worker, the Worker fetches and parses the files in memory, returns the results, and discards everything. Cloudflare edge logs may contain request metadata (timestamps, status codes, approximate region) for operational purposes under their own data retention policies. See the Privacy Policy for full details.