Robots.txt Check
Understand robots.txt discovery, crawler directives, sitemap references, and accidental blocking risks.
Check your websiteWhy this check matters
Robots.txt controls crawl access, not indexing guarantees. A broad disallow can hide important content from crawlers and prevent them from seeing canonical or noindex directives.
Passing evidence
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlCommon failing evidence
User-agent: *
Disallow: /Possible false positives
- Staging sites may intentionally block all crawling.
- Some specialized bots can be managed with their own user-agent group.
How to fix it safely
- Serve robots.txt from the site root.
- Remove broad rules that cover public pages.
- List canonical sitemap URLs and keep private routes explicitly disallowed.
How to verify the fix
- Open /robots.txt without authentication.
- Test representative public and private paths against each relevant user-agent group.
Related SEO / AEO / GEO checks
Run the complete audit
Check this signal together with the related category evidence on the current public page.
Open the seo / aeo / geo audit tool
