Crawler Access Check
Verify that the audited page and its controlling directives do not unintentionally block search and AI crawlers.
Check your websiteWhy this check matters
A page cannot earn organic visibility when important crawlers cannot retrieve it. FreeScan combines page access, robots directives, and crawler-specific evidence instead of relying on one signal.
Passing evidence
HTTP 200
User-agent: *
Allow: /Common failing evidence
HTTP 403
User-agent: *
Disallow: /Possible false positives
- Private staging and authenticated pages may be intentionally blocked.
- A crawler-specific restriction can be a deliberate licensing or traffic-control decision.
How to fix it safely
- Confirm the page returns successful HTML without a browser session.
- Review robots.txt rules for broad or inherited disallows.
- Check firewall and bot-management rules for the affected crawler identities.
How to verify the fix
- Request the page with representative crawler user agents.
- Re-run the audit and confirm page access is no longer blocked or inconclusive.
Related SEO / AEO / GEO checks
Run the complete audit
Check this signal together with the related category evidence on the current public page.
Open the seo / aeo / geo audit tool
