Free AI Crawler Access Test
Every other tool reads your robots.txt and stops. This one actually knocks: it requests your page as GPTBot, ClaudeBot, PerplexityBot and nine others, compares each against a browser, and tells you whether the block is in robots.txt, your server, or the CDN in front of it.
How it works
Configuration is what a site claims. This is what it does.
Enter a real content page
Rules often differ by path, so test an article rather than the home page.
We knock as each crawler
One request per AI bot, plus one as a browser, so blocked means blocked relative to a real visitor.
See where the block lives
robots.txt, your server, or the CDN in front of it. Different causes need different fixes.
To see what your robots.txt says on its own, use the AI crawler checker. Once a crawler can reach the page, check it can quote it with the retrieval readiness analyzer.
Frequently asked questions
How is this different from a robots.txt checker?
A robots.txt checker reads the file and tells you what it permits. This sends an actual HTTP request carrying each crawler's user-agent and reports what your server returned. Those are frequently different answers, because firewall and CDN rules are evaluated before robots.txt is ever read.
Is the AI crawler access test free?
Yes. You can run it 20 times a day with no account, and 60 times a day when you sign in with Google. Each run makes about thirteen requests to your site, which is why the hourly cap is tighter than our other tools.
Why does it say a bot is blocked when my robots.txt allows it?
Usually one of three things: a security plugin or server rule matching on user-agent, a CDN bot-management setting, or a managed robots.txt block your CDN injected above your own file. The results name which of these it can see.
Can this be wrong?
Yes, in both directions, and the results page says so. We send the crawler's user-agent from our own servers, which makes us an unverified bot. Services like Cloudflare verify real crawlers by IP range, reverse DNS and cryptographic signature. So a pass here can coexist with the verified bot being blocked, and a challenge here can coexist with the real bot getting through. What it reliably finds is user-agent blocking and the presence of an AI-blocking edge.
Should I block AI crawlers?
It depends which one, and they are not interchangeable. GPTBot trains models and sends you nothing back. OAI-SearchBot powers ChatGPT search results and can send referral clicks, so blocking it removes you from those answers. Google-Extended is training-only and does not affect AI Overviews, which run on Googlebot. The results label every bot with its purpose so you can decide per bot rather than per vendor.
What should I do after fixing access?
Check that a crawler which can reach the page can also use it. The retrieval readiness analyzer looks at whether your content is structured so an answer engine can quote a clean passage from it.