Free AI Crawler Checker
Is your site visible to ChatGPT, Claude, and Perplexity? Paste your URL and see the live robots.txt verdict for 12 AI crawlers, plus your llms.txt status and copyable fixes.
How it works
From unknown policy to a deliberate one in three steps.
Paste your site URL
We fetch your live robots.txt and llms.txt, with the same rules real crawlers follow.
See every bot's verdict
12 AI crawlers across OpenAI, Anthropic, Perplexity, Google, Apple, and more, grouped by what each one is for.
Fix it in one paste
Copy an AI-friendly robots.txt block and a starter llms.txt, adjusted to whatever policy you choose.
While you are auditing, check how your links look when shared with the social share preview and how your pages read in Google with the SERP preview tool.
Frequently asked questions
Is the AI crawler checker free?
Yes. You can run up to 20 checks a day with no account, and up to 60 a day when you sign in with Google. No credit card required.
Which AI bots does it check?
GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI); ClaudeBot and Claude-SearchBot (Anthropic); PerplexityBot and Perplexity-User; Google-Extended; Applebot-Extended; CCBot; meta-externalagent; and Bytespider.
Should I block AI crawlers or allow them?
It is a real tradeoff. Blocking training bots keeps your content out of future models; blocking AI-search bots makes you invisible in ChatGPT and Perplexity answers. Many sites allow search and block training; the tool shows exactly where you stand either way.
What is llms.txt?
An emerging standard: a Markdown file at /llms.txt that points AI systems at your most important pages. Adoption is around 10% of domains; it is a low-effort, low-risk signal, and the tool gives you a starter file.
Does blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot is the training crawler; OAI-SearchBot powers ChatGPT search citations. The tool evaluates them separately so you can allow one without the other.