AI Crawl Scan — see which AI crawlers a site lets in
A free tool that checks any domain's robots.txt and shows which AI crawlers it allows or blocks — GPTBot, ClaudeBot, PerplexityBot and more.
AI Crawl Scan is a free tool from SERP Forge that checks any domain's robots.txt and shows, in one click, which AI crawlers it allows, blocks, or leaves to the default rule. Enter a domain (or paste robots.txt) and it reports the state of each crawler. No sign-up.
Why I built it
As AI answers become a real discovery channel, robots.txt is the new gate — and most teams never set it on purpose. A security setting or a one-click "block AI bots" toggle can quietly cut a site out of ChatGPT, Claude, or Perplexity answers. This tool makes that invisible decision visible, so teams can choose instead of letting a default decide.
What it checks
- OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic — ClaudeBot, anthropic-ai
- Perplexity, Google-Extended, and the common training scrapers (CCBot, Bytespider, and more)
It's the same check behind my study of 95 leading SaaS sites.
Notes from the desk
New writing, experiments, and the occasional story — straight to your inbox. Two to four emails a month.
Suraj Shrivastava
Founder of SERP Forge, the organic growth engine for B2B SaaS. I write here about building the company, experiments with AI and search, and the occasional life detour.