AI crawlers are the automated bots that read your website so AI engines can understand it and cite it in their answers. If you block them, you can’t be named — it’s that direct.
Crawler access is the one genuinely technical gate in all of this. Most of getting named by AI is about clarity and trust, not code. But this one is a simple yes/no: can the bots read your site or not?
Not sure whether you’re letting the AI crawlers in? Run a free AI Readiness Glimpse — it checks crawler access as part of your site’s signals.
What is an AI crawler?
A crawler is a program that visits web pages and reads their content. Search engines have used crawlers for decades. AI companies now run their own so their models can find and understand current information — including who the best local business is for a given job.
When you allow these crawlers, you’re giving the AI a clean copy of your facts to work from. When you block them, the AI is working with nothing, or with stale second-hand information.
The AI crawlers worth knowing
You don’t need to memorize these, but here are the main ones and who they belong to:
- GPTBot — OpenAI’s crawler, which feeds ChatGPT.
- ClaudeBot — Anthropic’s crawler, which feeds Claude.
- PerplexityBot — Perplexity’s crawler, which feeds its answer engine.
- Google-Extended — Google’s setting that controls whether your content can be used for its AI features.
There are others, but these are the ones that matter most for a business that wants to show up when buyers ask AI for a recommendation.
How to check — and fix — crawler access
Your site has a small file called robots.txt that tells crawlers what they’re allowed to read. You can see yours by visiting yourdomain.com/robots.txt in a browser.
- If it says something like
Disallow: /under one of those bot names, that bot is being turned away. - To let AI in, that file should allow the crawlers above rather than block them.
This often gets set the wrong way by accident — a privacy plugin, a site template, or a setting someone flipped years ago. It’s a two-minute fix once you know it’s there, which is exactly why we check it for you.
The honest version
Letting the crawlers in doesn’t guarantee AI will name you — it just makes it possible. Think of it as unlocking the door. Everything else on our site is about giving AI a reason to walk through it and pick you: clear facts, direct answers, and real proof. But if the door’s locked, none of the rest matters.
Keep reading: Why isn’t my business showing up in AI answers? · What is AI visibility? · Does llms.txt work?