How to Check If GPTBot and ClaudeBot Can Actually Crawl Your Site

If ChatGPT or Claude has never accurately described what your company does, the problem might not be your content. It might be that the crawlers behind those tools never reached your site in the first place.

Most GEO advice focuses on what to write. Before any of that matters, you need to confirm the AI crawlers can actually get in. This is a quick technical check most teams skip, and it’s worth doing before you invest in content structure or citation-focused writing.

The crawlers that matter right now

Each major AI company runs its own crawler, and they behave differently. Here’s the current landscape.

Crawler	Operated by	Purpose
GPTBot	OpenAI	Collects content for model training
ChatGPT-User	OpenAI	Fetches pages in real time during a ChatGPT conversation
ClaudeBot	Anthropic	Collects content for model training
Claude-User	Anthropic	Fetches pages during live Claude conversations and tool use
PerplexityBot	Perplexity	Crawls and indexes for search-style retrieval
Google-Extended	Google	Controls whether Google can use your content for Gemini and AI Overviews, separate from standard Googlebot

Two of these matter more than people assume: the “-User” agents (ChatGPT-User, Claude-User) fire in real time when someone asks a question and the model decides to fetch a live page. If those are blocked, you can lose out on a citation in the exact moment someone is asking about your category, even if your training-data crawler access is fine.

Step 1: Check what robots.txt allows

Start with the basics. Pull up yoursite.com/robots.txt and look for any Disallow rules tied to User-agent: GPTBot, User-agent: ClaudeBot, User-agent: PerplexityBot, or User-agent: Google-Extended.

A common mistake is a blanket rule left over from an old security review, something like:

User-agent: GPTBot
Disallow: /

This fully blocks OpenAI’s crawler from every page on the site. It’s usually not intentional. It gets added once, during a period when a team was cautious about AI scraping, and nobody revisits it once GEO becomes a priority.

If you want AI crawlers in, the fix is straightforward: remove the disallow rule, or scope it narrowly to pages you genuinely don’t want indexed (internal tools, staging environments, account pages).

Step 2: Check what robots.txt doesn’t tell you

Robots.txt is a request, not an enforcement mechanism. A well-behaved crawler respects it. But robots.txt being clean doesn’t guarantee the crawler is actually reaching your pages, because plenty of infrastructure sits between the crawler and your content that has nothing to do with robots.txt.

The most common blocker teams miss is CDN-level bot protection. Services like Cloudflare, Akamai, and Fastly ship default bot-management rules that can silently block AI crawlers based on user-agent string or behavioral fingerprinting, regardless of what your robots.txt says. If your site sits behind one of these, check the bot-management dashboard specifically, not just your robots.txt file.

Step 3: Confirm with server logs, not assumptions

The only way to know for certain that a crawler visited is to look at your server or CDN access logs and search for the user-agent string directly.

1. Pull your access logs for the last 30 days (most hosting providers and CDNs keep this available in a dashboard or downloadable export).
2. Search for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended in the user-agent field.
3. Check the response codes returned to those requests. A 200 means the crawler got the page. A 403 or 429 means something blocked or rate-limited it.
4. Note which pages got hit most. If crawlers are only reaching your homepage and never your deeper content, that’s a discoverability problem separate from an access problem.

If you don’t see any hits at all from a given crawler, that’s a signal worth investigating further, whether it’s a robots.txt rule, a CDN block, or simply that the crawler hasn’t prioritized your domain yet.

Step 4: Test individual pages directly

For a faster spot check without pulling full logs, use a curl request with the crawler’s user-agent string to see how your server responds:

curl -A "GPTBot" -I https://yoursite.com/your-page

Look at the status code in the response. A 200 is a good sign. A 403 tells you something on your server or CDN is actively blocking that user-agent, and it’s worth tracing back to the specific rule causing it.

Why this is worth checking even if you’re not chasing new citations

A blocked crawler doesn’t just cost you a chance at a new citation. It affects how AI models describe your brand at all. Models retrain periodically, and retrieval-based tools recrawl content to keep answers current. If your site has been inaccessible to GPTBot or ClaudeBot for months, any pricing changes, product updates, or rebrands you’ve made may never make it into what AI tools say about you. Stale or outdated brand information often traces back to exactly this kind of access issue, not a content gap. For a deeper look at how models decide what to trust once they can reach your site, see how llms.txt fits into that picture.

What to do if you find a block

If you confirm a crawler is blocked and you want it in:

• Remove or narrow the disallow rule in robots.txt for that user-agent.
• Check CDN bot-management settings and add explicit allow rules for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
• Re-run the curl test from Step 4 to confirm the fix worked before moving on.
• Re-check server logs again in two to four weeks to confirm the crawler actually returned.

If you decide you don’t want a specific crawler accessing your content, that’s a legitimate call too. Just make it deliberately, not by default, since the tradeoff is real: no access generally means no citation in that tool’s answers.

Getting this right is infrastructure work, not content work, but it’s the foundation everything else in a GEO program sits on. If your content strategy is solid and your citations still aren’t showing up, this is the first place to look.