Content analyzer
What are AI crawlers (and which ones does Seeklyn check)?
Seeklyn checks for 14 AI crawler user-agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, and Meta-ExternalAgent. AI crawlers are the user-agent tokens that answer engines send when fetching content; blocking any of these in robots.txt silently opts you out of AI citations from that engine.
Seeklyn's crawl-gate audit shows which of the 14 AI crawlers are blocked in your robots.txt — run your live URL free at /dashboard/content.
What are AI crawlers?
AI crawlers are the web-fetching bots that answer engines deploy to build the indexes they use to retrieve and cite content. Unlike classic Googlebot (which builds the web search index), AI crawlers are purpose-built for training data collection, real-time retrieval, or both. Each engine uses 1 or more distinct user-agent strings, and publishers can allow or block them via the robots.txt standard — independently of blocking Googlebot.
Blocking an AI crawler in robots.txt causes the engine to skip your pages entirely when building its retrieval index. The effect is silent: the engine does not notify you, your analytics do not change, and your page may still rank in classic search. But from that engine's perspective, your content does not exist as a citation source. Because robots.txt is often managed by infrastructure or CMS plugins rather than content teams, accidental blocks are common.
Which 14 AI crawlers does Seeklyn check?
Seeklyn checks the following 14 user-agent tokens from the robots.txt file for the analyzed URL: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, and Meta-ExternalAgent. The 14 bots map to the major AI answer engines and training pipelines: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic/Claude (ClaudeBot, Claude-Web, anthropic-ai), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended), Apple (Applebot-Extended), Common Crawl (CCBot), ByteDance (Bytespider), Amazon (Amazonbot), and Meta (Meta-ExternalAgent).
| User-agent token | Engine / company | Primary use |
|---|---|---|
| GPTBot | OpenAI | Training data and retrieval for ChatGPT |
| OAI-SearchBot | OpenAI | Real-time web search in ChatGPT |
| ChatGPT-User | OpenAI | Live browsing during ChatGPT sessions |
| ClaudeBot | Anthropic | Web retrieval for Claude |
| Claude-Web | Anthropic | Alternate Claude web-fetch token |
| anthropic-ai | Anthropic | Anthropic infrastructure crawler |
| PerplexityBot | Perplexity | Indexing for Perplexity answers |
| Perplexity-User | Perplexity | Real-time fetch during Perplexity queries |
| Google-Extended | Gemini AI training (NOT AI Overviews) | |
| Applebot-Extended | Apple | Apple Intelligence / Siri knowledge |
| CCBot | Common Crawl | Open training corpus used by many LLMs |
| Bytespider | ByteDance | TikTok search and AI products |
| Amazonbot | Amazon | Alexa and Amazon AI services |
| Meta-ExternalAgent | Meta | Meta AI assistant and Llama training |
The Google-Extended misconception
Google-Extended is the most commonly misunderstood AI crawler token. Blocking it does not hide your page from Google AI Overviews. Google Search Central documentation is explicit: Google-Extended controls whether Google can use your content to train Gemini AI models — it has no effect on whether Googlebot indexes your page or whether your page appears in AI Overviews. AI Overviews use the normal Googlebot index.
Seeklyn surfaces this as an explicit 'don't bother' myth in the landing audit: if you block Google-Extended hoping to opt out of AI Overviews, you are only affecting Gemini training data collection. Your page will still appear in AI Overviews as long as Googlebot can crawl it and your content is relevant. To actually opt out of AI Overviews, you would need to block Googlebot — which would also deindex your page from all Google organic search.
Frequently asked questions
Does blocking CCBot protect me from LLM training?
CCBot is Common Crawl's bot — its corpus is used by many open-source and commercial LLMs as training data. Blocking it reduces the chance of your content appearing in future training sets that use Common Crawl. However, many LLMs have already ingested Common Crawl snapshots from before you blocked it, so blocking CCBot now only affects future crawls, not existing model weights.
If I block GPTBot, will ChatGPT stop citing me?
Yes — blocking GPTBot prevents OpenAI from fetching your content for its retrieval index. ChatGPT's web-search feature (OAI-SearchBot and ChatGPT-User) is separate from GPTBot, so you may want to block all 3 OpenAI tokens together if your intent is a full opt-out from ChatGPT citations.
Does Seeklyn tell me which bots to allow?
No — Seeklyn reports which of the 14 bots are blocked and which are allowed, without making a recommendation. Whether to allow a given crawler is a business and legal decision (training vs. citations, content licensing, etc.). The crawl-gate audit gives you the facts so you can decide deliberately.
Can I block some AI crawlers but not others?
Yes — robots.txt rules are per user-agent, so you can block GPTBot while allowing PerplexityBot, or allow all AI crawlers while blocking Bytespider. Seeklyn reports each of the 14 tokens individually so you can see exactly which engines have access.