Content analyzer
What is the crawl gate (and why does it block AI citations)?
The crawl gate is 3 hard prerequisites every page must pass before an AI answer engine can cite it. The page must return HTTP 2xx and carry no noindex directive. AI crawlers must not be blocked in robots.txt. Fail any 1 of these 3 checks and the page is invisible to that engine. Seeklyn reports all 3 checks with strong evidence.
The crawl gate appears as the "AI access" group in every Seeklyn live-URL content report — run your page free at /dashboard/content.
What is the crawl gate?
The crawl gate is the set of 3 technical prerequisites an answer engine must clear before it can ever index or cite a page. Unlike content quality (which is a gradient), crawl-gate failures are binary: a page either passes or it is invisible to that engine. The 3 checks are HTTP status (must be 2xx), indexability (must not carry a noindex directive), and AI-crawler access (the engine's specific user-agent must not be disallowed in robots.txt).
Seeklyn grades every crawl-gate check with strong evidence — the source is official search-engine documentation, and the mechanism is not correlational but definitional. A page that returns a 404 is not in the index. A page with noindex is not indexed. A page that blocks GPTBot is not fetched by OpenAI. The crawl-gate checks are not statistical risks; they are hard blockers. Content optimization is wasted effort on a page that fails any of these 3 gates.
What are the 3 crawl-gate checks?
Each of the 3 checks maps to a specific failure mode with its own fix. HTTP status fails if the page returns a non-2xx status code — serve a 200 to fix. Indexability fails if the page carries a noindex directive in the meta robots tag or X-Robots-Tag header — remove the noindex directive. AI-crawler access fails if the engine's bot is blocked by a Disallow rule in robots.txt — remove that rule to allow the engine to cite the page.
| Check | Pass condition | Fail consequence | Evidence |
|---|---|---|---|
| HTTP status | Returns 2xx | Page is not fetched — no index entry possible | Strong (engine docs) |
| No noindex | No noindex in meta robots or X-Robots-Tag | Page is invisible to every engine | Strong (Google Search Central) |
| AI crawlers not blocked | No Disallow for the bot in robots.txt | That specific engine cannot fetch or cite the page | Strong (robots.txt spec + engine docs) |
Frequently asked questions
Does a crawl-gate failure affect my regular Google ranking?
Yes — a noindex removes the page from all Google indexing, not just AI Overviews. Blocking Googlebot in robots.txt removes all organic ranking for that page. Crawl-gate checks are not AEO-specific; they are fundamental to any search visibility.
Can a page pass the crawl gate and still not be cited?
Yes — passing the crawl gate means the engine can find the page, but it still needs to retrieve the relevant passage (Retrieval stage) and find it quotable (Citation stage). The crawl gate is the prerequisite for the citation funnel's 3 stages, not a guarantee of citation.
What if my robots.txt is unreachable?
A missing or unreachable robots.txt means all crawlers are allowed by default — robots.txt absence is not a block. Seeklyn reports Fetched=false in this case and treats it as no restrictions. Only explicit Disallow rules count.
Should I allow all AI crawlers?
That is a business decision, not a technical one. Allowing them is required to be cited — block them and you opt out of AI citation for that engine. Seeklyn reports which bots are blocked without judgment; whether to unblock them is yours to decide.