Short answer
AI search such as ChatGPT Search or Perplexity can only cite what its crawlers are allowed to read and able to read. The most common blockers are a robots.txt that shuts out AI bots and content that only appears via JavaScript. This check tests both in seconds. A good result only means the door is open. Whether an AI names you depends on much more, and nobody can guarantee it.
What is checked (method)
- AI bots in robots.txt (25 points): 12 known crawlers, evaluated per RFC 9309 (most specific user-agent group, longest matching rule). Points only for search and fetch bots (e.g. OAI-SearchBot, PerplexityBot, ChatGPT-User). Training crawlers (GPTBot, Google-Extended, CCBot …) are shown but not scored: allowing training is your call.
- Text without JavaScript (20): how much visible text the delivered HTML contains. Many AI crawlers do not run JavaScript.
- Structured data (15): which JSON-LD types are present (detected, not fully validated).
- Reachability, noindex, meta (10 each): HTTP status, noindex via meta tag or X-Robots-Tag, title, description, canonical, exactly one H1.
- Sitemap (10), robots.txt (5), llms.txt (5): llms.txt is a young proposal; major providers have not confirmed using it, so it counts little.
- Limits: one page is checked, not the whole site. Firewalls that block AI bots regardless of robots.txt are invisible to us. The result is a snapshot.
Sources
- OpenAI: Crawler-Übersicht
- Anthropic: ClaudeBot & Co.
- Perplexity: PerplexityBot
- Google: Google-Extended
- Apple: Applebot-Extended
- RFC 9309 (robots.txt)
- llmstxt.org (Vorschlag)
- schema.org
Last updated:
What the check sees
Crawlers do not read designs; they read HTML, rules and data. The check shows you exactly that layer: what your robots.txt says, how much text arrives without JavaScript and which facts are machine-readable.
FAQ
Is this a ranking or visibility measurement?
No. It only measures whether AI crawlers may and can technically read your page. It does not measure whether ChatGPT or Perplexity mention you. Anyone guaranteeing that promises too much: AI answers vary from query to query.
Should I block or allow GPTBot?
That is a business decision. GPTBot collects training data; OAI-SearchBot and ChatGPT-User fetch content for search and specific user requests. Many publishers block training but allow search bots. The check shows both groups separately.
Do I need an llms.txt?
Not a must. It is a proposal from llmstxt.org and use by major AI providers is unconfirmed. It is cheap to add, so it counts for only 5 of 110 points.
Why is my text score low?
If your page builds its content in the browser with JavaScript (common with some React, Vue or site-builder pages), a crawler without JavaScript sees almost nothing. Fix: server-side rendering or static pre-rendering.
Does your checker respect robots.txt itself?
Yes. It identifies honestly as “AnonyfoxLabKICheck/1.0”. If your robots.txt blocks it or all bots (*), we do not fetch the page and only show the robots.txt analysis.
What does it cost?
The quick check is free with no sign-up. For the whole site, order the AI readiness audit for €29.
Privacy
To run the check we process the address you enter and fetch robots.txt, the page, llms.txt and the sitemap from it (Art. 6(1)(b) and (f) GDPR). We store only anonymous statistics: a truncated hash of the hostname, the top-level domain and the check values, never the address in plain text and never your IP address. Results stay in memory for up to 10 minutes to avoid repeat fetches. For rate limiting we count requests per IP address in memory for at most 1 minute. If you share a result link, the address is in the URL and therefore in our server logs (14 days). Please do not enter addresses that contain personal data or access tokens. Details (German): Datenschutzerklärung.
More than one page? The AI readiness audit
The quick check covers one page. The audit crawls up to 200 pages of your site, measures load time with Lighthouse (3 runs, median and range) and delivers a PDF with up to 10 prioritised actions, plus an llms.txt draft and a robots.txt template for AI bots. Fixed price €29, no subscription, no ranking promises.
See the audit (€29)Delivered by email within 48–72 hours. View the sample PDF.