Is OpenAI, Anthropic or Perplexity crawling your site?
Look for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in your access-log user agents. Verify the client IP address against the operator’s published ranges, because anyone can copy a user-agent string. Keep training crawlers, search crawlers and user-triggered fetches separate when choosing what to allow.
100 % client-side. Nothing you paste is uploaded.
The name is a claim.
The address is the check.
IPv4 and IPv6 · 18 agent tokens with IP snapshots · checked 2026-11-10
Load the synthetic sample to compare a matching GPTBot address with a deliberately spoofed request.
Common logs without a user-agent field cannot identify bots. CSV headers: ip,user_agent,status,path.
Official ranges load only when you run a check.
Log results
| Agent | Hits | IP matches | Verified share |
|---|
| Line | Detected agent / issue | Operator | Purpose | Client IP | HTTP status | IP verification |
|---|
Verified share = matching hits ÷ hits comparable against a published list for that IP family. Unknown formats, invalid IPs, missing families and unpublished lists are excluded from that denominator.
Yes: range matchThe address belongs to the operator’s published snapshot. A shared list identifies the operator, not the exact internal task.
No: likely spoofedThe address misses the corresponding list. Inspect proxy configuration and snapshot timing before attributing the request.
No list: unverifiedA matching name cannot establish operator identity. Missing address families remain unverified too.
Use the original client IP from trusted CDN logs. An origin’s remote address may be your proxy; an untrusted forwarded header can be forged. This dated snapshot does not prove who owned an address when an old request occurred.
Eight core AI agents. All eight publish IP ranges.
8 of 8 OpenAI, Anthropic and Perplexity agents in this reference have published IP ranges.
Scope: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Counted on 2026-11-10 from the OpenAI documentation, Anthropic documentation and Perplexity documentation. Anthropic shares one list across its three agents. This is a count of these eight tokens, not every AI crawler on the web.
The full reference below contains 26 tokens, including two policy-only tokens and two unconfirmed Cohere names. 18 tokens have a usable IP snapshot, backed by 16 distinct operator sources. Download the compact range snapshot.
Which bot is that? The operator reference
Purposes and robots policies come from operator documentation. Quotation marks identify short operator excerpts; an absent statement stays absent. “None published” means no dedicated verification list was found in the linked operator docs on 2026-11-10.
Google-Extended is a robots.txt control token, with no separate HTTP crawler. Apple says Applebot-Extended controls training use of Applebot data. Neither token should appear as a detected crawler in an ordinary access log.
| Operator / UA token | Purpose | robots.txt token | Respects robots.txt? Operator statement | Published IP source | Docs / checked |
|---|---|---|---|---|---|
OpenAIGPTBot | Training | GPTBot | Yes; independent training control | Published ranges | Operator docs |
OpenAIOAI-SearchBot | Search index | OAI-SearchBot | Yes; independent search control | Published ranges | Operator docs |
OpenAIChatGPT-User | User-triggered fetch | ChatGPT-User | “robots.txt rules may not apply” | Published ranges | Operator docs |
AnthropicClaudeBot | Training | ClaudeBot | Yes; “honoring industry standard directives in robots.txt”Shared Anthropic list verifies operator, not the precise bot role. | Published ranges | Operator docs |
AnthropicClaude-SearchBot | Search index | Claude-SearchBot | Yes; “honoring industry standard directives in robots.txt”Shared list; see the Anthropic verification method. | Published ranges | Operator docs |
AnthropicClaude-User | User-triggered fetch | Claude-User | Yes; “honoring industry standard directives in robots.txt”Shared list; see the Anthropic verification method. | Published ranges | Operator docs |
PerplexityPerplexityBot | Search index | PerplexityBot | Yes; independent robots control | Published ranges | Operator docs |
PerplexityPerplexity-User | User-triggered fetch | Perplexity-User | “generally ignores robots.txt rules” | Published ranges | Operator docs |
GoogleGoogle-ExtendedPolicy only; no HTTP agent | Training and Gemini grounding policy | Google-Extended | Policy token; “control capacity”No HTTP user agent. Does not control Google Search inclusion or ranking; Gemini grounding is also covered, so this is not training-only. | Not applicable: no separate crawler | Operator docs |
GoogleGooglebot | Search index, including AI Search features | Googlebot | Yes; “always obey robots.txt rules” | Published ranges | Operator docs |
AppleApplebot | Search; data may also train models | Applebot | Yes; “respects standard robots.txt directives” | Published ranges | Operator docs |
AppleApplebot-ExtendedPolicy only; no HTTP agent | Training-use policy | Applebot-Extended | Policy token; opt out of trainingNo separate crawler. Allowing Applebot while disallowing this token retains search crawling. | Not applicable: no separate crawler | Operator docs |
Microsoftbingbot | Bing index; supports Copilot grounding | bingbot | Yes; “Bingbot honors robots.txt directives” | Published ranges | Operator docs |
Metameta-externalagent | Training and product indexing | meta-externalagent | Uses “Disallow” (operator example)Official documentation was served in German; policy description translated here. No crawler-specific IP file linked by the operator. | None published | Operator docs |
Metameta-externalfetcher | User-triggered fetch | meta-externalfetcher | May bypass robots.txt; user requestsOperator wording: “kann die robots.txt-Datei umgehen”. No crawler-specific IP file linked; broad Meta network ownership is not proof of this bot. | None published | Operator docs |
AmazonAmazonbot | Products and services; may train AI | Amazonbot | Yes; “respect the Robots Exclusion Protocol” | Published ranges | Operator docs |
AmazonAmzn-SearchBot | Search index | Amzn-SearchBot | Yes; separate search control | Published ranges | Operator docs |
AmazonAmzn-User | User-triggered fetch | Amzn-User | “may not follow all robots.txt directives” | Published ranges | Operator docs |
ByteDanceBytespider | AI crawler; precise purpose not stated | Bytespider | No operator compliance statement foundBytePlus lists the name as an AI crawler. No operator training-purpose claim or dedicated IP verification list found; detection is a claimed token only. | None published | Operator docs |
Common CrawlCCBot | Open web corpus; downstream research and training | CCBot | Yes; robots exclusion example | Published ranges | Operator docs |
DuckDuckGoDuckAssistBot | Real-time AI answer retrieval; no training | DuckAssistBot | Yes; “after 72 hours” | Published ranges | Operator docs |
MistralMistralAI-User | User-triggered fetch | MistralAI-User | No explicit compliance statement | Published ranges | Operator docs |
MistralMistralAI-Index | Search index | MistralAI-Index | No explicit compliance statement | Published ranges | Operator docs |
MistralMistralAI-Training | Training | MistralAI-Training | “Webmasters can disallow this user agent” | None published | Operator docs |
Unconfirmed (claimed Cohere)cohere-aiUnconfirmed name | Not operator-documented | Not documented | No operator statement verifiedName requested for comparison; excluded from documented-agent counts and default policy recipes. No official current token documentation found. | None published | No operator token documentation found |
Unconfirmed (claimed Cohere)cohere-training-data-crawlerUnconfirmed name | Not operator-documented | Not documented | No operator statement verifiedUnconfirmed historical name; not included in the policy controls. | None published | No operator token documentation found |
Download the full agent table as CSV · JSON dataset
The cohere-ai and cohere-training-data-crawler names remain unconfirmed because no current operator token documentation was retrieved. BytePlus lists Bytespider as an AI crawler but does not establish its training purpose or robots compliance. The checker identifies these strings as claims and assigns no IP verification.
How to verify an AI crawler request
- Find the client IP.Export access or CDN request logs with the client IP, user-agent field, request path and HTTP status. Use trusted edge logs if a proxy sits in front of your server.
- Detect the claimed agent.Match the user-agent field against the documented token. GPTBot, OAI-SearchBot and ChatGPT-User have different purposes. Matching is case-insensitive and ignores version numbers.
- Match the right official range list.Compare IPv4 or IPv6 against the file for that agent. A shared operator list verifies operator membership only. If that list publishes no ranges for the address family, retain an unverified result.
- Read the response and choose policy.A 200 records a successful HTTP response; inspect the body if it could be a challenge page. A 403 records refusal and a 429 rate limiting. Choose a robots.txt policy separately from firewall enforcement.
OpenAI: three names, three range files
Match GPTBot to gptbot.json, OAI-SearchBot to searchbot.json and ChatGPT-User to chatgpt-user.json. A match against another OpenAI bot’s file is not a match against the claimed role. OpenAI separates search and training controls and says user-triggered actions may not follow robots rules.
Anthropic: current names and shared operator verification
Anthropic’s support article names ClaudeBot, Claude-SearchBot and Claude-User. Verify all three against the shared Anthropic IP list. The shared list cannot prove which internal system sent a request; the claimed token supplies the role. Anthropic says its bots honor robots.txt and supports Crawl-delay.
Perplexity: search and user fetches have separate lists
Use PerplexityBot’s list for search and Perplexity-User’s list for user actions. Perplexity states that neither is used for foundation-model training. The user fetcher generally ignores robots.txt; the search crawler has its own robots control.
Googlebot, Bingbot and Applebot: verify both DNS directions
Reverse-resolve the client IP to a hostname. Check the operator’s exact domain suffix at a label boundary, then forward-resolve that hostname and confirm it includes the original IP. A hostname such as googlebot.com.attacker.example fails the suffix check; a PTR record alone is insufficient.
Google’s verification guide documents Googlebot hostnames under .googlebot.com and the matching crawler category lists. Use the common-crawler list for Googlebot; the separate special-crawlers snapshot is retained in the source data and is not substituted for Googlebot verification. Bing’s DNS guide requires .search.msn.com and a forward match. Apple documents .applebot.apple.com and its CIDR file.
host 17.58.101.179
host 17-58-101-179.applebot.apple.comApple’s documented example, not a log observation. This browser tool checks CIDRs only; it does not perform DNS queries.
Amazon, Common Crawl, DuckDuckGo, Mistral and Meta
Amazon publishes separate sources for Amazonbot, Amzn-SearchBot and Amzn-User; this build extracts the JSON shown within those official HTML pages. Common Crawl publishes dedicated CCBot ranges and a forward-confirmed DNS method for IPv4. Its documentation says reverse DNS is not yet supported for its IPv6 ranges.
DuckAssistBot has a dedicated IP list. Mistral links ranges for MistralAI-User and MistralAI-Index; none is linked for MistralAI-Training. Meta documents crawler purposes and robots exceptions. A broad network or ASN match alone is weaker evidence than a crawler-specific range list, so the tool does not mark Meta requests as IP-verified.
After checking actual visits, use the domain robots.txt and llms.txt checker to inspect the policy currently served by your site. For a review of answer visibility beyond crawl access, the AI visibility audit describes its paid scope separately.
Block training. Keep search available.
This preset blocks the dedicated GPTBot, ClaudeBot and MistralAI-Training crawlers and Apple’s training-use token. It explicitly allows the listed search crawlers. Google-Extended also covers Gemini grounding, while Amazonbot and Meta-ExternalAgent have mixed purposes; those choices stay unchanged until you select them. Common Crawl supports many downstream uses beyond model training.
Choose allow, block or keep unchanged for each token
Merge these groups with your existing file at the host root; repeat any private-path restrictions in specific allow groups. A bot-specific group can supersede wildcard restrictions. OpenAI documents independent search and training controls. Anthropic states that its bots honor robots directives. Apple separates search access from training use.
User-triggered exceptions differ by operator. ChatGPT-User, Perplexity-User, Amzn-User and Meta-ExternalFetcher document exceptions to robots.txt. Claude-User is included in Anthropic’s robots commitment. A robots rule expresses a crawling preference; enforce required access restrictions in your server or CDN.
Changes are not immediate: OpenAI search and Perplexity describe about 24 hours; DuckAssistBot describes 72 hours. Recheck the actual response and crawler logs after updating.
Find the requests in Nginx, Apache or Cloudflare
A combined access log includes the user agent; a common access log usually does not. Filter the user-agent field for counts. Searching the whole line can also match a token in a path or referer, so treat grep output as candidates.
Nginx: extract candidate requests
grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.logApache: extract candidates and count claimed user agents
grep -Ei 'GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|ChatGPT-User|Claude-SearchBot|Claude-User|Perplexity-User' /var/log/apache2/access.log
awk -F'"' '$6 ~ /GPTBot|ClaudeBot|PerplexityBot/ {n[$6]++} END {for (ua in n) print n[ua], ua}' /var/log/apache2/access.logPaths shown are common Linux defaults; use the path and quote-delimited format configured on your host. The awk example assumes combined format with the user agent in field 6 and counts claims, not verified bots.
Cloudflare: AI Crawl Control
Select your domain → AI Crawl Control → Overview for the activity snapshot, Crawlers for individual controls, and Metrics for breakdowns. These names are from Cloudflare’s current get-started guide, checked 2026-11-10. The guide says free-plan detection uses user-agent strings; stronger Bot Management detection is plan-dependent. Edge-blocked requests may never reach origin logs.
A crawler checking a public page does not need repository credentials. For a separate review of coding tools and access boundaries, see coding-agent security. The Agent Kit and Repo Audit cover repository instructions rather than public-web crawling.
Questions from site owners
These questions follow dated public discussions. The short excerpts show the concern; the answers use the operator documentation checked for this reference.
How can I see whether AI bots are crawling my website?
Use access or CDN request logs, filter the user-agent field and compare the recorded client IP with the correct operator list. A JavaScript visitor counter can miss requests that never run its script. A token match alone is a claim.
“But unclear how much traffic those LLMs.txt pages get.”
Hacker News, 2025-03-21
Policy source: operator documentation.
Why does a bot still hit a site that disallows it in robots.txt?
First verify the client IP and the exact token. Check the robots.txt served on that hostname, the response status and cache timing. Some user-triggered fetchers document exceptions; Anthropic says its bots honor robots.txt. Use edge access rules when requests must be refused.
“Why doesn't ClaudeBot / Anthropic obey robots.txt?”
Reddit r/Anthropic, 2024-04-20
Policy source: operator documentation.
Can I block OpenAI training while allowing ChatGPT search?
Yes. Disallow GPTBot and allow OAI-SearchBot independently. ChatGPT-User is another role; OpenAI says robots.txt rules may not apply to user-initiated actions. The recipe below keeps the choices separate.
“Any other items to add to robots.txt to ban OpenAI?”
Webmasters Stack Exchange, 2023-05-13
Policy source: operator documentation.
Does an AI crawler visit mean my website will be cited?
A visit establishes a request, its response and possibly an operator match. It cannot show whether content was trained on, selected for an answer or cited. Check the actual answer separately.
“#AIretrieval & #AIcitation are not the same problem”
Bluesky, 2026-08-27
Policy source: operator documentation.
Why is GPTBot blocked when robots.txt allows it?
The request can be refused by a CDN rule, bot challenge, rate limit or origin configuration before content is served. Compare edge events and origin logs. An IP match with a 403 or 429 still describes a refused request.
“Our site returned 429 to GPTBot on every page.”
Bluesky, 2026-08-16
Policy source: operator documentation.
Do I need llms.txt to let AI crawlers read my site?
llms.txt is separate from crawler access. Check robots.txt, the actual HTTP response and firewall rules first. The linked domain checker inspects robots.txt and llms.txt; neither file proves past visits or citations.
“Noob here, did not know about llm.txt until I saw Semrush displaying an error for it.”
Reddit r/SEO, 2026-03-06
Policy source: operator documentation.
Should I let AI crawlers access my site at all?
Choose by purpose and operating cost. You can keep search access while refusing selected training crawlers. Check request volume, status codes and useful referrals before changing policy; broad blocks can affect search or requested fetches.
“As a Startup I absolutely want to get crawled.”
Hacker News, 2025-07-01
Policy source: operator documentation.
Sources, snapshot and reusable data
Checked 2026-11-10. The range snapshot contains 16 distinct sources for 18 detectable tokens. Each source carries its original URL, fetch timestamp and file creation time when supplied. Historical ownership, undeclared browser-like requests and the content of an AI answer cannot be inferred from this membership check.
- OpenAI crawler documentation · training, search and user-triggered fetches.
- Anthropic crawler support · names, shared IP list and robots policies.
- Perplexity crawler documentation · independent lists and user-fetch exception.
- Google common crawlers and request verification.
- Applebot, Bing crawlers, Meta crawlers and Amazon web search crawlers.
- Common Crawl CCBot, DuckAssistBot, Mistral crawlers and BytePlus crawler name reference.
- Cloudflare AI Crawl Control · dashboard naming and detection scope.
Complete page as Markdown · Agent CSV · Agent JSON · Dated ranges JSON.
The compiled agent reference is available under CC BY 4.0; operator excerpts and published range files retain their respective source terms.
Snapshot source inventory
Prefix counts below are read from the saved source objects during the build. Vendor creation time can predate the fetch date.
| Source file | Prefixes | Vendor creation time |
|---|---|---|
| ip_amazon.json | 1292 prefixes; IPv4 | 2026-09-08T05:24:10.610952+00:00 |
| ip_amazon_search.json | 816 prefixes; IPv4 | 2026-09-08T05:24:10.610952+00:00 |
| ip_amazon_user.json | 1023 prefixes; IPv4 | 2025-11-04T15:57:38.092840+00:00 |
| ip_apple.json | 24 prefixes; IPv4 | 2026-09-15T10:00:00.000000 |
| ip_bing.json | 29 prefixes; IPv4 | 2024-01-03T10:00:00.121331 |
| ip_chatgpt.json | 237 prefixes; IPv4 | 2026-11-06T18:12:14.689393 |
| ip_claude.json | 14 prefixes; IPv4 | 2026-10-20T00:00:00Z |
| ip_commoncrawl.json | 5 prefixes; IPv4 IPv6 | 2026-08-11T13:40:00Z |
| ip_duck.json | 486 prefixes; IPv4 | 2026-09-01T12:44:58.000000 |
| ip_google.json | 317 prefixes; IPv4 IPv6 | 2026-11-09T15:46:02.000000 |
| ip_google_common.json | 317 prefixes; IPv4 IPv6 | 2026-11-09T15:46:02.000000 |
| ip_google_special.json | 272 prefixes; IPv4 IPv6 | 2026-11-09T15:46:02.000000 |
| ip_gpt.json | 18 prefixes; IPv4 | 2026-09-22T02:00:07.000000 |
| ip_mistral_index.json | 2 prefixes; IPv4 | 2026-04-19T12:00:00.000000 |
| ip_mistral_user.json | 4 prefixes; IPv4 | 2025-02-19T12:00:00.000000 |
| ip_perplexity.json | 13 prefixes; IPv4 | 2026-10-28T16:48:51.000000 |
| ip_perplexity_user.json | 10 prefixes; IPv4 | 2026-10-28T16:48:51.000000 |
| ip_search.json | 39 prefixes; IPv4 | 2026-01-02T11:00:00.000000 |
Fetch timestamps and source hashes
SHA-256 values below refer to the archived JSON objects in this build. Amazon objects were extracted from the linked official pages.
ip_amazon.json
Source: https://developer.amazon.com/amazonbot/ip-addresses/
Fetched: 2026-11-10T00:14:45.976414+00:00
SHA-256: 9a079c663957ffe9b8caa59410479f8d1f7311f85a8a5d1bb1bf2c52f5b4a812ip_amazon_search.json
Source: https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
Fetched: 2026-11-10T00:14:46.403648+00:00
SHA-256: f5e29bc137cdd2613f775e3754ab9266c95acc43dded8dd5e0da9b8a08417b82ip_amazon_user.json
Source: https://developer.amazon.com/amazonbot/live-ip-addresses/
Fetched: 2026-11-10T00:14:46.504511+00:00
SHA-256: a3dee4f7a7ef0277c4e2fe94954767ba25d01f6b01643e35a9e323f0836a5482ip_apple.json
Source: https://search.developer.apple.com/applebot.json
Fetched: 2026-11-10T00:14:08.212910+00:00
SHA-256: 5a5e12188891d998341f4fb50d2b31c80aba5002f5cff9d0fa110c13e4e2edd0ip_bing.json
Source: https://www.bing.com/toolbox/bingbot.json
Fetched: 2026-11-10T00:14:07.362400+00:00
SHA-256: 14a769e25c9cc745889fb985850415eb901d519404c66206b796e93978eacab5ip_chatgpt.json
Source: https://openai.com/chatgpt-user.json
Fetched: 2026-11-10T00:14:07.234516+00:00
SHA-256: 7d467d6cdf1f68d9339072bfb7077ee145bba31ba6471f067b2c70b7b4674de9ip_claude.json
Source: https://claude.com/crawling/bots.json
Fetched: 2026-11-10T00:14:07.202755+00:00
SHA-256: c960bb55504c3c478d0335bc7153e9ca871c26388a69e403cbad121a5bb5c5ddip_commoncrawl.json
Source: https://index.commoncrawl.org/ccbot.json
Fetched: 2026-11-10T00:14:45.038283+00:00
SHA-256: 0a4c3b6707326d7f083139a979062604a57edfbb251bd29f23f5a86c5107ea30ip_duck.json
Source: https://duckduckgo.com/duckassistbot.json
Fetched: 2026-11-10T00:14:08.797742+00:00
SHA-256: fd9039362dfaaad236d50c07ddd12bf0cc7f0d530fbff59b8feb1b22e0db2459ip_google.json
Source: https://developers.google.com/static/search/apis/ipranges/googlebot.json
Fetched: 2026-11-10T00:14:07.771108+00:00
SHA-256: 336853c43e708e1619b4a3c9d184fbe4b7e04d41723868b72ccace7d3f82bc4dip_google_common.json
Source: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Fetched: 2026-11-10T00:14:44.605225+00:00
SHA-256: 336853c43e708e1619b4a3c9d184fbe4b7e04d41723868b72ccace7d3f82bc4dip_google_special.json
Source: https://developers.google.com/static/search/apis/ipranges/special-crawlers.json
Fetched: 2026-11-10T00:14:07.651439+00:00
SHA-256: 9788361c25391163440498ddcbd44cead4c5ef701cef7d00b770cb17924e9176ip_gpt.json
Source: https://openai.com/gptbot.json
Fetched: 2026-11-10T00:14:07.108937+00:00
SHA-256: a24d1fa34f094d303400bdf13c45c5e9726e0b553780d16a52f9ba1b790acc72ip_mistral_index.json
Source: https://mistral.ai/mistralai-index-ips.json
Fetched: 2026-11-10T00:14:44.158203+00:00
SHA-256: 35bf7aede93ea6a24b12606cf0abb809d8fe9846b2e054e0174dc53f91921c20ip_mistral_user.json
Source: https://mistral.ai/mistralai-user-ips.json
Fetched: 2026-11-10T00:14:44.229293+00:00
SHA-256: 6226c5829c7edb91995769a9f3f0a52b686b4a18cc5a8d6d92479d2908f793f6ip_perplexity.json
Source: https://www.perplexity.ai/perplexitybot.json
Fetched: 2026-11-10T00:14:07.230762+00:00
SHA-256: 063764a7bc6b262f019e7d8fb981be0f345f26bc549f08ce6a25292f981a84f2ip_perplexity_user.json
Source: https://www.perplexity.ai/perplexity-user.json
Fetched: 2026-11-10T00:14:07.229554+00:00
SHA-256: cd0c4e9393b392e06a5f7d8c87c26099552fb646f51bfacdc6175e861ebefb33ip_search.json
Source: https://openai.com/searchbot.json
Fetched: 2026-11-10T00:14:07.160302+00:00
SHA-256: 81806b1665e9bbbc5e0f307b847ad34f7027abc19e0d2d5d3eca1864bf361be6Common Crawl source note: IP ranges used by CCBot. For verification of IPv4 addresses, FCrDNS is also recommended. Sync token: 20260811134000.
About Anonyfox Lab
Published by Anonyfox Lab. Anonyfox e. K., owner Maximilian Stroh, Zehntland 2d, 21079 Hamburg, Germany. Amtsgericht Hamburg HRA 131320.
This checker is free. Terms (German) apply to lab services.