Anonyfox LabAgent reference ↗

Is OpenAI, Anthropic or Perplexity crawling your site?

Look for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in your access-log user agents. Verify the client IP address against the operator’s published ranges, because anyone can copy a user-agent string. Keep training crawlers, search crawlers and user-triggered fetches separate when choosing what to allow.

100 % client-side. Nothing you paste is uploaded.

UA + IP

The name is a claim.
The address is the check.

IPv4 and IPv6 · 18 agent tokens with IP snapshots · checked 2026-11-10

Load the synthetic sample to compare a matching GPTBot address with a deliberately spoofed request.

Common logs without a user-agent field cannot identify bots. CSV headers: ip,user_agent,status,path.

Official ranges load only when you run a check.

Yes: range matchThe address belongs to the operator’s published snapshot. A shared list identifies the operator, not the exact internal task.

No: likely spoofedThe address misses the corresponding list. Inspect proxy configuration and snapshot timing before attributing the request.

No list: unverifiedA matching name cannot establish operator identity. Missing address families remain unverified too.

Use the original client IP from trusted CDN logs. An origin’s remote address may be your proxy; an untrusted forwarded header can be forged. This dated snapshot does not prove who owned an address when an old request occurred.

Eight core AI agents. All eight publish IP ranges.

8 of 8 OpenAI, Anthropic and Perplexity agents in this reference have published IP ranges.

Scope: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Counted on 2026-11-10 from the OpenAI documentation, Anthropic documentation and Perplexity documentation. Anthropic shares one list across its three agents. This is a count of these eight tokens, not every AI crawler on the web.

The full reference below contains 26 tokens, including two policy-only tokens and two unconfirmed Cohere names. 18 tokens have a usable IP snapshot, backed by 16 distinct operator sources. Download the compact range snapshot.

Which bot is that? The operator reference

Purposes and robots policies come from operator documentation. Quotation marks identify short operator excerpts; an absent statement stays absent. “None published” means no dedicated verification list was found in the linked operator docs on 2026-11-10.

Google-Extended is a robots.txt control token, with no separate HTTP crawler. Apple says Applebot-Extended controls training use of Applebot data. Neither token should appear as a detected crawler in an ordinary access log.

26 tokens · operator documentation and IP sources · checked 2026-11-10
Operator / UA tokenPurposerobots.txt tokenRespects robots.txt? Operator statementPublished IP sourceDocs / checked
OpenAI
GPTBot
TrainingGPTBotYes; independent training controlPublished rangesOperator docs
OpenAI
OAI-SearchBot
Search indexOAI-SearchBotYes; independent search controlPublished rangesOperator docs
OpenAI
ChatGPT-User
User-triggered fetchChatGPT-User“robots.txt rules may not apply”Published rangesOperator docs
Anthropic
ClaudeBot
TrainingClaudeBotYes; “honoring industry standard directives in robots.txt”Shared Anthropic list verifies operator, not the precise bot role.Published rangesOperator docs
Anthropic
Claude-SearchBot
Search indexClaude-SearchBotYes; “honoring industry standard directives in robots.txt”Shared list; see the Anthropic verification method.Published rangesOperator docs
Anthropic
Claude-User
User-triggered fetchClaude-UserYes; “honoring industry standard directives in robots.txt”Shared list; see the Anthropic verification method.Published rangesOperator docs
Perplexity
PerplexityBot
Search indexPerplexityBotYes; independent robots controlPublished rangesOperator docs
Perplexity
Perplexity-User
User-triggered fetchPerplexity-User“generally ignores robots.txt rules”Published rangesOperator docs
Google
Google-ExtendedPolicy only; no HTTP agent
Training and Gemini grounding policyGoogle-ExtendedPolicy token; “control capacity”No HTTP user agent. Does not control Google Search inclusion or ranking; Gemini grounding is also covered, so this is not training-only.Not applicable: no separate crawlerOperator docs
Google
Googlebot
Search index, including AI Search featuresGooglebotYes; “always obey robots.txt rules”Published rangesOperator docs
Apple
Applebot
Search; data may also train modelsApplebotYes; “respects standard robots.txt directives”Published rangesOperator docs
Apple
Applebot-ExtendedPolicy only; no HTTP agent
Training-use policyApplebot-ExtendedPolicy token; opt out of trainingNo separate crawler. Allowing Applebot while disallowing this token retains search crawling.Not applicable: no separate crawlerOperator docs
Microsoft
bingbot
Bing index; supports Copilot groundingbingbotYes; “Bingbot honors robots.txt directives”Published rangesOperator docs
Meta
meta-externalagent
Training and product indexingmeta-externalagentUses “Disallow” (operator example)Official documentation was served in German; policy description translated here. No crawler-specific IP file linked by the operator.None publishedOperator docs
Meta
meta-externalfetcher
User-triggered fetchmeta-externalfetcherMay bypass robots.txt; user requestsOperator wording: “kann die robots.txt-Datei umgehen”. No crawler-specific IP file linked; broad Meta network ownership is not proof of this bot.None publishedOperator docs
Amazon
Amazonbot
Products and services; may train AIAmazonbotYes; “respect the Robots Exclusion Protocol”Published rangesOperator docs
Amazon
Amzn-SearchBot
Search indexAmzn-SearchBotYes; separate search controlPublished rangesOperator docs
Amazon
Amzn-User
User-triggered fetchAmzn-User“may not follow all robots.txt directives”Published rangesOperator docs
ByteDance
Bytespider
AI crawler; precise purpose not statedBytespiderNo operator compliance statement foundBytePlus lists the name as an AI crawler. No operator training-purpose claim or dedicated IP verification list found; detection is a claimed token only.None publishedOperator docs
Common Crawl
CCBot
Open web corpus; downstream research and trainingCCBotYes; robots exclusion examplePublished rangesOperator docs
DuckDuckGo
DuckAssistBot
Real-time AI answer retrieval; no trainingDuckAssistBotYes; “after 72 hours”Published rangesOperator docs
Mistral
MistralAI-User
User-triggered fetchMistralAI-UserNo explicit compliance statementPublished rangesOperator docs
Mistral
MistralAI-Index
Search indexMistralAI-IndexNo explicit compliance statementPublished rangesOperator docs
Mistral
MistralAI-Training
TrainingMistralAI-Training“Webmasters can disallow this user agent”None publishedOperator docs
Unconfirmed (claimed Cohere)
cohere-aiUnconfirmed name
Not operator-documentedNot documentedNo operator statement verifiedName requested for comparison; excluded from documented-agent counts and default policy recipes. No official current token documentation found.None publishedNo operator token documentation found
Unconfirmed (claimed Cohere)
cohere-training-data-crawlerUnconfirmed name
Not operator-documentedNot documentedNo operator statement verifiedUnconfirmed historical name; not included in the policy controls.None publishedNo operator token documentation found

Download the full agent table as CSV · JSON dataset

The cohere-ai and cohere-training-data-crawler names remain unconfirmed because no current operator token documentation was retrieved. BytePlus lists Bytespider as an AI crawler but does not establish its training purpose or robots compliance. The checker identifies these strings as claims and assigns no IP verification.

How to verify an AI crawler request

  1. Find the client IP.Export access or CDN request logs with the client IP, user-agent field, request path and HTTP status. Use trusted edge logs if a proxy sits in front of your server.
  2. Detect the claimed agent.Match the user-agent field against the documented token. GPTBot, OAI-SearchBot and ChatGPT-User have different purposes. Matching is case-insensitive and ignores version numbers.
  3. Match the right official range list.Compare IPv4 or IPv6 against the file for that agent. A shared operator list verifies operator membership only. If that list publishes no ranges for the address family, retain an unverified result.
  4. Read the response and choose policy.A 200 records a successful HTTP response; inspect the body if it could be a challenge page. A 403 records refusal and a 429 rate limiting. Choose a robots.txt policy separately from firewall enforcement.
OpenAI: three names, three range files

Match GPTBot to gptbot.json, OAI-SearchBot to searchbot.json and ChatGPT-User to chatgpt-user.json. A match against another OpenAI bot’s file is not a match against the claimed role. OpenAI separates search and training controls and says user-triggered actions may not follow robots rules.

Anthropic: current names and shared operator verification

Anthropic’s support article names ClaudeBot, Claude-SearchBot and Claude-User. Verify all three against the shared Anthropic IP list. The shared list cannot prove which internal system sent a request; the claimed token supplies the role. Anthropic says its bots honor robots.txt and supports Crawl-delay.

Perplexity: search and user fetches have separate lists

Use PerplexityBot’s list for search and Perplexity-User’s list for user actions. Perplexity states that neither is used for foundation-model training. The user fetcher generally ignores robots.txt; the search crawler has its own robots control.

Googlebot, Bingbot and Applebot: verify both DNS directions

Reverse-resolve the client IP to a hostname. Check the operator’s exact domain suffix at a label boundary, then forward-resolve that hostname and confirm it includes the original IP. A hostname such as googlebot.com.attacker.example fails the suffix check; a PTR record alone is insufficient.

Google’s verification guide documents Googlebot hostnames under .googlebot.com and the matching crawler category lists. Use the common-crawler list for Googlebot; the separate special-crawlers snapshot is retained in the source data and is not substituted for Googlebot verification. Bing’s DNS guide requires .search.msn.com and a forward match. Apple documents .applebot.apple.com and its CIDR file.

host 17.58.101.179
host 17-58-101-179.applebot.apple.com

Apple’s documented example, not a log observation. This browser tool checks CIDRs only; it does not perform DNS queries.

Amazon, Common Crawl, DuckDuckGo, Mistral and Meta

Amazon publishes separate sources for Amazonbot, Amzn-SearchBot and Amzn-User; this build extracts the JSON shown within those official HTML pages. Common Crawl publishes dedicated CCBot ranges and a forward-confirmed DNS method for IPv4. Its documentation says reverse DNS is not yet supported for its IPv6 ranges.

DuckAssistBot has a dedicated IP list. Mistral links ranges for MistralAI-User and MistralAI-Index; none is linked for MistralAI-Training. Meta documents crawler purposes and robots exceptions. A broad network or ASN match alone is weaker evidence than a crawler-specific range list, so the tool does not mark Meta requests as IP-verified.

After checking actual visits, use the domain robots.txt and llms.txt checker to inspect the policy currently served by your site. For a review of answer visibility beyond crawl access, the AI visibility audit describes its paid scope separately.

Block training. Keep search available.

This preset blocks the dedicated GPTBot, ClaudeBot and MistralAI-Training crawlers and Apple’s training-use token. It explicitly allows the listed search crawlers. Google-Extended also covers Gemini grounding, while Amazonbot and Meta-ExternalAgent have mixed purposes; those choices stay unchanged until you select them. Common Crawl supports many downstream uses beyond model training.

Choose allow, block or keep unchanged for each token

Merge these groups with your existing file at the host root; repeat any private-path restrictions in specific allow groups. A bot-specific group can supersede wildcard restrictions. OpenAI documents independent search and training controls. Anthropic states that its bots honor robots directives. Apple separates search access from training use.

User-triggered exceptions differ by operator. ChatGPT-User, Perplexity-User, Amzn-User and Meta-ExternalFetcher document exceptions to robots.txt. Claude-User is included in Anthropic’s robots commitment. A robots rule expresses a crawling preference; enforce required access restrictions in your server or CDN.

Changes are not immediate: OpenAI search and Perplexity describe about 24 hours; DuckAssistBot describes 72 hours. Recheck the actual response and crawler logs after updating.

Find the requests in Nginx, Apache or Cloudflare

A combined access log includes the user agent; a common access log usually does not. Filter the user-agent field for counts. Searching the whole line can also match a token in a path or referer, so treat grep output as candidates.

Nginx: extract candidate requests

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log

Apache: extract candidates and count claimed user agents

grep -Ei 'GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|ChatGPT-User|Claude-SearchBot|Claude-User|Perplexity-User' /var/log/apache2/access.log
awk -F'"' '$6 ~ /GPTBot|ClaudeBot|PerplexityBot/ {n[$6]++} END {for (ua in n) print n[ua], ua}' /var/log/apache2/access.log

Paths shown are common Linux defaults; use the path and quote-delimited format configured on your host. The awk example assumes combined format with the user agent in field 6 and counts claims, not verified bots.

Cloudflare: AI Crawl Control

Select your domain → AI Crawl Control → Overview for the activity snapshot, Crawlers for individual controls, and Metrics for breakdowns. These names are from Cloudflare’s current get-started guide, checked 2026-11-10. The guide says free-plan detection uses user-agent strings; stronger Bot Management detection is plan-dependent. Edge-blocked requests may never reach origin logs.

A crawler checking a public page does not need repository credentials. For a separate review of coding tools and access boundaries, see coding-agent security. The Agent Kit and Repo Audit cover repository instructions rather than public-web crawling.

Questions from site owners

These questions follow dated public discussions. The short excerpts show the concern; the answers use the operator documentation checked for this reference.

How can I see whether AI bots are crawling my website?

Use access or CDN request logs, filter the user-agent field and compare the recorded client IP with the correct operator list. A JavaScript visitor counter can miss requests that never run its script. A token match alone is a claim.

“But unclear how much traffic those LLMs.txt pages get.”

Hacker News, 2025-03-21

Policy source: operator documentation.

Why does a bot still hit a site that disallows it in robots.txt?

First verify the client IP and the exact token. Check the robots.txt served on that hostname, the response status and cache timing. Some user-triggered fetchers document exceptions; Anthropic says its bots honor robots.txt. Use edge access rules when requests must be refused.

“Why doesn't ClaudeBot / Anthropic obey robots.txt?”

Reddit r/Anthropic, 2024-04-20

Policy source: operator documentation.

Can I block OpenAI training while allowing ChatGPT search?

Yes. Disallow GPTBot and allow OAI-SearchBot independently. ChatGPT-User is another role; OpenAI says robots.txt rules may not apply to user-initiated actions. The recipe below keeps the choices separate.

“Any other items to add to robots.txt to ban OpenAI?”

Webmasters Stack Exchange, 2023-05-13

Policy source: operator documentation.

Does an AI crawler visit mean my website will be cited?

A visit establishes a request, its response and possibly an operator match. It cannot show whether content was trained on, selected for an answer or cited. Check the actual answer separately.

“#AIretrieval & #AIcitation are not the same problem”

Bluesky, 2026-08-27

Policy source: operator documentation.

Why is GPTBot blocked when robots.txt allows it?

The request can be refused by a CDN rule, bot challenge, rate limit or origin configuration before content is served. Compare edge events and origin logs. An IP match with a 403 or 429 still describes a refused request.

“Our site returned 429 to GPTBot on every page.”

Bluesky, 2026-08-16

Policy source: operator documentation.

Do I need llms.txt to let AI crawlers read my site?

llms.txt is separate from crawler access. Check robots.txt, the actual HTTP response and firewall rules first. The linked domain checker inspects robots.txt and llms.txt; neither file proves past visits or citations.

“Noob here, did not know about llm.txt until I saw Semrush displaying an error for it.”

Reddit r/SEO, 2026-03-06

Policy source: operator documentation.

Should I let AI crawlers access my site at all?

Choose by purpose and operating cost. You can keep search access while refusing selected training crawlers. Check request volume, status codes and useful referrals before changing policy; broad blocks can affect search or requested fetches.

“As a Startup I absolutely want to get crawled.”

Hacker News, 2025-07-01

Policy source: operator documentation.

Sources, snapshot and reusable data

Checked 2026-11-10. The range snapshot contains 16 distinct sources for 18 detectable tokens. Each source carries its original URL, fetch timestamp and file creation time when supplied. Historical ownership, undeclared browser-like requests and the content of an AI answer cannot be inferred from this membership check.

Complete page as Markdown · Agent CSV · Agent JSON · Dated ranges JSON.

The compiled agent reference is available under CC BY 4.0; operator excerpts and published range files retain their respective source terms.

Snapshot source inventory

Prefix counts below are read from the saved source objects during the build. Vendor creation time can predate the fetch date.

Archived source objects · fetched 2026-11-10
Source filePrefixesVendor creation time
ip_amazon.json1292 prefixes; IPv42026-09-08T05:24:10.610952+00:00
ip_amazon_search.json816 prefixes; IPv42026-09-08T05:24:10.610952+00:00
ip_amazon_user.json1023 prefixes; IPv42025-11-04T15:57:38.092840+00:00
ip_apple.json24 prefixes; IPv42026-09-15T10:00:00.000000
ip_bing.json29 prefixes; IPv42024-01-03T10:00:00.121331
ip_chatgpt.json237 prefixes; IPv42026-11-06T18:12:14.689393
ip_claude.json14 prefixes; IPv42026-10-20T00:00:00Z
ip_commoncrawl.json5 prefixes; IPv4 IPv62026-08-11T13:40:00Z
ip_duck.json486 prefixes; IPv42026-09-01T12:44:58.000000
ip_google.json317 prefixes; IPv4 IPv62026-11-09T15:46:02.000000
ip_google_common.json317 prefixes; IPv4 IPv62026-11-09T15:46:02.000000
ip_google_special.json272 prefixes; IPv4 IPv62026-11-09T15:46:02.000000
ip_gpt.json18 prefixes; IPv42026-09-22T02:00:07.000000
ip_mistral_index.json2 prefixes; IPv42026-04-19T12:00:00.000000
ip_mistral_user.json4 prefixes; IPv42025-02-19T12:00:00.000000
ip_perplexity.json13 prefixes; IPv42026-10-28T16:48:51.000000
ip_perplexity_user.json10 prefixes; IPv42026-10-28T16:48:51.000000
ip_search.json39 prefixes; IPv42026-01-02T11:00:00.000000
Fetch timestamps and source hashes

SHA-256 values below refer to the archived JSON objects in this build. Amazon objects were extracted from the linked official pages.

ip_amazon.json
Source: https://developer.amazon.com/amazonbot/ip-addresses/
Fetched: 2026-11-10T00:14:45.976414+00:00
SHA-256: 9a079c663957ffe9b8caa59410479f8d1f7311f85a8a5d1bb1bf2c52f5b4a812
ip_amazon_search.json
Source: https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
Fetched: 2026-11-10T00:14:46.403648+00:00
SHA-256: f5e29bc137cdd2613f775e3754ab9266c95acc43dded8dd5e0da9b8a08417b82
ip_amazon_user.json
Source: https://developer.amazon.com/amazonbot/live-ip-addresses/
Fetched: 2026-11-10T00:14:46.504511+00:00
SHA-256: a3dee4f7a7ef0277c4e2fe94954767ba25d01f6b01643e35a9e323f0836a5482
ip_apple.json
Source: https://search.developer.apple.com/applebot.json
Fetched: 2026-11-10T00:14:08.212910+00:00
SHA-256: 5a5e12188891d998341f4fb50d2b31c80aba5002f5cff9d0fa110c13e4e2edd0
ip_bing.json
Source: https://www.bing.com/toolbox/bingbot.json
Fetched: 2026-11-10T00:14:07.362400+00:00
SHA-256: 14a769e25c9cc745889fb985850415eb901d519404c66206b796e93978eacab5
ip_chatgpt.json
Source: https://openai.com/chatgpt-user.json
Fetched: 2026-11-10T00:14:07.234516+00:00
SHA-256: 7d467d6cdf1f68d9339072bfb7077ee145bba31ba6471f067b2c70b7b4674de9
ip_claude.json
Source: https://claude.com/crawling/bots.json
Fetched: 2026-11-10T00:14:07.202755+00:00
SHA-256: c960bb55504c3c478d0335bc7153e9ca871c26388a69e403cbad121a5bb5c5dd
ip_commoncrawl.json
Source: https://index.commoncrawl.org/ccbot.json
Fetched: 2026-11-10T00:14:45.038283+00:00
SHA-256: 0a4c3b6707326d7f083139a979062604a57edfbb251bd29f23f5a86c5107ea30
ip_duck.json
Source: https://duckduckgo.com/duckassistbot.json
Fetched: 2026-11-10T00:14:08.797742+00:00
SHA-256: fd9039362dfaaad236d50c07ddd12bf0cc7f0d530fbff59b8feb1b22e0db2459
ip_google.json
Source: https://developers.google.com/static/search/apis/ipranges/googlebot.json
Fetched: 2026-11-10T00:14:07.771108+00:00
SHA-256: 336853c43e708e1619b4a3c9d184fbe4b7e04d41723868b72ccace7d3f82bc4d
ip_google_common.json
Source: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Fetched: 2026-11-10T00:14:44.605225+00:00
SHA-256: 336853c43e708e1619b4a3c9d184fbe4b7e04d41723868b72ccace7d3f82bc4d
ip_google_special.json
Source: https://developers.google.com/static/search/apis/ipranges/special-crawlers.json
Fetched: 2026-11-10T00:14:07.651439+00:00
SHA-256: 9788361c25391163440498ddcbd44cead4c5ef701cef7d00b770cb17924e9176
ip_gpt.json
Source: https://openai.com/gptbot.json
Fetched: 2026-11-10T00:14:07.108937+00:00
SHA-256: a24d1fa34f094d303400bdf13c45c5e9726e0b553780d16a52f9ba1b790acc72
ip_mistral_index.json
Source: https://mistral.ai/mistralai-index-ips.json
Fetched: 2026-11-10T00:14:44.158203+00:00
SHA-256: 35bf7aede93ea6a24b12606cf0abb809d8fe9846b2e054e0174dc53f91921c20
ip_mistral_user.json
Source: https://mistral.ai/mistralai-user-ips.json
Fetched: 2026-11-10T00:14:44.229293+00:00
SHA-256: 6226c5829c7edb91995769a9f3f0a52b686b4a18cc5a8d6d92479d2908f793f6
ip_perplexity.json
Source: https://www.perplexity.ai/perplexitybot.json
Fetched: 2026-11-10T00:14:07.230762+00:00
SHA-256: 063764a7bc6b262f019e7d8fb981be0f345f26bc549f08ce6a25292f981a84f2
ip_perplexity_user.json
Source: https://www.perplexity.ai/perplexity-user.json
Fetched: 2026-11-10T00:14:07.229554+00:00
SHA-256: cd0c4e9393b392e06a5f7d8c87c26099552fb646f51bfacdc6175e861ebefb33
ip_search.json
Source: https://openai.com/searchbot.json
Fetched: 2026-11-10T00:14:07.160302+00:00
SHA-256: 81806b1665e9bbbc5e0f307b847ad34f7027abc19e0d2d5d3eca1864bf361be6

Common Crawl source note: IP ranges used by CCBot. For verification of IPv4 addresses, FCrDNS is also recommended. Sync token: 20260811134000.

About Anonyfox Lab

Published by Anonyfox Lab. Anonyfox e. K., owner Maximilian Stroh, Zehntland 2d, 21079 Hamburg, Germany. Amtsgericht Hamburg HRA 131320.

This checker is free. Terms (German) apply to lab services.