Reference

Search and AI crawler IP ranges: the official sources

The user-agent string is not evidence. Anything can send Mozilla/5.0 (compatible; Googlebot/2.1), and plenty of things do, it is the oldest trick in the scraping book, and it is why a "block everything except the crawlers we like" rule written against user agents protects nothing.

There are exactly two trustworthy ways to establish that a request really came from a crawler operator: match the source address against a range list the operator publishes itself, or run a forward-confirmed reverse DNS lookup against the hostnames the operator documents. This page collects the official sources for both, for the crawlers worth allowing. Everything linked below is published by the operator, not by us; we keep the page current because we consume the same sources.

Last verified: 2026-09-19.

Operators that publish machine-readable IP ranges

These are the easy ones. Fetch the JSON, match the address against the prefixes, and you are done: no DNS round trip, no spoofable hostname.

All of the JSON files above use the same shape, which makes them trivial to consume together: a creationTime and a prefixes array of ipv4Prefix / ipv6Prefix objects.

$ curl -s https://developers.google.com/search/apis/ipranges/googlebot.json | head -12
{
  "creationTime": "2026-09-18T14:46:19.000000",
  "prefixes": [
    { "ipv4Prefix": "192.0.2.0/27" },
    { "ipv6Prefix": "2001:db8:c0f::/44" },
    ...
  ]
}

(prefixes above replaced with documentation ranges; fetch the file for the real ones.)

Because the shape is shared, one loop covers all of them:

#!/bin/sh
# Collect every published crawler prefix into one file, for an allow rule.
# Run it daily; the operators do change these.
set -eu
for url in \
  https://developers.google.com/search/apis/ipranges/googlebot.json \
  https://developers.google.com/search/apis/ipranges/special-crawlers.json \
  https://developers.google.com/search/apis/ipranges/user-triggered-fetchers.json \
  https://www.bing.com/toolbox/bingbot.json \
  https://search.developer.apple.com/applebot.json \
  https://openai.com/gptbot.json \
  https://openai.com/searchbot.json \
  https://openai.com/chatgpt-user.json \
  https://claude.com/crawling/bots.json \
  https://www.perplexity.ai/perplexitybot.json
do
  curl -fsSL --max-time 20 "$url" \
    | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
done | sort -u > crawler-prefixes.txt

Two practical notes. Fetch with a timeout and keep the previous copy on failure: an empty allow list is worse than a stale one. And do not block these ranges as a way of opting out of AI training: robots.txt is the opt-out, and a crawler that cannot reach you cannot read the robots.txt that tells it to go away.

Operators that verify by reverse DNS

The rest do not publish ranges, because their crawl infrastructure moves. For these, the documented method is forward-confirmed reverse DNS: look up the PTR record of the source address, check that the hostname ends in one of the operator's documented domains, then resolve that hostname back and confirm it returns the original address. The second step is the one people skip, and it is the one that matters. PTR records can be set by whoever controls the address block.

The check, in the form you would actually write it:

# forward-confirmed reverse DNS, by hand
$ host 192.0.2.10
10.2.0.192.in-addr.arpa domain name pointer crawl-192-0-2-10.googlebot.com.

$ host crawl-192-0-2-10.googlebot.com
crawl-192-0-2-10.googlebot.com has address 192.0.2.10

Both steps must succeed, the hostname must end in a documented domain (match on the suffix, anchored: notgooglebot.com and googlebot.com.evil.example are not Googlebot), and the forward lookup must return the address you started with. If any of those fails, the request is not from the operator, whatever its user agent claims.

Two things this list will not do for you

It is not a whitelist of everything you should allow. Public DNS resolvers, NTP servers, the DNS root, cloud health-checkers, uptime monitors and attributed research scanners all get swept into IP blocklists too, and none of them publishes a tidy JSON file. Assembling and maintaining that reference set is the actual work.

It does not stop your reputation feed from listing them anyway. Allowing crawlers ahead of a blocklist helps only where you control the ordering. Where the list feeds a scoring pipeline, an automated report, or a vendor product you do not control, the false positives travel with it, and as we measured on 2026-07-06, a widely-used abuse-report feed lists 3,725 legitimate addresses in its top 100,000 at a mean confidence of 97.5. Details in why public blocklists block Googlebot.