Analysis

Why public IP blocklists block Googlebot, and what it costs you

If you have ever pulled a free IP-reputation list into your firewall and later found that your site stopped appearing in search results, that your uptime checks went silent, or that an AI answer engine suddenly could not read you, this page is about why that happens. It is not a configuration mistake at your end. It is a property of the lists.

We measured it. On 2026-07-06 we cross-referenced several widely-used public IP-reputation lists against our own curated reference set of known-good infrastructure: search and AI crawlers, public DNS resolvers, and attributed research scanners. The headline result: 3.7% of a widely-used abuse-report feed's top 100,000 entries, 3,725 addresses, is legitimate infrastructure that the list flags at high confidence.

We do not name the lists on this page. The point is not that one of them is bad; the pattern is consistent across all of the ones that try to answer the question "is this address scanning or attacking?"

What the measurement found

The two clean lists score zero because each answers one narrow question well, and neither of them tells you anything about scanners. The lists that do answer the scanner question are the ones carrying the false positives. That is the trade you are making, whether or not anyone told you about it.

Among those false positives: 281 Baidu and 15 Yandex crawler addresses on a single community blocklist, and 103 OpenAI and 24 Google crawler addresses across the lists we measured. Consume them raw and you silently cut yourself off from four search engines and from the AI answer engines being built on top of them.

Why it happens

None of this requires anyone to be careless. It falls out of how the lists are built.

• Crawlers look like scanners. A well-behaved crawler fetches URLs that do not exist, probes hosts that never asked to be probed, and does so from many addresses in parallel. To a sensor that is counting unsolicited connections, that is the signature of a scan. The difference between a crawler and a scanner is intent and identity, and neither is visible in a packet count.
• Firewall logs become abuse reports. A large share of community reporting is generated automatically from edge-device logs. If a device logs a blocked connection and the pipeline turns logs into reports, whatever was blocked gets reported, including the crawler that was blocked because it was on the list in the first place. The loop closes on itself.
• Cloud addresses change hands. Crawler and fetcher infrastructure increasingly runs on the same commodity cloud ranges as everything else. An address that genuinely hosted something hostile last quarter may be a legitimate fetcher this quarter, and lists rarely expire entries as fast as providers reassign them.
• Nobody is paid to remove entries. Adding an address costs one report. Removing one costs a human. The asymmetry is structural, and it accumulates.

The workaround that does not work

The obvious defence is to keep only the entries a list is most confident about: the "just filter to confidence ≥ 90" school of blocklist hygiene. It does not help here, and the measurement says why.

Of the 3,725 legitimate addresses in that abuse-report feed, 3,060 carry a 100% abuse-confidence score. The mean confidence across all of them is 97.5. The legitimate infrastructure is not sitting in the uncertain tail of the list; it is exactly what the list is most certain is malicious, because it is exactly what generates the most reports.

You cannot threshold your way out of a curation problem. Raising the confidence floor removes the marginal entries and keeps the crawlers.

What it actually costs

The failure mode is quiet, which is what makes it expensive. Nothing alerts.

• De-indexing. Block the crawler and the crawl fails; the crawler backs off; the pages age out. By the time anyone notices the traffic drop, the cause is weeks in the past and buried in a firewall rule nobody remembers adding.
• Invisibility to AI answer engines. The same block applies to the crawlers feeding AI search products. If your documentation cannot be read, it cannot be cited, and the answer your customer gets is somebody else's.
• Broken monitoring. Cloud load-balancer health checks and third-party uptime probes look exactly like scanners to a naive sensor. Blocking them produces "outages" that only exist inside your own dashboards.
• Blinded research. Attributed internet-measurement scanners are how exposure data about your own estate gets collected. Blocking them does not make you less exposed; it makes you less informed, and it does nothing to the unattributed scanning you actually care about.
• Support load. Every false positive eventually arrives as a ticket from someone who cannot reach you, and each one costs more to diagnose than the list saved.

What to do instead

Verify crawlers properly rather than trusting a user-agent string. Every major search and AI crawler either publishes its own IP ranges as machine-readable JSON or supports forward-confirmed reverse DNS. We keep the official sources, for every crawler worth allowing, on one page: search and AI crawler IP ranges. Fetch those, refresh them on a schedule, and allow them ahead of any reputation list.

Screen the list before you enforce it. If a reputation feed is going into a blocking decision, it needs to be intersected with a known-good set first. Doing that yourself means maintaining that known-good set, which is the work, not the intersection.

Or use a feed that has already done it. That is what the SciScope Scanner Feed is: first-party signal from our own sensors, screened continuously against exactly the categories above, so that none of the 3,725 addresses in that measurement, and nothing like them, reaches your blocklist. Of those false positives, 0 reach the SciScope feed. We re-measure the public lists and publish the results rather than asking to be trusted.