Integration guide

Load an IP blocklist into ipset and iptables

On a legacy iptables host the blocklist belongs in an ipset (a kernel hash set matched in constant time) and never in iptables rules themselves. Ten thousand -s <ip> -j DROP rules are evaluated linearly for every packet; one ipset match is a single lookup regardless of size.

The interesting part of this guide is the update. ipset has no transaction file the way nftables does, but it has something just as good: ipset swap, which exchanges the contents of two sets atomically. Build the new set beside the live one, swap, then destroy the old contents. The rule never sees an empty set.

One-time setup

# A hash:net set, because the feed can contain CIDR ranges as well as addresses.
sudo ipset create sciscope hash:net family inet  maxelem 262144 -exist
sudo ipset create sciscope6 hash:net family inet6 maxelem 262144 -exist

# Match as early as possible, and count what you drop.
sudo iptables  -I INPUT   1 -m set --match-set sciscope  src -j DROP
sudo iptables  -I FORWARD 1 -m set --match-set sciscope  src -j DROP
sudo ip6tables -I INPUT   1 -m set --match-set sciscope6 src -j DROP
sudo ip6tables -I FORWARD 1 -m set --match-set sciscope6 src -j DROP

Size maxelem generously: it is fixed at creation, and a set that fills up starts refusing entries rather than growing. The feed is in the thousands today; 262,144 costs a few megabytes of kernel memory and removes the question.

Persist the rules the way your distribution does it (iptables-persistent, netfilter-persistent, or your configuration manager). Persist only the rules, not the set contents. The refresh below repopulates them after boot, and a restored-from-disk blocklist is a stale blocklist.

The refresh script

#!/bin/bash
# /usr/local/sbin/sciscope-ipset-update
# Rebuild the ipset blocklists from the SciScope Scanner Feed, atomically.
set -euo pipefail

KEY_FILE=/etc/sciscope/key
URL=https://api.sciscope.ee/v1/feeds/blocklist.txt
MIN_ENTRIES=500

tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT

curl -fsSL --max-time 60 --retry 3 --retry-delay 5 \
     -H "Authorization: Bearer $(cat "$KEY_FILE")" \
     -o "$tmp/raw" "$URL"

grep -Eo '^[^#[:space:]]+' "$tmp/raw" | sort -u > "$tmp/all"
count=$(wc -l < "$tmp/all")
[ "$count" -ge "$MIN_ENTRIES" ] || {
    echo "sciscope: refusing to install $count entries (< $MIN_ENTRIES)" >&2; exit 1; }

install_set() {          # $1 = live set, $2 = family, $3 = address file
    local live=$1 fam=$2 src=$3 tmpset="${1}_tmp"
    ipset create "$tmpset" hash:net family "$fam" maxelem 262144 -exist
    ipset flush  "$tmpset"
    # ipset restore takes the whole batch on stdin, far faster than one
    # `ipset add` per address, which forks thousands of times.
    { while read -r ip; do echo "add $tmpset $ip"; done < "$src"; } \
        | ipset restore -exist
    ipset swap    "$tmpset" "$live"   # <- the atomic moment
    ipset destroy "$tmpset"
}

grep -vF ':' "$tmp/all" > "$tmp/v4" || true
grep -F  ':' "$tmp/all" > "$tmp/v6" || true
[ -s "$tmp/v4" ] && install_set sciscope  inet  "$tmp/v4"
[ -s "$tmp/v6" ] && install_set sciscope6 inet6 "$tmp/v6"

echo "sciscope: installed $count entries"

ipset swap is the line that matters. Everything before it happens in a set nothing is matching against; everything after it is cleanup. At no point is the set referenced by the iptables rule empty or partial.

Note that the temporary set must be created with the same type and family as the live one, or the swap is rejected, which is a feature, not an annoyance.

Scheduling it

With systemd, use the timer from the nftables guide with the ExecStart pointed at this script. With cron, spread the load off the hour:

# /etc/cron.d/sciscope
17 * * * * root /usr/local/sbin/sciscope-ipset-update >/dev/null 2>&1
@reboot     root sleep 180 && /usr/local/sbin/sciscope-ipset-update >/dev/null 2>&1

The @reboot line is not optional. ipset contents do not survive a restart, and without it the blocklist is empty until the next hourly run.

Checking it

# entry count and memory use
sudo ipset list sciscope -terse

# is a specific address matched?
sudo ipset test sciscope 192.0.2.10

# how much is actually being dropped?
sudo iptables -L INPUT 1 -v -n --line-numbers

If the packet counter on that rule stays at zero for days, either nothing hostile is reaching you or the rule is sitting behind something that already accepted the traffic. Check the rule's position before concluding the feed is quiet.