ThreatCull

Self-hosted threat-feed compiler. ThreatCull downloads public blocklists, drops duplicates, private ranges and your own infrastructure, scores each indicator by how many independent sources list it, and serves the result to your firewalls, DNS servers and SIEM.

sources ──> drop duplicates, private ranges, your allowlist ──> score by source agreement ──> feed URLs

• Catalog of public blocklist sources, plus your own sources (URL or local file)
• Built-in CDN allowlists, your own entries and your own allowlist URLs; mark your own network to get an alert when a source lists it
• Confidence tiers (high, medium, low) from source agreement
• Outputs as plain, hosts, AdGuard, RPZ, CSV or JSON, each at its own URL with a Feed Token
• Web UI with a dashboard, lookup, run history and a built-in scheduler
• Works offline and on a bare LAN IP

brag.mp4

Dashboard

Sources

Lookup

_{Click a screenshot to open it full size.}

Quick start

You need Docker with Compose. The image is published for amd64 and arm64.

curl -fsSLO https://raw.githubusercontent.com/spydisec/threatcull/main/docker-compose.yaml

Set THREATCULL_ADMIN_PASSWORD in docker-compose.yaml (12 or more characters), then:

docker compose up -d

Open http://<server-ip>:6969 and log in as admin. ThreatCull reads the password on the first start only; you can clear it from the file afterwards. Data lives on the threatcull-data volume.

Portainer: Stacks > Add stack, paste docker-compose.yaml, set the password, Deploy.

First steps

1. On Sources, enable the lists you want.
2. On Outputs, rotate a Feed Token to get the feed URL. Tokens show once.
3. Click Run now, or wait for the hourly schedule.
4. Point your firewall or DNS server at the feed URL:

curl http://<server-ip>:6969/o/ip-high/<feed-token>

Settings

Set these under environment: in docker-compose.yaml.

Variable	Default	What it does
THREATCULL_ADMIN_PASSWORD	none	Creates the admin user on the first start (12 or more characters)
THREATCULL_ADMIN_PASSWORD_FILE	none	Same, read from a file such as a Docker secret; wins over the above
TZ	UTC	Timezone for times in the web UI, as an IANA name such as Australia/Melbourne

ThreatCull stores and schedules everything in UTC, so you can change TZ at any time. Hover a time in the web UI to see its UTC value.

Sizing

What ThreatCull needs depends on how many lines your enabled Sources list, not on the platform. It runs the same on a single-board computer, a small VM or a large server; bigger lists take longer on slower processors and disks.

Measured with scripts/bench.py on an 8-core x86 machine with an SSD:

Lines (unique)	First Fetch	Unchanged re-Fetch	Compile	Peak memory	Database
500,000 IPs (300,000)	14 s	1 s	8 s	79 MB	45 MB
1,000,000 domains (600,000)	19 s	1 s	16 s	79 MB	96 MB
3,000,000 domains (1,800,000)	79 s	3 s	56 s	79 MB	287 MB

• Memory stays under about 100 MB for lists of any size: downloads go to a temporary file and are read line by line.
• Disk: plan on about 100 MB of database per million unique entries, plus the Output files. Settings and the dashboard show the current sizes.
• Time grows with the number of lines. A Source whose list hasn't changed is skipped in seconds, so only the first Fetch, and Fetches of lists that change, take long.

Run uv run python scripts/bench.py --lines 1000000 on your own host for numbers that apply to it. On a small host (1 to 2 GB of RAM, an SD card or a slow disk, as on a Raspberry Pi or a small VPS):

• start with the catalog's default Sources and add large lists one at a time;
• remove the # in front of mem_limit: 1g in docker-compose.yaml, so Docker stops ThreatCull before the host runs out of memory, and watch docker stats during the first Fetch of a large list;
• keep other busy containers off the same SD card if you can: database writes compete for it.

Upgrade

docker compose pull && docker compose up -d

The :2 tag follows 2.x releases. Pin a version such as :2.0.0 to upgrade by hand. Sources you disabled stay disabled.

Catalog only: to pick up new or corrected sources between releases, use Check for updates on the Sources page, or run threatcull catalog update. New sources arrive disabled. Without internet access, upload a newer catalog.yaml there, or run threatcull catalog update --file.

Back up and restore

docker compose run --rm -T threatcull config export > threatcull-config.yaml
docker compose run --rm -T threatcull config import - < threatcull-config.yaml

The file holds settings, source choices, the allowlist and outputs. It holds no passwords, tokens or threat data. Settings > Configuration in the web UI does the same. For a full backup, copy the threatcull-data volume.

Run without Docker

git clone https://github.com/spydisec/threatcull.git && cd threatcull
uv sync
uv run threatcull init
uv run threatcull user create admin
uv run threatcull serve --host <lan-ip> --port 6969

init prints each default output's Feed Token once. uv run threatcull --help lists every command.

Notes

• Offline networks: add a custom source with a file:// URL. Sources added in the web UI must point to files in data/imports/.
• Ranges: outputs list single IP addresses and domains. A source that lists a CIDR range counts on the dashboard as "Ranges left out", because one range can block many unrelated hosts. Ranges on the allowlist still cover every address inside them.
• Frozen sources: a blocklist source whose list has not gained or lost an entry for 30 days (frozen_after_days) shows as frozen on the Sources page and the dashboard. Its entries still count; check the feed and disable it if the publisher stopped updating it.
• Your network: add your public addresses to the allowlist with My network ticked (or run threatcull allow detect). ThreatCull keeps them out of every output and warns you when a source lists one.
• Scripts: create an API token with docker exec threatcull threatcull api-token create <name> --user admin and send it as Authorization: Bearer <token> to /api/v1/. docs/API.md lists every endpoint.
• Hardening: the image is built on Docker Hardened Images (no shell or package manager inside) and runs as a non-root user. The compose file drops all Linux capabilities and blocks privilege escalation. It also works with read_only: true and tmpfs: [/tmp]: SQLite writes its temporary files to /data, so they don't use RAM.
• Reverse proxy with TLS: start serve with --secure-cookies and --trusted-proxy <proxy-ip>.
• Scheduling: serve runs fetches and compiles itself. Don't also run threatcull run from cron on the same data directory.

Sources

ThreatCull ships no threat data. Each install downloads every source straight from its publisher, and you decide which sources to use. The Sources page links each one to its feed, so you can see who publishes it and follow their terms.

Development

make setup   # dependencies and git hooks
make check   # lint, types and security checks, as in CI

See CONTRIBUTING.md for issues and pull requests, SECURITY.md to report a vulnerability, and CHANGELOG.md for what changed in each release.

Licence

GNU AGPL-3.0