crawlerid

Is that really GPTBot? crawlerid identifies web crawlers and AI agents from their User-Agent, then checks the claim against each operator's official IP lists and DNS records.

Anyone can send User-Agent: Googlebot or ClaudeBot, and scrapers routinely do it to slip past rules written for the real crawlers. Every operator publishes a different way to check: a JSON list here, a reverse DNS suffix there, sometimes both, each on its own docs page. crawlerid keeps all of it in one registry where every entry links to its official source, and gives you one function to call.

• 47 crawlers from 12 operators: Google, Microsoft, Apple, OpenAI, Anthropic, Perplexity, Mistral, Common Crawl, Amazon, Meta, DuckDuckGo and Yandex.
• Real verification: official IP lists (bundled and refreshed weekly) plus forward-confirmed reverse DNS, the method Google and Bing document.
• Zero dependencies, edge-ready: Node.js, Bun, Deno, Cloudflare Workers and Vercel Edge. The core never imports Node built-ins.
• Fast on the hot path: human visitors are ruled out with a single scan of the header, with no I/O. verifySync() checks IP lists without touching the network.
• robots.txt by category: block AI training while staying in AI search, using each operator's exact documented tokens.
• CLI log audit: find spoofed crawlers in your existing access logs in one command.

npm install crawlerid

Quick start

import { createVerifier } from 'crawlerid';
import { nodeResolver } from 'crawlerid/node';

const verifier = createVerifier({ resolver: nodeResolver() }); // create once, reuse

const result = await verifier.verify({
  userAgent: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)',
  ip: '203.0.113.7',
});

result.status;          // 'failed': the IP is not in OpenAI's published GPTBot list
result.reason;          // 'ip-not-in-published-ranges'
result.crawler?.name;   // 'GPTBot'
result.crawler?.category; // 'ai-training'

Only need to know what a header claims?

import { identify } from 'crawlerid';

identify('Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)');
// → { id: 'bingbot', name: 'Bingbot', operator: 'Microsoft', category: 'search', … }

What the verdicts mean

status	Meaning	Typical action
verified	The IP is in the operator's published list, or forward-confirmed reverse DNS matched.	Trust the identity.
failed	Claims to be a known crawler, but the operator's data says the IP isn't theirs.	Almost always spoofed: block, rate-limit, or treat as an ordinary visitor.
unverifiable	A known crawler, but no check could be completed (see reason).	Don't grant special treatment based on the header alone.
unknown	Not a known crawler's User-Agent.	Ordinary traffic.

reason tells you why a request was not verified: ip-not-in-published-ranges, no-ptr-record, ptr-hostname-mismatch, forward-dns-mismatch (all failed), and no-official-method, no-range-data, dns-resolver-required, dns-error, invalid-ip (all unverifiable).

crawlerid fails safe: a DNS outage or timeout produces unverifiable, never failed. The Node resolver also checks that DNS is actually working before it trusts a "not found" answer.

Recipes

Use a client IP you can trust. Take it from the socket, or from a header your own platform sets (cf-connecting-ip on Cloudflare, x-real-ip on Vercel). Never trust a raw X-Forwarded-For that the client could have written.

Express

import express from 'express';
import { createVerifier } from 'crawlerid';
import { nodeResolver } from 'crawlerid/node';

const verifier = createVerifier({ resolver: nodeResolver() });
const app = express();
app.set('trust proxy', 'loopback'); // only if you run behind your own reverse proxy

app.use(async (req, res, next) => {
  const result = await verifier.verify({ userAgent: req.get('user-agent'), ip: req.ip });
  if (result.status === 'failed') return res.status(403).send('Forbidden');
  res.locals.crawler = result;
  next();
});

Next.js (proxy.ts, or middleware.ts before Next.js 16)

import { NextResponse, type NextRequest } from 'next/server';
import { verifySync } from 'crawlerid';

export function proxy(request: NextRequest) {
  const { status } = verifySync({
    userAgent: request.headers.get('user-agent'),
    ip: request.headers.get('x-real-ip'), // set by Vercel
  });
  if (status === 'failed') return new NextResponse('Forbidden', { status: 403 });
  return NextResponse.next();
}

Cloudflare Workers (and other edge runtimes)

There's no node:dns here, so reverse DNS goes over DNS-over-HTTPS:

import { createDohResolver, createVerifier } from 'crawlerid';

const verifier = createVerifier({ resolver: createDohResolver() });

export default {
  async fetch(request: Request): Promise<Response> {
    const result = await verifier.verify({
      userAgent: request.headers.get('user-agent'),
      ip: request.headers.get('cf-connecting-ip'),
    });
    if (result.status === 'failed') return new Response('Forbidden', { status: 403 });
    return fetch(request);
  },
};

Keep IP ranges fresh at runtime

The package ships with a snapshot of every official list, and each release refreshes it. Long-running services can also pull the live lists themselves:

import { bundledRanges, createVerifier, fetchRanges } from 'crawlerid';

let ranges = bundledRanges;
const verifier = createVerifier({ ranges });

async function refresh() {
  const { snapshot, errors } = await fetchRanges({ previous: ranges }); // failed sources keep old data
  ranges = snapshot;
  verifier.setRanges(snapshot);
  for (const e of errors) console.warn(`crawlerid: ${e.url}: ${e.error}`);
}
await refresh();
setInterval(refresh, 24 * 60 * 60 * 1000).unref();

robots.txt by category

import { robotsTxt } from 'crawlerid';

robotsTxt({
  categories: { 'ai-training': 'disallow' },
  crawlers: { 'Applebot-Extended': 'allow' }, // per-crawler overrides win
  sitemaps: ['https://example.com/sitemap.xml'],
});

# Generated by crawlerid 0.1.0 - https://github.com/daniel63194/crawlerid

# ai-training: Google, OpenAI, Anthropic, Mistral AI, Common Crawl, Amazon, Meta
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Amazonbot
User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Only blocked crawlers get a group. A crawler named in its own group stops reading your User-agent: * rules, so "allow" is expressed by leaving it out. Append the output to your existing robots.txt. Typos in names throw an error instead of being silently ignored.

Remember that robots.txt is a request, not an enforcement mechanism. Several user-triggered agents (ai-assistant, ai-agent) say they may ignore it. Pair it with verify() if you need enforcement.

CLI

npx crawlerid logs /var/log/nginx/access.log --live --dns

Example output:

48211 of 48211 lines parsed; 9120 requests claim to be a known crawler.

crawler             operator   category     requests  verified  FAILED  unverifiable
------------------  ---------  -----------  --------  --------  ------  ------------
Googlebot           Google     search       5310      5310      0       0
GPTBot              OpenAI     ai-training  1874      1702      172     0
Meta-ExternalAgent  Meta       ai-training  1201      0         0       1201
ClaudeBot           Anthropic  ai-training  735       735       0       0

Top IPs that failed verification (likely spoofed User-Agents):
  203.0.113.7                              172 requests

Command	What it does
crawlerid identify "<ua>"	Which crawler does this header claim to be?
crawlerid verify --ua "<ua>" --ip <ip>	Is it really that crawler? Exits 1 on failed.
crawlerid logs <file|->	Audit a combined-format access log (Nginx, Apache, Caddy).
crawlerid robots --disallow ai-training	Print robots.txt groups.
crawlerid list [--category ai-search]	List known crawlers.

Add --live to download fresh IP lists, --dns for reverse DNS checks, and --json for machine-readable output.

API

Export	Description
identify(userAgent)	The Crawler a User-Agent claims to be, or null. Whole-token, case-insensitive matching.
createVerifier(options?)	A Verifier with verify(), verifySync() and setRanges(). Caches reverse DNS results (10 000 entries, 6 h) and de-duplicates concurrent lookups.
verify(input, options?) / verifySync(input, options?)	One-off versions of the above.
fetchRanges(options?)	Downloads the official IP lists via fetch. Sources that fail keep their previous data.
bundledRanges	The snapshot shipped with this release. Check generatedAt to see its age.
createDohResolver(options?)	DNS-over-HTTPS resolver for edge runtimes (Cloudflare or Google JSON API).
nodeResolver(options?) from crawlerid/node	node:dns-backed resolver with timeouts and a broken-DNS safeguard.
robotsTxt(options)	robots.txt groups by category or crawler.
crawlers, getCrawler(idOrToken), crawlersInCategory(category)	The registry.
parseLogLine(line)	{ ip, userAgent } from a combined-format log line.
RangeSet, parseIp, parseCidr, extractCidrs	The dependency-free IPv4/IPv6 utilities used internally.

Everything is fully typed.

Not using JavaScript?

The registry and IP snapshot are plain JSON with a JSON Schema, so any language can use them:

• https://cdn.jsdelivr.net/npm/crawlerid/data/crawlers.json
• https://cdn.jsdelivr.net/npm/crawlerid/data/ip-ranges.json

Supported crawlers

47 crawlers from 12 operators. Each name links to the official source.

Crawler	Token	Operator	Category	Obeys robots.txt	Verified with
Googlebot	Googlebot	Google	search	yes	IP list + reverse DNS
Googlebot Image	Googlebot-Image	Google	search	yes	IP list + reverse DNS
Googlebot Video	Googlebot-Video	Google	search	yes	IP list + reverse DNS
Googlebot News	Googlebot-News	Google	search	yes	robots.txt token only
Google StoreBot	Storebot-Google	Google	search	yes	IP list + reverse DNS
Google-InspectionTool	Google-InspectionTool	Google	search	yes	IP list + reverse DNS
GoogleOther	GoogleOther	Google	other	yes	IP list + reverse DNS
GoogleOther-Image	GoogleOther-Image	Google	other	yes	IP list + reverse DNS
GoogleOther-Video	GoogleOther-Video	Google	other	yes	IP list + reverse DNS
Google-CloudVertexBot	Google-CloudVertexBot	Google	other	yes	IP list + reverse DNS
Google-Extended	Google-Extended	Google	ai-training	yes	robots.txt token only
AdsBot	AdsBot-Google	Google	ads	yes	IP list + reverse DNS
Mediapartners-Google	Mediapartners-Google	Google	ads	yes	IP list + reverse DNS
APIs-Google	APIs-Google	Google	other	yes	IP list + reverse DNS
Google-Safety	Google-Safety	Google	other	no	IP list + reverse DNS
Feedfetcher	FeedFetcher-Google	Google	fetcher	no	IP list + reverse DNS
Google Read Aloud	Google-Read-Aloud	Google	fetcher	no	IP list + reverse DNS
Google Site Verifier	Google-Site-Verification	Google	fetcher	no	IP list + reverse DNS
Gemini Notebook	Google-GeminiNotebook	Google	ai-assistant	no	IP list + reverse DNS
Google-Agent	Google-Agent	Google	ai-agent	no	IP list + reverse DNS
Bingbot	bingbot	Microsoft	search	yes	IP list + reverse DNS
Applebot	Applebot	Apple	search	yes	IP list + reverse DNS
Applebot-Extended	Applebot-Extended	Apple	ai-training	yes	robots.txt token only
GPTBot	GPTBot	OpenAI	ai-training	yes	IP list
OAI-SearchBot	OAI-SearchBot	OpenAI	ai-search	yes	IP list
ChatGPT-User	ChatGPT-User	OpenAI	ai-assistant	no	IP list
OAI-AdsBot	OAI-AdsBot	OpenAI	ads	?	IP list
ClaudeBot	ClaudeBot	Anthropic	ai-training	yes	IP list
Claude-SearchBot	Claude-SearchBot	Anthropic	ai-search	yes	IP list
Claude-User	Claude-User	Anthropic	ai-assistant	yes	IP list
PerplexityBot	PerplexityBot	Perplexity	ai-search	yes	IP list
Perplexity-User	Perplexity-User	Perplexity	ai-assistant	no	IP list
MistralAI-User	MistralAI-User	Mistral AI	ai-assistant	?	IP list
MistralAI-Index	MistralAI-Index	Mistral AI	ai-search	?	IP list
MistralAI-Training	MistralAI-Training	Mistral AI	ai-training	yes	none published
CCBot	CCBot	Common Crawl	ai-training	yes	IP list + reverse DNS
Amazonbot	Amazonbot	Amazon	ai-training	yes	none published
Amzn-SearchBot	Amzn-SearchBot	Amazon	ai-search	yes	none published
Amzn-User	Amzn-User	Amazon	ai-assistant	no	none published
Meta-ExternalAgent	meta-externalagent	Meta	ai-training	yes	none published
Meta-ExternalFetcher	meta-externalfetcher	Meta	ai-assistant	no	none published
Meta-WebIndexer	meta-webindexer	Meta	ai-search	yes	none published
Meta-ExternalAds	meta-externalads	Meta	ads	yes	none published
FacebookExternalHit	facebookexternalhit	Meta	preview	no	none published
DuckDuckBot	DuckDuckBot	DuckDuckGo	search	yes	none published
DuckAssistBot	DuckAssistBot	DuckDuckGo	ai-assistant	yes	none published
YandexBot	YandexBot	Yandex	search	yes	reverse DNS

"none published" means the operator offers no machine-readable list or DNS suffix yet, so requests can only be identified, not verified. Those are good places to contribute (see below).

How the data stays correct

1. Every registry entry links to the operator's own documentation (docs). Third-party bot lists are never used as a source.
2. A weekly workflow downloads every official IP list and opens a pull request when a prefix changes. Merged changes go out as patch releases.
3. Releases are blocked if the snapshot is empty, incomplete or older than 45 days. Packages are published from CI with npm provenance.

Contributing

Crawlers change constantly, and the registry is only as good as its upkeep. Help is very welcome, especially in these areas:

• New crawlers and corrections: open an issue or edit data/crawlers.json.
• Verification for operators that publish lists only as web pages (Amazon, DuckDuckGo) or only via their network registration (Meta's AS32934).
• Web Bot Auth (HTTP Message Signatures): cryptographic verification for agents that sign their requests.
• Framework adapters and ports: Fastify, Hono, Astro, Nuxt, plus Python and Go readers for the JSON data.

See CONTRIBUTING.md to get started. It takes about five minutes to set up.

Related projects

• isbot separates bots from humans using User-Agent heuristics. crawlerid answers a different question: is a self-declared crawler genuine? They work well together.
• ai.robots.txt is a community list of AI crawlers for blocking.

License

MIT