Decentralized browser-based web crawler. Turn your browser into a voluntary crawl node that feeds the shared SIP-01 index on Nostr — the canonical Search Index Protocol v1.2. No backend. No tracking. No accounts required.
Every page you crawl becomes a kind 39697 web index observation — instantly searchable by 0xSearchstr, 0xPresearchstr, UNCAGED, and any future SIP-01 compatible client.
Live: https://crawlstr.shakespeare.wtf
You add a seed URL
│
▼
┌─────────────┐
│ Crawler │ IndexedDB queue, persistent across sessions
│ Engine │ Battery/WiFi/bandwidth aware
└──────┬──────┘
│
▼
Fetch page (respects robots.txt, rate-limited per domain)
│
▼
Parse HTML (title, description, text, links, language)
│
▼
SHA-256 content hash (dedup across the network)
│
▼
Sign kind 39697 event with per-device indexer key
│
▼
Publish to Nostr relays (SIP-01)
│
▼
┌──────────────────────────────────────┐
│ Shared SIP-01 Index │
│ │
│ 0xSearchstr reads it │
│ 0xPresearchstr reads it │
│ UNCAGED reads it │
│ Your fork reads it │
│ Any SIP-01 client reads it │
└──────────────────────────────────────┘
Most "decentralized search" projects still run centralized crawlers. Crawlstr makes every browser a potential crawler — opt-in, transparent, resource-aware.
| Feature | Description |
|---|---|
| Opt-in only | Nothing runs without explicitly pressing "Start Crawling" |
| 🎲 Random Scout | One button — picks a curated starting point you haven't scouted and goes |
| SIP-01 native | Same protocol as 0xSearchstr, 0xPresearchstr, UNCAGED — one shared index |
| Per-device identity | Anonymous indexer keypair, separate from your Nostr identity |
| No query leakage | Events contain page metadata only — never what anyone searched for |
| Resource aware | Battery, WiFi, bandwidth limits. Eco mode. Charging-only mode. |
| robots.txt | Respected by default (configurable) |
| Rate limited | 5–8 seconds between requests per domain |
| Persistent queue | IndexedDB-backed, survives browser restarts |
| Offline capable | Crawl queue persists; publishes when Nostr is reachable |
| Network heartbeat | Kind 16919 — visible on the SIP-01 dashboard while running |
| PWA | Installable, works on mobile and desktop |
Crawlstr is deliberately not a small copy of Indexstr. They're two classes of node:
THE INDEXSTR NETWORK
│
┌─────────────┴─────────────┐
│ │
🪶 CRAWLSTR 🏭 INDEXSTR
Lightweight scout Heavyweight indexer
"I found something" "I operate indexing capacity"
│ │
└─────────────┬─────────────┘
▼
NOSTR / SIP-01
│
┌─────────────┼─────────────┐
▼ ▼ ▼
0xSearchstr UNCAGED Other engines
| 🪶 Crawlstr (this) | 🏭 Indexstr | |
|---|---|---|
| Role | Human-directed & random discovery | Systematic distributed crawling |
| Seeds | You paste a URL, or Random Scout picks | Bundled curated SQLite collections |
| Queue | Small, session-scoped | Massive, sharded across 256 slots |
| Discovery | Links, feeds, sitemaps, canonical | Collection URLs at scale |
| Device | Any browser, incl. a phone on WiFi | Desktop/VPS-class contribution |
| Coupling | None. Both speak SIP-01 to Nostr. | None. Both speak SIP-01 to Nostr. |
Crawlstr finds. Indexstr maintains. Nostr distributes. Searchstr searches.
git clone https://github.com/NostrDanish/Crwalstr.git
cd Crwalstr
npm install
npm run devOpen the printed URL, add a seed URL, press Start Crawling.
The fastest way to contribute: press Explore a random corner of the web and Crawlstr shows you a starting point — its category and URL — before anything happens. Scout This starts the crawl; Another reshuffles. No account, no configuration, no URL needed.
Selection history stays in your browser (localStorage) and is never published — only the resulting page observations go to Nostr.
The corpus lives in src/data/seeds/*.txt — one category per plain-text file, so improving the dataset never means touching crawler logic:
src/data/seeds/
├── nostr.txt ← Nostr ecosystem
├── bitcoin.txt ← Bitcoin & open money
├── dev.txt ← Dev resources & awesome lists
├── science.txt ← Science & education
├── books.txt ← Books, archives & libraries
├── music.txt ← Music, art & culture
├── games.txt ← Games
├── opendata.txt ← Open data & internet infrastructure
└── indie.txt ← Indie blogs & the small web
Not plain Math.random() over a flat list. Each pick rolls a weighted strategy, tuned for long-tail coverage:
| Weight | Strategy | Picks from |
|---|---|---|
| 40% | Fresh | Seeds this device has never scouted |
| 25% | Rare | Seeds this device has scouted least often |
| 15% | Category | The category this device has explored least |
| 10% | Stale | Seeds not scouted for the longest time |
| 10% | Random | Pure randomness |
Previewing a seed (the "random corner" card) does not count as scouting it — a dismissed preview never penalizes the seed. The selection commits only when the crawl actually starts.
Enter any URL in the Seed URLs tab:
https://bitcoin.org
The crawler fetches the page, extracts content, hashes it, signs a SIP-01 observation, and publishes to Nostr. Then it follows links up to depth 3.
| Mode | Budget | Purpose |
|---|---|---|
| Quick Scan | ~5 pages | Fast inspection of one site |
| Site Scan | ~30 pages | A useful, complete crawl |
| Deep Scan | ~150 pages | Deeper crawl of a larger site |
| Volunteer | until stopped | Continuous contribution, every limit still applies |
When a mode's budget is spent, the crawler stops cleanly on its own. Random Explorer (continuous random scouting) is opt-in via startExplorer — never the default.
Crawlstr is a scout, so it prioritizes cheap, high-value discovery over bulk fetching:
- RSS/Atom feeds — one small XML file yields current content URLs with titles and dates; entries are indexed directly and queued for full fetches
- Sitemaps —
Sitemap:declarations in robots.txt, plus/sitemap.xml; sampled and bounded, never a firehose - Canonical URLs — observations are filed under the page's claimed canonical identity
- Link graph — same-domain links followed with per-domain rate limits; login/cart/calendar traps filtered out
| Setting | Default | Description |
|---|---|---|
| WiFi Only | Off | Only crawl on WiFi networks |
| Charging Only | Off | Only crawl while device is charging |
| Respect robots.txt | On | Follow website crawling policies |
| Eco Mode | On | Slower crawling, less resource usage |
| Follow RSS/Atom feeds | On | Index feed entries (cheap discovery) |
| Read sitemaps | On | Use sitemap.xml for discovery (sampled) |
Each browser gets its own anonymous indexer keypair (visible in the dashboard). This key signs all kind 39697 observations. It is:
- Pseudonymous — not linked to your personal Nostr identity
- Replaceable — regenerating creates a new indexer
- Local — the secret key never leaves your browser
- Exportable — for backup or migration
Crawlstr publishes SIP-01 (Search Index Protocol) events — the same protocol used by the entire Searchstr ecosystem.
{
"kind": 39697,
"pubkey": "<device indexer pubkey>",
"created_at": 1786250000,
"content": "{\"title\":\"Example Page\",\"description\":\"A page about...\"}",
"tags": [
["d", "widx:9f86d081884c7d659a2feaa0c55ad015"],
["u", "https://example.com/page"],
["l", "en"],
["x", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"],
["v", "1"],
["source", "crawlstr/1"],
["network", "clearnet"],
["type", "page"],
["alt", "Web index observation: Example Page"]
]
}| Tag | Meaning |
|---|---|
d |
"widx:" + sha256(normalized_url)[0:32] — URL identity, identical across all indexers |
u |
Canonical URL (normalized per SIP-01 §7) |
x |
Content hash: sha256(title + "\n" + description) |
v |
Schema version "1" |
l |
ISO 639-1 language code |
source |
"crawlstr/1" |
network |
Extension registry (§9.2) — always clearnet for a browser crawler |
type |
Extension registry — repository for GitHub/GitLab, else page |
alt |
Human-readable description (the alt convention, spec §12.3) |
Full schema documentation: NIP.md · Canonical spec: SIP-01 v1.1
Observations are published to:
SIP-01-aware index relays:
wss://relay-na1.metanomalist.com/wss://test-sip-relay.sip-01test.workers.dev/wss://sip-relay-2.sip-booster-relay.workers.dev/wss://sip-relay-3.uncaged-sip.workers.dev/wss://sip-relay-4.sip-relay-4.workers.dev/
Search relays (NIP-50):
wss://relay.nostr.band/wss://relay.ditto.pub/wss://search.nos.today/(read-only — "blocked: writes disabled")wss://relay.noswhere.com/
Write relays (for propagation):
wss://jskitty.cat/nostrwss://relay.primal.net/wss://relay.damus.io/wss://nostr.hifish.org/ws://acuy3mjnv26tkyaaucndlxmg2ocntz4rtebhavk57vgruozm42iaznqd.onion/(Tor only — reachable from Tor Browser; clearnet browsers skip it)
Each observation is pushed to every relay in the set via targeted per-relay connections.
Custom relays: add your own in Settings → Publish relays. They persist locally and merge into the publish set. The built-in set can't be removed.
Auto-discovery: Settings → Discover NIP-50 / SIP-01 relays queries relay-monitor announcements (NIP-66 kind 30166), filters to relays advertising NIP-50, then verifies each candidate with a live capability probe (NIP-11 — checking supported_nips for 50 and the uncaged_index block for SIP-01). Verified candidates appear with one-click add; nothing is added automatically. Discovery is a hint, never a requirement — the built-in set keeps the app fully functional without it (NIP-66's risk guidance).
Capability testing: every relay in the list has a test button that probes its NIP-11 document and shows SIP-01 / NIP-50 badges plus latency.
Crawlstr is one more independent indexer in the SIP-01 ecosystem:
Crawlstr (browser crawler)
│
▼ kind 39697, signed by device indexer key
Nostr Relays
│
├──→ 0xSearchstr (search engine) reads it
├──→ 0xPresearchstr (search engine) reads it
├──→ UNCAGED (search engine template) reads it
└──→ Any SIP-01 client reads it
Multiple crawlers observing the same URL produce events with the same d tag and different pubkeys — search nodes group by d and count distinct authors ("7 independent indexers saw this page").
Crawlstr is honest about what a browser can and cannot do:
- CORS — A browser cannot read cross-origin responses unless the site sends CORS headers, and most don't. Crawlstr tries a direct fetch first, then falls back to a CORS proxy so real websites can actually be crawled. The trade-off is honest: when the proxy is used, the proxy operator sees which URL was fetched (never a search query, never a user identity). The dashboard shows the direct/proxy split per session.
- JavaScript rendering — Crawlstr parses static HTML. Single-page apps that require JavaScript rendering won't have their full content extracted.
- Background execution — Mobile browsers may throttle or kill background tabs. The crawler is most effective when the tab is active.
- Rate limits — Per-domain rate limiting is built-in (5–8 seconds between requests). This is respectful by design.
- Resource budgets are real — the bandwidth cap counts every byte (pages, robots.txt, feeds, sitemaps, proxy overhead), and the pages/hour cap is enforced on a sliding window. The page-size cap is clamped to the remaining hourly budget so a single page can't overshoot it.
- SSRF guard at the proxy boundary — before any URL is handed to fetch (direct or proxied), it's checked against localhost, RFC1918, link-local (incl. cloud metadata
169.254.169.254), CGNAT, and the odd IPv4 forms browsers accept (integer, hex, octal, short), plus IPv6 loopback/ULA/link-local. Direct-path redirects are re-checked against the same list. The remaining gap — a public URL that 302s into private space at the proxy — is the proxy operator's to close, and we say so.
For unrestricted crawling, run a desktop/CLI SIP-01 crawler alongside Crawlstr.
- React 19 + TypeScript + Vite
- TailwindCSS 4 + shadcn/ui
- Nostrify — Nostr relay pool
- nostr-tools — Event signing (
finalizeEvent) - idb — IndexedDB wrapper for the crawl queue
- TanStack Query — Data fetching + caching
- PWA — Service worker, manifest, installable
src/
├── crawler/
│ ├── engine.ts ← Main crawler orchestrator (crawl loop, queue, scheduling)
│ ├── queue.ts ← IndexedDB queue (ready-job scheduling) + observed/fetched split
│ ├── fetcher.ts ← HTTP fetcher (CORS, timeout, SSRF-guarded, metered)
│ ├── safety.ts ← SSRF guard — refuses non-public targets at the proxy boundary
│ ├── meter.ts ← Sliding-window resource accounting (every byte, every page)
│ ├── parser.ts ← HTML parser (title, description, text, links, language)
│ ├── hasher.ts ← SHA-256 content hashing for local dedup
│ ├── webIndex.ts ← SIP-01: URL normalization, event build/parse (byte-compatible)
│ ├── indexerIdentity.ts ← Per-device anonymous indexer keypair
│ ├── publisher.ts ← Signs + publishes kind 39697 via finalizeEvent
│ ├── relays.ts ← Ecosystem relay pool configuration
│ ├── robots.ts ← robots.txt parser (policies + Sitemap: discovery)
│ ├── feed.ts ← RSS/Atom detection + parsing (cheap discovery)
│ ├── sitemap.ts ← XML sitemap parsing (urlset + sitemapindex, sampled)
│ ├── seeds.ts ← Random Scout selection engine (weighted strategies)
│ ├── limits.ts ← Per-domain rate limiting
│ └── types.ts ← TypeScript interfaces (incl. crawl modes)
├── data/
│ └── seeds/ ← The seed corpus: one plain-text file per category
├── components/
│ └── crawler/
│ └── CrawlerDashboard.tsx ← Main UI (toggle, stats, seed, history, settings)
├── hooks/
│ └── useCrawler.ts ← React hook wiring engine to Nostr
├── pages/
│ └── Index.tsx ← Landing page + dashboard
└── NIP.md ← Protocol documentation
| Project | Role | URL |
|---|---|---|
| Crawlstr (this) | Lightweight scout → SIP-01 publisher | crawlstr.shakespeare.wtf |
| Indexstr | Heavyweight distributed indexer w/ collections | — |
| 0xSearchstr | Search engine → SIP-01 reader | 0xsearchstr.shakespeare.wtf |
| 0xPresearchstr | Community fork with keyword staking | presearchstr.shakespeare.wtf |
| UNCAGED-ENGINE | Minimal search engine template | uncaged.shakespeare.wtf |
- No login required to crawl. No account. No tracking.
- Crawl observations are signed by a per-device anonymous keypair, never your personal Nostr identity.
- Events contain page metadata only — never search queries, never browsing history.
- Your crawl history stays in your browser (IndexedDB). Clearing browser data removes it.
- Relay operators see the observation event and your IP address — that's how Nostr works. Key separation is guaranteed; network anonymity is not.
- Use a VPN or Tor — we recommend NymVPN.
Support us: https://nym.com/pricing?ref=aYPKAFmGpJi
MIT
Vibed with Shakespeare