diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 1a9ec0d..380ca17 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -310,7 +310,8 @@ The loser is discarded, never archived. On a tie the existing version stays, so Merging only works if every source spells a URL the same way. The **key** is computed from a normalized URL; the **original** URL is kept for display. -- Lowercase scheme and host; remove default ports (`:80`, `:443`). +- Lowercase scheme and host — except the "hosts" of `ipfs://`, `ipns://` and `ar://`, which are content IDs, keys and Arweave transaction IDs, where base58 (`Qm…`, `12D3Koo…`) and base64 are case-sensitive; remove default ports (`:80`, `:443`; `ws://` and `wss://` too). +- Nostr URIs (NIP-21) have no `//`: `nostr:npub1…` is a URL whose host is what follows `nostr:` (bech32, lowercase; `nostr://npub1…` is read the same way), and is written back as `nostr:npub1…`. - `http`/`https` and `www.`/no `www.` are treated as the same page. - Remove tracking parameters (`utm_*`, `fbclid`, `gclid`, `mc_cid`, `mc_eid`, `ref`, …); keep all other query parameters as they are. - Remove a trailing slash from the path (except for the root `/`). @@ -793,8 +794,17 @@ lost/ ├── http.rs HTTP/1.1 and HTML, for the web networks ├── gemini.rs Gemini: TLS, responses, gemtext ├── gopher.rs Gopher: selectors, menus + ├── spartan.rs Spartan: Gemini without TLS + ├── nex.rs Nex: paths, directories + ├── finger.rs Finger: people's .plans ├── onion.rs .onion sites through Tor; Ahmia's banned list - └── i2p.rs I2P sites through a router; address books + ├── i2p.rs I2P sites through a router; address books + ├── ipfs.rs IPFS through a gateway; CIDs, one way to write them + ├── arweave.rs Arweave through a gateway; GraphQL listings, ArNS + ├── nostr.rs Nostr relays: listings, posts, profiles, NIP-19 + ├── ton.rs TON Sites through a TON proxy + ├── swarm.rs Swarm through a gateway; ENS names, bzz.link, CIDs + └── websocket.rs WebSocket (RFC 6455), for Nostr relays ``` ### 13.1 Library API (sketch) @@ -849,10 +859,10 @@ A crawler for the networks LOST can't get dumps of. It saves what it finds as a ``` lost-spider NETWORK [-o PATH] [-c] [--link URL]… [--links FILE] [--dont-crawl] [--search PHRASE]… [--searches FILE] [--blacklist FILE] [--delay SECONDS] - [--host-limit N] [--proxy URL] [-t SECONDS] + [--host-limit N] [--proxy URL] [--gateway URL] [-t SECONDS] ``` -`NETWORK` is `gemini` (§15.4), `gopher` (§15.5), `onion` (§15.6) or `i2p` (§15.7); the options are the same for each. +`NETWORK` is `gemini` (§15.4), `gopher` (§15.5), `onion` or `i2p` (§15.6), `ipfs` (§15.7), `arweave` (§15.8), `nostr` (§15.9), `ton` (§15.10), `swarm` (§15.11), or `spartan`, `nex` or `finger` (§15.12); the options are the same for each, but for `--gateway`, which is `ipfs`'s, `arweave`'s and `swarm`'s. | Option | What it does | |---|---| @@ -861,16 +871,17 @@ lost-spider NETWORK [-o PATH] [-c] [--link URL]… [--links FILE] [--dont-crawl] | `--sign` | Sign the file with your ID (§9.2), made if there's none yet; with `-c`, your signature replaces the file's, whoever's it was. Without it the file is unsigned (no ID is read or made), and with `-c` a signature it had is removed: the header loses its `signer`, the signed flag is cleared and no signature follows the footer (§9.4). Unsigned is the default because a signature says who made the file, and anonymity is one of LOST's points | | `--link URL`, `--links FILE` | Crawled first, even if nothing links to them (`example.org` means `gemini://example.org/`, or `gopher://…` for Gopher). A file has one a line; blank lines and `#` comments are skipped | | `--dont-crawl` | Fetch only the `--link` URLs (following their redirects) and none of the links on them: an easy way to add certain pages to a file, and nothing else. No seeds, and with `-c` nothing from the file's pages; a search box among them gets the `--search` phrases, and the results pages are saved. Needs `--link` or `--links` | -| `--search PHRASE`, `--searches FILE` | Typed into the search boxes found (§15.2) | -| `--blacklist FILE` | URLs and pages to skip, this run: the draft blocklist format of §9.2 (domains, URL beginnings, hosts-file lines, `/regex/` on the URL, `title:/regex/`, `body:/regex/`). Skipped pages aren't saved and their links aren't followed | -| `--delay SECONDS` | Between two requests to the same server; default 1, decimals allowed | +| `--search PHRASE`, `--searches FILE` | Typed into the search boxes found (§15.2); for `nostr`, searched for on the relays that can (§15.9) | +| `--blacklist FILE` | URLs and pages to skip, this run: the draft blocklist format of §9.2 (domains, URL beginnings, hosts-file lines, `/regex/` on the URL, `title:/regex/`, `body:/regex/`; `nostr:…` URIs are URL beginnings too, and for `nostr` an `npub1…`, `nostr:npub1…` or `nprofile1…` is an author whose posts are all skipped, §15.9). Skipped pages aren't saved and their links aren't followed | +| `--delay SECONDS` | Between two requests to the same server (for `nostr`, the same relay); default 1, decimals allowed | | `--host-limit N` | Pages saved per host at most, counting those already in the file; default: no limit | -| `--proxy URL` | Go through a proxy (`net.rs`): `socks5://HOST:PORT` for any network (Tor's is `socks5://127.0.0.1:9050`; host names are resolved by the proxy, so nothing reaches the local DNS, and Gemini and Gopher on `.onion` addresses become reachable), or `http://HOST:PORT` for I2P's HTTP proxy, which is sent whole requests. `onion` uses Tor's by default, `i2p` `http://127.0.0.1:4444`. Checked before anything else: nothing answering there stops the spider with a message (is Tor running?) | -| `-t`, `--timeout SECONDS` | How long to wait for a server to connect, or to send the next bit of an answer; a whole answer gets four times that. Defaults: Gopher 10 s, Gemini 15 s, either through a proxy 30 s (Tor adds a few seconds a hop), `.onion` 60 s, I2P 120 s (reaching a site for the first time can take a minute or more while the router builds tunnels) | +| `--proxy URL` | Go through a proxy (`net.rs`): `socks5://HOST:PORT` for any network (Tor's is `socks5://127.0.0.1:9050`; host names are resolved by the proxy, so nothing reaches the local DNS, and Gemini and Gopher on `.onion` addresses become reachable), or `http://HOST:PORT` for I2P's HTTP proxy, which is sent whole requests. `onion` uses Tor's by default, `i2p` `http://127.0.0.1:4444`, `ton` `http://127.0.0.1:8080`. Checked before anything else: nothing answering there stops the spider with a message (is Tor running?) | +| `--gateway URL` | `ipfs`, `arweave` and `swarm`: the gateway to ask, `http://HOST:PORT` or `https://HOST` (`arweave`: default `https://turbo-gateway.com`, §15.8; `swarm`: default `https://api.gateway.ethswarm.org`, or a Bee node's `http://127.0.0.1:1633`, §15.11; `ipfs`: default `http://127.0.0.1:8080`, Kubo's path gateway; `localhost` is asked as `127.0.0.1`, since Kubo's gateway there sends every request on to a subdomain). A gateway on this computer is asked about many sites at once; any other (a public gateway such as `https://ipfs.io`) counts as one server, asked one thing at a time (`--delay`). `--proxy` reaches a gateway elsewhere through Tor, say | +| `-t`, `--timeout SECONDS` | How long to wait for a server to connect, or to send the next bit of an answer; a whole answer gets four times that. Defaults: Gopher and Finger 10 s, Gemini, Spartan and Nex 15 s, any of those through a proxy 30 s (Tor adds a few seconds a hop), `.onion` 60 s, Nostr 20 s, Arweave 30 s, IPFS 60 s, TON 60 s, Swarm 60 s (content found on the network for the first time), I2P 120 s (reaching a site for the first time can take a minute or more while the router builds tunnels) | - Nothing given on the command line is stored, so `-c` needs the options again. - Unsigned ("Saving to NAME, unsigned."), or with `--sign` signed with your ID ("Saving to NAME, signed as 3f9a 1c2e 77b0 1d4e."). If `--sign` is given and the ID can't be read or made, the spider stops before crawling rather than leave the file unsigned. -- Progress goes to stderr (the pages counted are all the file's, with `-c` too): `Crawling Geminispace for 2 hours, 5 minutes, 9 seconds… 1,234 pages saved · 56 capsules · 7,890 links waiting` (the time is this run's) — `Scraping Geminispace…` with `--dont-crawl`, since it only fetches what it's given — redrawn every second on a terminal, a line a minute otherwise. +- Progress goes to stderr (the pages counted are all the file's, with `-c` too): `Crawling Geminispace for 2 hours, 5 minutes, 9 seconds… 1,234 pages saved · 56 capsules · 7,890 links waiting` (the time is this run's) — `Scraping Geminispace…` with `--dont-crawl`, since it only fetches what it's given — redrawn every second on a terminal, a line a minute otherwise. Once something fetched isn't saved, the line says how much (`· 40 skipped`, after `pages saved`), and the summary at the end says why, leaving out the reasons that didn't come up: `Not saved: 30 copies of pages saved already, 8 not found, 2 unreachable, 1 blacklisted.` (a copy is a page whose text is the same as one saved already; unreachable means no answer, or only errors, after trying again). - Ctrl-C (or SIGTERM, SIGHUP) stops: requests under way are abandoned, pages already received are written, and the file is finished (and signed, with `--sign`). A second Ctrl-C quits at once, leaving the file as of its last checkpoint. The last lines say how to continue and how to import. ### 15.2 Crawling (`crawl.rs`) @@ -938,6 +949,81 @@ Both are ordinary websites reached through a proxy, so they share an HTTP client Darknets in particular host material no one wants in a dataset: `--blacklist`'s `title:/…/` and `body:/…/` rules keep pages out by what they say, and their links aren't followed. +### 15.7 IPFS (`ipfs.rs`) + +Sites and files addressed by their content (`ipfs://CID/…`), and names that point to the latest version of something (`ipns://NAME/…`), fetched over HTTP from an IPFS node's gateway (`--gateway`, §15.1), which finds the content on the network and looks names up. With Kubo, that includes ENS names (`vitalik.eth`), resolved through eth.limo's DNS-over-HTTPS by default in current versions (`ipfs config --json DNS.Resolvers '{"eth.": "https://dns.eth.limo/dns-query"}'` in older ones). + +- **Requests:** `ipfs://CID/path?query` is `GET /ipfs/CID/path?query` on the gateway, `ipns://NAME/…` `GET /ipns/NAME/…`, through the HTTP client of §15.6. Its answers are read as there, but for a 5xx (not found on the network in time, or a name the node can't look up), which counts as the site being unreachable (§15.2's failures) — unless it says the path isn't in the content ("no link named …", a robots.txt asked of a file, say), which is "not found" — and redirects, which come back as `ipfs://` and `ipns://` URLs (the gateway's `/ipfs/CID/dir/` for a folder without its `/`). A folder without an `index.html` is the gateway's listing of it: a page of file names, whose links are followed. +- **One way to write an address.** Pages are saved, and links kept, as: + - `ipfs://` + the CID as CIDv1 in base32 (`bafy…`), whatever it was written as: CIDv0 (`Qm…`, base58), or CIDv1 in base32, base36, base58 or hex. Its bytes are checked: a CID's version, codec and multihash. + - `ipns://` + a key as CIDv1 in base36 (`k51…`), from an Ed25519 peer ID (`12D3Koo…`) or a key's CIDv1 in any base; or a domain, lowercase (DNSLink, `docs.ipfs.tech`; ENS, `vitalik.eth`). A CID given as an IPNS name is content (`ipfs://`); so is `Qm…`, which old RSA peer IDs looked like, but almost always is content. + - From gateways: a path (`https://ipfs.io/ipfs/CID/…`, any host, when what follows `/ipfs/` is a CID, or `/ipns/` a key or a domain), or a subdomain (`https://CID.ipfs.dweb.link/…`; `https://docs-ipfs-tech.ipns.dweb.link/…`, a DNSLink name made one label: `-` for `.`, `--` for `-`). `/ipfs/CID/…` on an IPFS page, as folder listings link, is that CID's. + - Queries (gateways' `?filename=`, `?format=`…) and fragments go. + - `--link` takes any of them: `--link docs.ipfs.tech` is `ipns://docs.ipfs.tech/`, `--link Qm…` is its `ipfs://bafy…/`. +- **Links:** those of §15.6's HTML, plus `ipfs://` and `ipns://` URLs written in the text. Only `ipfs://` and `ipns://` are crawled; the others (the web) are kept in `links`. +- **Servers:** every `ipfs://` CID and `ipns://` name is a host, with its own robots.txt (`/robots.txt` under it) and `--host-limit`. Through a gateway on this computer, each is a server of its own, so many are asked at once; through any other, they're all one server. +- Before crawling, the gateway must answer a connection, or the spider stops ("is your IPFS node running? (ipfs daemon)"). Then, since a node finds nothing until it's connected to other nodes (its gateway answers every request with a 504 after its own timeout, 30 s in Kubo), the first seed is asked for until something other than a 5xx comes back: every 10 seconds for up to 10 minutes, saying "Waiting for the IPFS node to find content (the gateway answered 504: …)…" with the gateway's reason, and giving up with it. +- Seeds: the IPFS project's sites (`docs.ipfs.tech`, `blog.ipfs.tech`, `specs.ipfs.tech`, `ipfs.tech`, all DNSLink) and an ENS site (`vitalik.eth`). More with `--links`. +- Default timeout 60 s: content not cached by the node is looked up on the network first. + +### 15.8 Arweave (`arweave.rs`) + +Data stored for good, each piece (a transaction, or a data item in a bundle) under its ID — 43 characters of base64url — fetched over HTTP from a gateway (`--gateway`; `https://turbo-gateway.com` by default, ar.io's; an ar.io node of your own works too). + +- **Addresses:** `ar://ID/path`, or `ar://NAME/path` for ArNS names (`ardrive`; undernames as `docs_ardrive`), lowercase. IDs keep their case (§6.5). Links to known gateways (arweave.net, ar.io, turbo-gateway.com, permagate.io, g8way.io, …) become `ar://` links: `/ID/…` or `/raw/ID` on the gateway or any subdomain of it (sandbox subdomains, which gateways use to give data its own origin), and `NAME.gateway/…`. Queries and fragments go. +- **Requests:** `ar://ID/` is `GET /ID`, `ar://ID/path` `GET /ID/path` (a path inside a manifest, a site's map of its files). A name is looked up first (`GET /ar-io/resolver/NAME`, its `txId`; once a crawl), and a name the gateway doesn't know is "not found". A redirect within the gateway (to its sandbox subdomain, or adding a `/`) is followed at once, up to 3; elsewhere, it's a link. 5xx counts as unreachable. Answers are read as in §15.6, and links are those of §15.6's HTML plus `ar://` URLs written in the text. +- **The listing.** A gateway's GraphQL endpoint (`POST /graphql`) knows every item stored, so the crawl also walks through those tagged `Content-Type` `text/html`, `text/plain`, `text/markdown` or `application/x.arweave-manifest+json` (sites), newest first (`sort: HEIGHT_DESC`), 100 at a time — **a range of blocks at a time** (`block: {min, max}`), since a gateway won't go through all of Arweave for one answer (turbo-gateway.com stops at 30 million rows read: "Limit for rows or bytes to read exceeded"). It starts at the newest block (`GET /info`'s `height`) with 20 blocks; a range the gateway answers with an error is asked for again in halves (a single block that still fails is skipped), and after a range with fewer than 50 items the next one is twice as big, up to 5,000 blocks. Each answer is a page that's followed, not saved, whose links are the items (`ar://ID/`, a `Title` or `Page-Title` tag as their text) and where the listing goes on: the same blocks' next page, `ar://graphql.ar/?max=BLOCK&size=BLOCKS&after=CURSOR`, or the blocks before them (no name or ID has a dot, so that host is no item's). Programs' messages to each other, which are most of what's newest and are tagged `text/plain` too, aren't items: AO's (any `Data-Protocol` tag) and SmartWeave's (an `App-Name` starting with `SmartWeave`) are dropped from each answer, since GraphQL can only ask for tags that are there. `ar://graphql.ar/` is the first seed, with `ar://ardrive/`. Nothing about it is stored: with `-c` it starts again from the newest, skipping what the file has. Before crawling, the gateway's newest block is asked for — one that can't be reached stops the spider — and then the newest blocks' listing: "Listing Arweave from block N, newest first.", or, if GraphQL says no, a line with its reason, and only links are followed. +- **Answers that are data** (the name resolver's and `/info`'s JSON) are read whatever their type (`Http::get_data`); pages, as in §15.6. +- **robots.txt** only for names (sites): an ID is one piece of data, mostly a single file, so it's crawled without asking (`Protocol::has_robots`), and so is the listing. +- **Servers:** a gateway on this computer is asked about many items at once; any other is one server, asked one thing at a time (`--delay`). +- Default timeout 30 s. + +### 15.9 Nostr (`nostr.rs`, `websocket.rs`) + +Nostr has no sites: posts ("events", NIP-01) are signed by their authors and kept by relays, servers anyone runs, reached over WebSocket (`wss://`). The same post is usually on many relays and none of them is its home, so a post is named by itself. + +- **Addresses** (NIP-19, NIP-21): `nostr:note1…` (a post, by its ID), `nostr:naddr1…` (an article or wiki entry: author, kind and name, so its newest version) and `nostr:npub1…` (a profile, by its key). These are what pages are saved under: `nevent1…` and `nprofile1…` (which add relays to look on) become `note1…` and `npub1…`, and an `naddr1…` loses its relays, so a post has one URL whoever linked to it and how. `nsec1…` (a secret key) is never taken as an address. Relays are `wss://HOST/PATH` (or `ws://`). +- **What's saved:** notes (kind 1) and comments (1111), titled "AUTHOR: its first 60 characters…", the text as it is, with mentions of people whose names are known written `@name`; notes with fewer than 10 letters and digits once links and mentions are left out ("gm", "🤙", a lone link) are followed, not saved. Long-form articles (30023, NIP-23): their `title`, their `summary` then the Markdown as text (images by their descriptions; links by their text, and kept as links). Wiki entries (30818, NIP-54): `title` or the entry's name, the AsciiDoc as text. Profiles (0): the name (`display_name`, else `name`), and the `about`, `name`, NIP-05 address and website as text. Reposts (6) lead to what they repost. Other kinds aren't pages. +- **Searching** (`--search`, NIP-50): with phrases to search for, four relays known to search are seeds too (search.nos.today, relay.nostr.band, relay.ditto.pub, relay.noswhere.com), and every relay's page asks it what it can do first (its NIP-11 document: `GET https://RELAY/`, `Accept: application/nostr+json`, once a relay): one that lists NIP 50 among its `supported_nips` gets a listing per phrase, `?kinds=0,1,30023,30818&search=PHRASE` (profiles too, then), 100 results at a time and paged like the others (`&until=TIME`); a relay that doesn't say it searches isn't asked (it might ignore `search` and send everything). A relay whose document says it needs logging in or paying (`limitation.auth_required`, `payment_required`) isn't listed at all. +- **Authors can be blacklisted:** an `npub1…`, `nostr:npub1…` or `nprofile1…` line in `--blacklist` skips that author's posts everywhere (listings, profiles, links to them): "Skipping the posts of N blacklisted authors." A `nostr:note1…` line skips that post, like any URL beginning. +- **Deletions** (NIP-09): a deletion request (kind 5) names posts by ID (`e`) or address (`a`); a post whose own author asked for it to be deleted isn't saved (anyone else's request doesn't count). Requests are read where they'd be: in the notes' listings (`?kinds=1,5`, so, since they're newer than what they delete, before the notes come up), with a profile (the author's last 200), and with any post looked up (`#e`/`#a` filters on the same request). A post kept from a listing is checked again when its turn comes. +- **Topics:** an article's or wiki entry's `t` tags are added to its text as `#hashtags`, those it doesn't have already. +- **Checked:** each post's ID must be the SHA-256 of what NIP-01 says to write out, and its signature (BIP-340 Schnorr, secp256k1; the `k256` crate) must be its author's, so no relay can make up a post in someone's name. One that fails isn't saved or followed. **Posts that ask to expire** (NIP-40, an `expiration` tag) aren't kept at all, whatever the date: they were meant to go away, and a `.lost` file is for good. +- **Links:** a post's author (their profile), what its tags point at (`p`: people; `e`, `q`: posts; `a`: articles), the `nostr:` references in its text (NIP-27) and its `http(s)://` links (an article's, from what's shown, not its images'). A profile's links are its website, its author's relays (their NIP-65 list, kind 10002: how the crawl gets from the seeds to the rest of the network) and their newest 50 notes and 50 articles or wiki entries. +- **The listings.** Each relay is listed, newest first, 200 posts at a time: `wss://RELAY/` is a page that's followed, not saved, whose links are its two listings, `?kinds=30023,30818` (articles and wiki entries, which would be lost among notes) and `?kinds=1,5` (notes, and deletion requests); each listing page's links are its posts and the page before it, `?kinds=…&until=TIME`: up to the oldest post's time, that second included, in case its posts didn't all fit — unless the page started at that very second, and then one second before it, so the listing always moves on. Seeds: relay.damus.io, nos.lol, relay.primal.net, relay.snort.social, nostr.mom, purplepag.es. With `-c` the listings start again from the newest, skipping what the file has. +- **Posts from listings are kept** (up to 50,000, the oldest let go first), so fetching them asks no relay; the authors of a listing page are asked for in one go, for their names. Anything else is looked for where it was last seen (a listing, a profile, a tag's relay hint, an `nevent`'s relays), then on two or three of the seeds: three relays at most before it's taken as gone. +- **Relays** get one connection each, kept open, and one subscription at a time on it (`REQ`, events, `EOSE`, `CLOSE`), `--delay` apart. A connection that fails is opened again once; a relay that can't be reached three times in a row isn't asked again in this crawl. `CLOSED` with `rate-limited:` waits a minute; with any other reason (`auth-required:`, NIP-42: nothing is logged into) that page is gone. To the crawl, every post, profile and relay is a server of its own (§15.2): the waiting is done per relay, here. +- **WebSocket:** RFC 6455's client side, as much as relays need: the HTTP upgrade (`101`), masked frames out, text messages in (in parts, too), pings answered; no compression. TLS accepts any certificate (§15.6): what's checked is the posts' signatures. `--proxy socks5://…` (Tor) works, and reaches `.onion` relays; an HTTP proxy doesn't. +- **No robots.txt:** relays are servers for programs, and posts aren't sites. +- Before crawling, the seeds are connected to at once: "Reached N of 6 relays." None at all stops the spider. Default timeout 20 s (a whole answer gets four times that). + +### 15.10 TON Sites (`ton.rs`) + +TON Sites are web sites on TON's network, served over RLDP (TON's own transport) and reached through a TON proxy, an HTTP proxy on this computer: `rldp-http-proxy` (`rldp-http-proxy -p 8080 -c 3333 -C global.config.json`) or Tonutils Proxy, at `http://127.0.0.1:8080` by default (`--proxy` for another; Kubo's gateway, §15.7, wants that port too). + +- **Addresses:** `http://NAME.ton/` (TON DNS names, subdomains too), `http://ADDRESS.adnl/` (a site by its ADNL address) and `http://BAG.bag/` (TON Storage). Only those are crawled: the proxy sends any other host to the ordinary web, so links to anything else (`https://`, gateways such as `NAME.ton.website`) aren't followed. +- **Seeds:** foundation.ton, storage.ton, howtorunsite.ton, tondnsdomain.ton. +- **Pages** as in §15.6 (HTML and text); robots.txt is obeyed. An answer of 500 or more is the proxy's (it couldn't reach the site): the site counts as unreachable. +- **Waiting for the proxy:** before crawling, foundation.ton is asked for until it answers, for up to 5 minutes ("Waiting for the TON proxy to be able to reach sites…"); a proxy that isn't running at all stops the spider at once (is the TON proxy running?). Default timeout 60 s. + +### 15.11 Swarm (`swarm.rs`) + +Swarm is Ethereum's storage network: a site is a manifest (its files by path) under the hash of its content, fetched over HTTP from a gateway, `GET /bzz/HASH/path`: `https://api.gateway.ethswarm.org` by default (the Swarm Foundation's API gateway; `gateway.ethswarm.org` answers every address with the same empty app page), or `--gateway` for another, such as a Bee node's API at `http://127.0.0.1:1633`. + +- **Addresses:** `bzz://HASH/path` (64 hex digits, lowercase) or `bzz://NAME.eth/path` (ENS names, which the gateway looks up). Gateways' links become `bzz://` ones: `/bzz/HASH/…` on bzz.link or gateway.ethswarm.org (and their subdomains), `NAME.bzz.link` (`NAME.eth`) and `CID.bzz.link`. CIDs (`bah5…`: base32, version 1, the Swarm manifest or feed codec, a Keccak-256 multihash) are written as their hashes, wherever they are. Links written for gateways, `/bzz/HASH/…` on a page, are read as `bzz://HASH/…`. `bzz://` URLs written in text are links too; queries and fragments go. +- **No listing:** nothing lists Swarm's sites, so the crawl only follows links, from the seeds: `bzz://swarm.eth/`, `bzz://fairdataprotocol.eth/` and a site the Swarm gateway guide links to by CID. `--link` adds others. +- **Public gateways serve only some sites:** the Swarm gateway's default rules refuse websites asked for by hash, unless allowed, by sending the visitor to `/forbidden` (ENS names and single files are let through). That redirect isn't followed: the page is gone (it shows as "not found" in the crawl's summary). Before crawling, the gateway is asked for swarm.eth — one that can't be reached stops the spider — and, if it isn't on this computer, a line says that a Bee node of your own reaches all of Swarm. +- **Redirects** the gateway makes to its own paths (a directory's `/` added) are followed, three at most; 500 or more is the gateway failing (unreachable). Pages as in §15.6; robots.txt is obeyed (a manifest can have one). +- **Servers:** a gateway on this computer is asked about many sites at once; any other is one server, asked one thing at a time (`--delay`). Default timeout 60 s (content not cached anywhere yet is found on the network first). + +### 15.12 Spartan, Nex and Finger (`spartan.rs`, `nex.rs`, `finger.rs`) + +Three of the small internet's protocols, each plain TCP, one request a connection, the answer read until the server closes it (`net::exchange`). `--proxy` takes a SOCKS5 proxy (Tor), as for Gopher. + +- **Spartan** (`spartan://`, port 300): Gemini's sibling without TLS. The request is `HOST PATH 0` and CRLF (no data sent); the answer is a one-digit status, a space, a meta line, CRLF: `2 TYPE` and the document (gemtext when no type is given; read as Gemini's, §15.4, plus `spartan://` URLs written in the text), `3 PATH` moved (followed), `4` gone, `5` the server failing (tried later). Seeds: mozz.us, spartan.mozz.us. robots.txt as for Gemini. +- **Nex** (`nex://`, port 1900): the request is the path and CRLF, the answer the file itself, with no status or type. A path ending in `/` is a directory, read as gemtext (its links are `=> URL label` lines); `.gmi` files are gemtext too; anything with a NUL byte isn't text (saved by its link); the rest is plain text. An empty answer is gone. Seed: nightfall.city. +- **Finger** (`finger://HOST/USER`, port 79, RFC 1288): the request is the user name (none for the server's own page) and CRLF, the answer text about them, their `.plan`. Pages are titled `user@host` (the host, for its own page); links are the URLs written in the text (`finger://`, `gemini://`, `gopher://`, `spartan://`, `nex://`, `http(s)://`: only `finger://` ones are followed). No robots.txt (it would be a user's name); names with `@` (forwarding to another server) aren't asked for. Seeds: plan.cat, happynetbox.com. Nothing lists a server's users, so the crawl goes where `.plan`s link to. + --- ## 16. Open questions diff --git a/Cargo.lock b/Cargo.lock index a4f2486..c980e4a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -495,6 +495,12 @@ dependencies = [ "tracing", ] +[[package]] +name = "base16ct" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd307490d624467aa6f74b0eabb77633d1f758a7b25f12bceb0b22e08d9726f6" + [[package]] name = "base64" version = "0.22.1" @@ -793,6 +799,12 @@ dependencies = [ "error-code", ] +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + [[package]] name = "codespan-reporting" version = "0.13.1" @@ -830,6 +842,12 @@ dependencies = [ "crossbeam-utils", ] +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + [[package]] name = "constant_time_eq" version = "0.4.2" @@ -876,6 +894,12 @@ dependencies = [ "libc", ] +[[package]] +name = "cpubits" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15b85f9c39137c3a891689859392b1bd49812121d0d61c9caf00d46ed5ce06ae" + [[package]] name = "cpufeatures" version = "0.3.1" @@ -934,6 +958,21 @@ version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" +[[package]] +name = "crypto-bigint" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a52aa3fcda4e6302a9f48734f234d35d4721b96f8fe07d073f07ce9df4f0271" +dependencies = [ + "cpubits", + "ctutils", + "hybrid-array", + "num-traits", + "rand_core", + "subtle", + "zeroize", +] + [[package]] name = "crypto-common" version = "0.2.2" @@ -941,6 +980,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" dependencies = [ "hybrid-array", + "rand_core", ] [[package]] @@ -975,6 +1015,16 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "ctutils" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03bb0e1cc970d482d121d9a1744999169b69a07470b3d644a7894e53fcaf4574" +dependencies = [ + "cmov", + "subtle", +] + [[package]] name = "cursor-icon" version = "1.2.0" @@ -1079,6 +1129,16 @@ dependencies = [ "thiserror 2.0.21", ] +[[package]] +name = "der" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a878c850e9e421b20262e9b41f9c860e4785fa07541c266b62ff9d1ef998a80a" +dependencies = [ + "const-oid", + "zeroize", +] + [[package]] name = "deranged" version = "0.5.8" @@ -1095,6 +1155,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" dependencies = [ "block-buffer", + "const-oid", "crypto-common", ] @@ -1190,6 +1251,19 @@ version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d8b14ccef22fc6f5a8f4d7d768562a182c04ce9a3b3157b91390b52ddfdf1a76" +[[package]] +name = "ecdsa" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0681a4fc24c767085329728d8dfba959af91228aa4610cca4f8ce317ba46ae0" +dependencies = [ + "der", + "digest", + "elliptic-curve", + "signature", + "zeroize", +] + [[package]] name = "ecolor" version = "0.36.2" @@ -1344,6 +1418,25 @@ version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e9c71c2167ca323c882b99918929403426e2373ea17242ff5653e0d5e1058be" +[[package]] +name = "elliptic-curve" +version = "0.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d65aa39b3a5c1c9c1b745c9a019234bb7a21b77abcb4f4d266d706e2d577d65" +dependencies = [ + "base16ct", + "crypto-bigint", + "crypto-common", + "digest", + "ff", + "group", + "hybrid-array", + "rand_core", + "sec1", + "subtle", + "zeroize", +] + [[package]] name = "emath" version = "0.36.2" @@ -1519,6 +1612,16 @@ version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b97b65636e5b9ef369943878ac74335ba1c55c1cb6adbf1e2c293c624248d693" +[[package]] +name = "ff" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1f686ab92a9fb0eaf188f6c6c87b89490baa6fdb0db4544ba4dc47f7942489f" +dependencies = [ + "rand_core", + "subtle", +] + [[package]] name = "fiat-crypto" version = "0.3.0" @@ -1998,6 +2101,17 @@ dependencies = [ "system-deps", ] +[[package]] +name = "group" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fd1a1c7a5206c5b7a3f5a0d7ccd3ff85d0c8f5133d62a02680255b0004af5f4" +dependencies = [ + "ff", + "rand_core", + "subtle", +] + [[package]] name = "gtk" version = "0.18.2" @@ -2195,7 +2309,9 @@ version = "0.4.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17" dependencies = [ + "subtle", "typenum", + "zeroize", ] [[package]] @@ -2545,6 +2661,21 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "k256" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93f50113171a713f4a4231ef82eb26703607139b35dcb56241f0ceab2ae1f7d8" +dependencies = [ + "cpubits", + "ecdsa", + "elliptic-curve", + "primeorder", + "sha2", + "signature", + "wnaf", +] + [[package]] name = "keyboard-types" version = "0.8.3" @@ -2719,6 +2850,7 @@ dependencies = [ "getrandom 0.4.3", "interprocess", "jiff", + "k256", "ksni", "libc", "redb", @@ -3608,6 +3740,31 @@ dependencies = [ "syn 3.0.8", ] +[[package]] +name = "primefield" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c555a6e4eb7d4e158fcb028c835c3b8642206ddc279b5c6b202ef9a8bdb592f4" +dependencies = [ + "crypto-bigint", + "crypto-common", + "ff", + "rand_core", + "subtle", + "zeroize", +] + +[[package]] +name = "primeorder" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c9f42978c78a00e3d68f69fc03e57a234debae69da4020a4fb588fcdcd07b06" +dependencies = [ + "elliptic-curve", + "primefield", + "wnaf", +] + [[package]] name = "proc-macro-crate" version = "1.3.1" @@ -3718,6 +3875,12 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + [[package]] name = "raw-window-handle" version = "0.6.2" @@ -4034,6 +4197,20 @@ dependencies = [ "tiny-skia", ] +[[package]] +name = "sec1" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56d437c2f19203ce5f7122e507831de96f3d2d4d3be5af44a0b0a09d8a80e4d" +dependencies = [ + "base16ct", + "ctutils", + "der", + "hybrid-array", + "subtle", + "zeroize", +] + [[package]] name = "self_cell" version = "1.3.0" @@ -4173,6 +4350,10 @@ name = "signature" version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "28d567dcbaf0049cb8ac2608a76cd95ff9e4412e1899d389ee400918ca7537f5" +dependencies = [ + "digest", + "rand_core", +] [[package]] name = "simd-adler32" @@ -5871,6 +6052,18 @@ version = "0.57.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" +[[package]] +name = "wnaf" +version = "0.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "795ca18b3fdb5e62bf982199278341ddcf7ebf7d32e25e212ad05d496e95f6fa" +dependencies = [ + "ff", + "group", + "hybrid-array", + "primefield", +] + [[package]] name = "writeable" version = "0.6.4" diff --git a/Cargo.toml b/Cargo.toml index efffad7..195e12a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -37,7 +37,7 @@ server = ["dep:axum", "dep:tokio", "dep:libc"] # egui desktop app and tray icon. gui = ["cli", "dep:eframe", "dep:rfd", "dep:ksni", "dep:winit", "dep:tray-icon", "dep:windows-sys"] # lost-spider. Not in the default set: only the spider goes online. -spider = ["dep:rustls", "dep:ctrlc", "dep:regex"] +spider = ["dep:rustls", "dep:ctrlc", "dep:regex", "dep:k256"] [dependencies] blake3 = "1" @@ -76,6 +76,8 @@ getrandom = "0.4" rustls = { version = "0.23", optional = true, default-features = false, features = ["ring", "std", "tls12"] } ctrlc = { version = "3.4", optional = true, features = ["termination"] } regex = { version = "1", optional = true } +# lost-spider nostr: checking posts' signatures (BIP-340 Schnorr, secp256k1). +k256 = { version = "0.14", optional = true, default-features = false, features = ["schnorr"] } [target.'cfg(unix)'.dependencies] libc = { version = "0.2", optional = true } diff --git a/README.md b/README.md index 7af33c5..a2c64c4 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ LOST searches through whatever datasets you import. It comes out of the box wit - Common Crawl (WET and WAT files) - Hacker News (BigQuery CSV export) - `.lost` files — LOST's own format -- Gemini, Gopher, `.onion` and I2P — crawled with `lost-spider` (see [below](#crawling-gemini-gopher-onion-and-i2p)) +- Gemini, Gopher, Spartan, Nex, Finger, `.onion`, I2P, IPFS, Arweave, Nostr, TON and Swarm — crawled with `lost-spider` (see [below](#crawling-the-small-internet-darknets-and-web3)) ### Datasets @@ -104,9 +104,9 @@ SELECT * FROM `bigquery-public-data.hacker_news.full` WHERE timestamp >= '2026-0 LOST is capable of importing and exporting `.lost` dataset files, which makes it possible to freely and safely exchange crawl data with other LOST users. They aren't signed unless you ask (`lost -e FILE --sign`, or the "Sign it" checkbox): a signature lets others tell which files are yours and keeps your database from getting corrupted or spammed, but it also says who made the file. Both exports and imports can be made either via CLI (see [Usage](#working-with-datasets)) or GUI (Settings → Data). -#### Crawling: Gemini, Gopher, .onion and I2P +#### Crawling: the small internet, darknets and web3 -LOST itself never goes online. For [Geminispace](https://geminiprotocol.net/), Gopherspace, `.onion` sites and I2P sites there's `lost-spider`, a separate crawler that isn't built or installed by default: +LOST itself never goes online. For [Geminispace](https://geminiprotocol.net/), Gopherspace, Spartan, Nex and Finger, `.onion` sites, I2P sites, [IPFS](https://ipfs.tech/), [Arweave](https://arweave.org/), [Nostr](https://nostr.com/), [TON Sites](https://docs.ton.org/foundations/web3/ton-sites) and [Swarm](https://www.ethswarm.org/) there's `lost-spider`, a separate crawler that isn't built or installed by default: ```console cargo install lost --features spider # or: make spider @@ -115,15 +115,27 @@ lost-spider gemini --link example.org -o my.lost # start at a capsule nothing lost-spider gemini --link example.org --dont-crawl # only that page, nothing else lost-spider gemini -c -o my.lost # carry on where it stopped lost-spider gopher # crawl all of Gopherspace +lost-spider spartan # Spartan capsules (Gemini without TLS) +lost-spider nex # Nex servers +lost-spider finger --link finger://plan.cat/someone # people's .plans lost-spider onion # .onion sites, through Tor (127.0.0.1:9050) lost-spider i2p # I2P sites, through an I2P router (127.0.0.1:4444) +lost-spider ipfs # IPFS sites, through an IPFS node's gateway (127.0.0.1:8080) +lost-spider ipfs --link vitalik.eth # an ENS name, or a CID, or an IPNS name +lost-spider arweave # the permaweb, through a public gateway +lost-spider nostr # Nostr: notes, articles and profiles, from relays +lost-spider nostr --link npub1… # someone's profile and posts first +lost-spider nostr --search "bitcoin mining" # and what relays find for that +lost-spider ton # TON Sites, through a TON proxy (127.0.0.1:8080) +lost-spider swarm # Swarm sites, through a public gateway +lost-spider swarm --gateway http://127.0.0.1:1633 # … or through your own Bee node lost-spider gemini --proxy socks5://127.0.0.1:9050 # Gemini through Tor lost -d my.lost # import what it found ``` It obeys robots.txt, asks each server for one thing a second (`--delay`), and saves everything into one `.lost` file, unsigned unless you ask for `--sign`. The file is complete even if you stop it with Ctrl-C. See `lost-spider --help` for search boxes, blacklists, proxies and limits. -`onion` needs [Tor](https://www.torproject.org/) running, and `i2p` an I2P router ([i2pd](https://i2pd.website/) or [Java I2P](https://geti2p.net/)). I2P is slow: reaching a site for the first time can take a minute or more, so its timeout is 120 seconds by default (`-t` changes it, for any network), and on a router (or Tor) that has just started, `lost-spider` waits a few minutes until it can reach sites. Images and other files are never downloaded, only their links. Before crawling `.onion` sites, `lost-spider` gets [Ahmia](https://ahmia.fi/)'s list of onion sites banned for child abuse material and skips them; it won't start without that list. Darknets host all sorts of things, so it's worth keeping a `--blacklist` with `title:/…/` and `body:/…/` rules for what you don't want in your dataset. +`onion` needs [Tor](https://www.torproject.org/) running, `ton` a [TON proxy](https://docs.ton.org/foundations/web3/ton-proxy) (`rldp-http-proxy -p 8080 -c 3333 -C global.config.json`, or Tonutils Proxy), `i2p` an I2P router ([i2pd](https://i2pd.website/) or [Java I2P](https://geti2p.net/)), and `ipfs` an IPFS node ([Kubo](https://github.com/ipfs/kubo), `ipfs daemon`), or `--gateway https://ipfs.io` to ask a public gateway instead (one request at a time). `arweave` needs nothing: it asks a public gateway (one request at a time), and also goes through everything Arweave stores, newest first, besides following links. Neither does `swarm`, but public Swarm gateways only serve the sites they allow, so a [Bee](https://docs.ethswarm.org/) node of your own (`--gateway http://127.0.0.1:1633`) reaches more; Swarm has no index, so it follows links only. `nostr` needs nothing either: it lists what well-known relays have, newest first, and goes from authors to their profiles and the relays they use; every post's signature is checked, and posts meant to expire, or that their authors asked to delete, aren't kept. With `--search`, relays that can search are asked for each phrase, and an `npub1…` line in a `--blacklist` skips that author's posts. I2P is slow: reaching a site for the first time can take a minute or more, so its timeout is 120 seconds by default (`-t` changes it, for any network), and on a router (or Tor) that has just started, `lost-spider` waits a few minutes until it can reach sites. Images and other files are never downloaded, only their links. Before crawling `.onion` sites, `lost-spider` gets [Ahmia](https://ahmia.fi/)'s list of onion sites banned for child abuse material and skips them; it won't start without that list. Darknets host all sorts of things, so it's worth keeping a `--blacklist` with `title:/…/` and `body:/…/` rules for what you don't want in your dataset. ### Database diff --git a/src/spider/arweave.rs b/src/spider/arweave.rs new file mode 100644 index 0000000..66242a8 --- /dev/null +++ b/src/spider/arweave.rs @@ -0,0 +1,915 @@ +//! Arweave, "the permaweb": data stored for good, each piece under its +//! transaction ID (43 characters of base64url), fetched over HTTP from a +//! gateway (`https://turbo-gateway.com` by default; `--gateway` names +//! another, a local ar.io node say). +//! +//! Besides links, Arweave can be listed: a gateway's GraphQL endpoint +//! answers for everything stored, so the crawl also walks through every +//! item tagged as HTML, text, Markdown or a site's manifest, newest first, +//! a hundred at a time, a range of blocks at a time (a gateway won't go +//! through all of Arweave for one answer). Those listings are pages that +//! are followed, not saved, under `ar://graphql.ar/?max=BLOCK&size=BLOCKS` +//! (no name or ID has a dot). +//! +//! Pages are saved under `ar://TXID/path`, or `ar://NAME/path` for ArNS +//! names (`ardrive`, undernames `docs_ardrive`), which the gateway looks up. +//! Links to gateways (`https://arweave.net/TXID`, `https://NAME.ar.io/`) +//! are written that way too (ARCHITECTURE.md §15.8). + +use std::collections::HashMap; +use std::net::{IpAddr, SocketAddr, ToSocketAddrs}; +use std::sync::Mutex; +use std::time::Duration; + +use lost::urls::Url; + +use crate::crawl::{Blacklist, Link, Parsed, Protocol, Response, Route, mentions, percent_decode}; +use crate::http::{self, Answer, Http}; +use crate::net::Proxy; + +pub const TIMEOUT: Duration = Duration::from_secs(30); +pub const GATEWAY: &str = "https://turbo-gateway.com"; + +/// The host of the listing pages. +const LISTING: &str = "graphql.ar"; +/// What the listing pages say they are. +const LISTING_TYPE: &str = "application/x-lost-spider-listing"; +/// Listed: pages, text, and sites (path manifests: a site's files by path). +const LISTED_TYPES: &[&str] = &[ + "text/html", + "text/plain", + "text/markdown", + "application/x.arweave-manifest+json", +]; +const LISTING_SIZE: usize = 100; +/// Blocks listed at a time, to begin with and at most; halved when the +/// gateway finds a range too much to go through, doubled when it's sparse. +const START_BLOCKS: u64 = 20; +const MAX_BLOCKS: u64 = 5_000; + +/// Where a crawl starts: the listing of everything (from the newest +/// blocks), and a well-known site. +const SEEDS: &[&str] = &["ar://graphql.ar/", "ar://ardrive/"]; + +/// Gateways, whose links are links to Arweave: the data's ID first in the +/// path, or a name as the subdomain. +const GATEWAYS: &[&str] = &[ + "arweave.net", + "arweave.dev", + "ar-io.net", + "ar-io.dev", + "ar.io", + "turbo-gateway.com", + "permagate.io", + "g8way.io", + "perma.online", +]; + +pub struct Arweave { + http: Http, + /// The gateway's scheme, host and port. + gateway: Url, + /// Where it is, for telling servers apart (none through a proxy). + address: Option, + /// It's on this computer: requests to it needn't wait for each other. + local: bool, + /// ArNS names looked up so far: their data's ID, or none. + names: Mutex>>, +} + +impl Arweave { + pub fn new( + gateway: Option, + proxy: Option, + timeout: Option, + ) -> lost::Result { + let text = gateway.unwrap_or_else(|| GATEWAY.to_string()); + let gateway = Url::parse(text.trim().trim_end_matches('/')) + .filter(|u| { + (u.scheme == "http" || u.scheme == "https") && u.path == "/" && u.query.is_none() + }) + .ok_or_else(|| { + format!("--gateway takes http://HOST:PORT or https://HOST (not \"{text}\")") + })?; + let ip = gateway + .host + .trim_start_matches('[') + .trim_end_matches(']') + .parse::() + .ok(); + let local = proxy.is_none() + && (gateway.host == "localhost" || ip.is_some_and(|ip| ip.is_loopback())); + let address = match proxy { + Some(_) => None, + None => Some(lookup(&gateway)?), + }; + Ok(Arweave { + http: Http::new(proxy, timeout.unwrap_or(TIMEOUT))?, + gateway, + address, + local, + names: Mutex::new(HashMap::new()), + }) + } + + /// Asks the gateway (or one of its subdomains) for `url`: a page, its + /// API's data (JSON), or a GraphQL `query`. + fn send(&self, url: &Url, ask: Ask) -> Result { + let route = if self.http.proxy().is_some() { + Route::Proxied + } else { + Route::Direct(lookup(url)?) + }; + match ask { + Ask::Page => self.http.get(route, url), + Ask::Data => self.http.get_data(route, url), + Ask::Query(body) => self.http.post(route, url, "application/json", body), + } + } + + /// The newest block's height. + fn height(&self) -> Result { + let answer = self.send(&self.at("/info".into()), Ask::Data)?; + if answer.status != 200 { + return Err(format!("the gateway's /info answered {}", answer.status)); + } + serde_json::from_slice::(&answer.body) + .ok() + .and_then(|v| v.get("height").and_then(|h| h.as_u64())) + .ok_or_else(|| "the gateway's /info has no height".to_string()) + } + + fn at(&self, path: String) -> Url { + let mut url = self.gateway.clone(); + url.path = path; + url + } + + /// The data an ArNS name points to, asked of the gateway once a crawl. + fn look_up(&self, name: &str) -> Result, String> { + if let Some(found) = self + .names + .lock() + .unwrap_or_else(|e| e.into_inner()) + .get(name) + { + return Ok(found.clone()); + } + let answer = self.send(&self.at(format!("/ar-io/resolver/{name}")), Ask::Data)?; + let found = match answer.status { + 200..=299 => serde_json::from_slice::(&answer.body) + .ok() + .and_then(|v| v.get("txId").and_then(|t| t.as_str()).map(str::to_string)) + .filter(|id| is_id(id)), + 400..=499 => None, + status => return Err(format!("can't look up {name} ({status})")), + }; + self.names + .lock() + .unwrap_or_else(|e| e.into_inner()) + .insert(name.to_string(), found.clone()); + Ok(found) + } + + /// One page of the listing: the items of `size` blocks up to `max`, + /// after `after`, as the gateway's GraphQL answer. The first, without a + /// range, goes on to the newest blocks. + fn listing(&self, url: &Url) -> Result { + let Some(range) = Range::of(url) else { + return Ok(Response::Redirect( + Range { + max: self.height()?, + size: START_BLOCKS, + after: None, + } + .url(), + )); + }; + let query = format!( + "query($min: Int, $max: Int, $after: String) {{ transactions(\ + tags: [{{name: \"Content-Type\", values: {}}}], block: {{min: $min, max: $max}}, \ + sort: HEIGHT_DESC, first: {LISTING_SIZE}, after: $after) {{ pageInfo {{ hasNextPage }} \ + edges {{ cursor node {{ id tags {{ name value }} }} }} }} }}", + serde_json::to_string(LISTED_TYPES).expect("strings") + ); + let body = serde_json::json!({ + "query": query, + "variables": { "min": range.min(), "max": range.max, "after": range.after }, + }); + let answer = self.send( + &self.at("/graphql".into()), + Ask::Query(body.to_string().as_bytes()), + )?; + match answer.status { + 200..=299 => Ok(Response::Document { + mime: LISTING_TYPE.into(), + body: Some(answer.body), + }), + 500.. => Err(format!("the gateway's GraphQL answered {}", answer.status)), + _ => Ok(Response::Gone), + } + } + + /// Where a redirect goes: within the gateway (to a sandbox subdomain + /// for the data's own origin, say) it's followed here; anything else + /// is a link, as an `ar://` one if it is one. + fn follow(&self, mut answer: Answer) -> Result { + for _ in 0..3 { + let location = match answer.status { + 301 | 302 | 303 | 307 | 308 => answer.header("location").map(str::to_string), + _ => None, + }; + let Some(location) = location else { + break; + }; + let target = if location.starts_with('/') { + let mut url = self.gateway.clone(); + let (path, query) = match location.split_once('?') { + Some((p, q)) => (p, Some(q.to_string())), + None => (location.as_str(), None), + }; + url.path = path.to_string(); + url.query = query; + Some(url) + } else { + Url::parse(&location) + }; + let Some(target) = target else { + return Ok(Response::Redirect(location)); + }; + let ours = target.host == self.gateway.host + || target + .host + .strip_suffix(self.gateway.host.as_str()) + .is_some_and(|sub| sub.ends_with('.')); + if !ours { + return Ok(Response::Redirect(canonical(&location).unwrap_or(location))); + } + answer = self.send(&target, Ask::Page)?; + } + if answer.status >= 500 { + return Err(format!("unreachable ({})", answer.status)); + } + Ok(match http::response(answer) { + Response::Redirect(location) => { + Response::Redirect(canonical(&location).unwrap_or(location)) + } + other => other, + }) + } +} + +/// The address of a host, looked up here. +fn lookup(url: &Url) -> Result { + let port = url + .port + .unwrap_or(if url.scheme == "https" { 443 } else { 80 }); + let host = url.host.trim_start_matches('[').trim_end_matches(']'); + (host, port) + .to_socket_addrs() + .ok() + .and_then(|mut a| a.next()) + .ok_or_else(|| format!("can't find {}", url.host)) +} + +impl Protocol for Arweave { + const NAME: &'static str = "arweave"; + /// `--link ardrive` is a name: `ar://ardrive/`. + const SCHEME: &'static str = "ar"; + const SPACE: &'static str = "the permaweb"; + const HOSTS: &'static str = "items"; + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + fn proxy(&self) -> Option<&Proxy> { + self.http.proxy() + } + + /// Through a gateway on this computer, every item is a server of its + /// own; through any other, they all are that one server, asked one + /// thing at a time (`--delay`). + fn proxied(&self) -> bool { + self.local + } + + fn resolve(&self, _url: &Url) -> Option { + Some( + self.address + .unwrap_or_else(|| SocketAddr::from(([0, 0, 0, 0], 0))), + ) + } + + fn accepts(url: &Url) -> bool { + url.scheme == "ar" && (url.host == LISTING || is_id(&url.host) || is_name(&url.host)) + } + + /// Only names are sites that can have a robots.txt; an ID is one piece + /// of data, mostly a single file, and the listing is the gateway's. + fn has_robots(origin: &Url) -> bool { + is_name(&origin.host) + } + + fn rewrite(link: &str) -> Option { + canonical(link) + } + + fn fetch(&self, _route: Route, url: &Url) -> Result { + if url.host == LISTING { + return self.listing(url); + } + let id = if is_id(&url.host) { + url.host.clone() + } else { + match self.look_up(&url.host)? { + Some(id) => id, + None => return Ok(Response::Gone), + } + }; + let path = if url.path == "/" { + format!("/{id}") + } else { + format!("/{id}{}", url.path) + }; + let answer = self.send(&self.at(path), Ask::Page)?; + self.follow(answer) + } + + fn parse(&self, url: &Url, mime: &str, body: &[u8]) -> Parsed { + if mime == LISTING_TYPE { + return match Range::of(url) { + Some(range) => listed(&range, body), + None => Parsed::default(), + }; + } + let mut parsed = http::parse(url, mime, body); + let written = mentions(&parsed.body, "ar"); + let mut links: Vec = Vec::new(); + for mut link in std::mem::take(&mut parsed.links).into_iter().chain(written) { + if let Some(url) = canonical(&link.url) { + link.url = url; + } + if !links.iter().any(|l| l.url == link.url) { + links.push(link); + } + } + parsed.links = links; + parsed + } + + /// The gateway must answer (its newest block), and says how listing goes. + fn prepare(&self, _blacklist: &mut Blacklist) -> lost::Result<()> { + let height = self.height().map_err(|e| { + format!( + "can't reach the Arweave gateway at {} ({e})", + self.gateway.host + ) + })?; + let first = Range { + max: height, + size: START_BLOCKS, + after: None, + }; + let probe = Url::parse(&first.url()).expect("a valid URL"); + let problem = match self.listing(&probe) { + Ok(Response::Document { + body: Some(body), .. + }) => graphql_error(&body), + Ok(_) => Some("no GraphQL".to_string()), + Err(e) => Some(e), + }; + match problem { + None => eprintln!("Listing Arweave from block {height}, newest first."), + Some(problem) => eprintln!( + "The gateway at {} can't list Arweave ({problem}): only links will be followed.", + self.gateway.host + ), + } + Ok(()) + } +} + +/// How the gateway is asked. +enum Ask<'a> { + Page, + Data, + Query(&'a [u8]), +} + +/// Where a listing page is: blocks `max - size + 1 ..= max`, after a cursor. +#[derive(Debug, Clone, PartialEq)] +struct Range { + max: u64, + size: u64, + after: Option, +} + +impl Range { + /// From `ar://graphql.ar/?max=…&size=…&after=…`; `None` for the start. + fn of(url: &Url) -> Option { + let query = url.query.as_deref()?; + let value = |name: &str| { + query + .split('&') + .find_map(|p| p.strip_prefix(name)?.strip_prefix('=')) + }; + Some(Range { + max: value("max")?.parse().ok()?, + size: value("size")?.parse::().ok()?.clamp(1, MAX_BLOCKS), + after: value("after").map(percent_decode), + }) + } + + fn min(&self) -> u64 { + (self.max + 1).saturating_sub(self.size).max(1) + } + + fn url(&self) -> String { + let mut url = format!("ar://{LISTING}/?max={}&size={}", self.max, self.size); + if let Some(after) = &self.after { + url.push_str(&format!("&after={}", percent_encode(after))); + } + url + } +} + +/// What a GraphQL answer's error says, if it is one. +fn graphql_error(body: &[u8]) -> Option { + let Ok(answer) = serde_json::from_slice::(body) else { + return Some("an answer that isn't JSON".into()); + }; + let errors = answer["errors"].as_array()?; + let first = errors.first()?; + let message = first["message"].as_str().unwrap_or("an error"); + Some(message.chars().take(200).collect()) +} + +/// A page of the listing: its items to crawl (each with its title, if it +/// has one), and where the listing goes on — the next page of this range, +/// or the blocks before it. A range the gateway found too much to go +/// through is asked for again in halves; one with few items is followed by +/// a range twice its size. Not saved. +/// Messages between programs, not pages: AO's (most of what's newest on +/// Arweave, as "text/plain") and SmartWeave's, by their tags. +fn is_message(tags: &[serde_json::Value]) -> bool { + tags.iter().any(|tag| { + let name = tag["name"].as_str().unwrap_or_default(); + let value = tag["value"].as_str().unwrap_or_default(); + name.eq_ignore_ascii_case("data-protocol") + || (name.eq_ignore_ascii_case("app-name") + && value.to_ascii_lowercase().starts_with("smartweave")) + }) +} + +fn listed(range: &Range, body: &[u8]) -> Parsed { + let mut parsed = Parsed { + skip: true, + ..Parsed::default() + }; + let next = |max: u64, size: u64, after: Option| Link { + url: Range { max, size, after }.url(), + label: None, + }; + if graphql_error(body).is_some() { + let link = if range.size > 1 { + next(range.max, range.size / 2, None) + } else if range.max > 1 { + // A single block too much: it's skipped. + next(range.max - 1, 1, None) + } else { + return parsed; + }; + parsed.links.push(link); + return parsed; + } + let Ok(answer) = serde_json::from_slice::(body) else { + return parsed; + }; + let transactions = &answer["data"]["transactions"]; + let edges = transactions["edges"] + .as_array() + .cloned() + .unwrap_or_default(); + let mut last = None; + for edge in &edges { + last = edge["cursor"].as_str().map(str::to_string); + let Some(id) = edge["node"]["id"].as_str().filter(|id| is_id(id)) else { + continue; + }; + let tags = edge["node"]["tags"].as_array(); + if tags.is_some_and(|tags| is_message(tags)) { + continue; + } + let title = tags.and_then(|tags| { + tags.iter().find_map(|tag| { + let name = tag["name"].as_str()?.to_ascii_lowercase(); + (name == "title" || name == "page-title") + .then(|| tag["value"].as_str().map(str::to_string)) + .flatten() + }) + }); + parsed.links.push(Link { + url: format!("ar://{id}/"), + label: title, + }); + } + let more = transactions["pageInfo"]["hasNextPage"].as_bool() == Some(true); + if more && let Some(cursor) = last { + parsed.links.push(next(range.max, range.size, Some(cursor))); + } else if range.min() > 1 { + let size = if edges.len() < LISTING_SIZE / 2 { + (range.size * 2).min(MAX_BLOCKS) + } else { + range.size + }; + parsed.links.push(next(range.min() - 1, size, None)); + } + parsed +} + +/// The address pages of Arweave are saved under, for a link to Arweave: +/// `ar://ID/…` or `ar://NAME/…` (names lowercase), or a gateway's link to +/// data (`https://arweave.net/ID/…`, `https://….arweave.net/ID/…`) or to a +/// name (`https://NAME.ar.io/…`). Queries and fragments go. `None` for +/// anything else, and for the listing's pages, which are left as they are. +pub fn canonical(text: &str) -> Option { + let text = text.trim(); + let (scheme, rest) = text.split_once("://")?; + let scheme = scheme.to_ascii_lowercase(); + let rest = rest.split(['#', '?']).next().unwrap_or(""); + let address = |host: &str, path: &str| -> Option { + let path = if path.is_empty() { "/" } else { path }; + if is_id(host) { + return Some(format!("ar://{host}{path}")); + } + let name = host.to_ascii_lowercase(); + is_name(&name).then(|| format!("ar://{name}{path}")) + }; + match scheme.as_str() { + "ar" => { + let end = rest.find('/').unwrap_or(rest.len()); + let (host, path) = rest.split_at(end); + address(host, path) + } + "http" | "https" => { + let url = Url::parse(text)?; + let gateway = GATEWAYS.iter().find(|g| { + url.host == **g + || url + .host + .strip_suffix(**g) + .is_some_and(|sub| sub.ends_with('.')) + })?; + // `/ID/…`, or `/raw/ID`: the data, whatever the subdomain (a + // sandbox one, for its own origin). + let path = url.path.strip_prefix("/raw").unwrap_or(&url.path); + let first = path.trim_start_matches('/').split('/').next().unwrap_or(""); + if is_id(first) { + return address(first, &path[1 + first.len()..]); + } + // `NAME.gateway`: a name. + let label = url.host.strip_suffix(*gateway)?.strip_suffix('.')?; + (!label.contains('.')) + .then(|| address(label, &url.path)) + .flatten() + } + _ => None, + } +} + +/// A transaction or data item ID: 32 bytes in base64url, 43 characters. +fn is_id(text: &str) -> bool { + text.len() == 43 + && text + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_') +} + +/// An ArNS name (or `undername_name`): lowercase letters, digits, `-` and +/// `_`, at most 51 characters (a gateway's 52-letter sandbox subdomains +/// aren't names). +fn is_name(text: &str) -> bool { + (1..=51).contains(&text.len()) + && text + .bytes() + .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-' || b == b'_') + && !text.starts_with('-') + && !text.ends_with('-') + && !is_id(text) +} + +fn percent_encode(text: &str) -> String { + text.bytes() + .map(|b| { + if b.is_ascii_alphanumeric() || b"-_.~".contains(&b) { + (b as char).to_string() + } else { + format!("%{b:02X}") + } + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{BufRead, BufReader, Read, Write}; + + const TX: &str = "bNbA3TEQVL60xlgCcqdz4ZPHFZ711cZ3hmkpGttDt_U"; + + fn url(text: &str) -> Url { + Url::parse(text).unwrap() + } + + #[test] + fn one_way_to_write_each_address() { + let c = |s: &str| canonical(s); + assert_eq!(c(&format!("ar://{TX}")), Some(format!("ar://{TX}/"))); + assert_eq!( + c(&format!("ar://{TX}/a.html?x#y")), + Some(format!("ar://{TX}/a.html")) + ); + assert_eq!(c("ar://ArDrive/about"), Some("ar://ardrive/about".into())); + assert_eq!(c("ar://docs_ardrive"), Some("ar://docs_ardrive/".into())); + assert_eq!( + c(&format!("https://arweave.net/{TX}")), + Some(format!("ar://{TX}/")) + ); + assert_eq!( + c(&format!("https://arweave.net/raw/{TX}")), + Some(format!("ar://{TX}/")) + ); + assert_eq!( + c(&format!( + "https://ugs6ntfppsulcv3l6ixq2huelmpsr2b56wxjnmtj6iwhfm6qvdyq.turbo-gateway.com/{TX}/b/c" + )), + Some(format!("ar://{TX}/b/c")) + ); + assert_eq!( + c("https://ardrive.ar.io/files"), + Some("ar://ardrive/files".into()) + ); + // Not Arweave. + assert_eq!(c(&format!("https://example.com/{TX}")), None); + assert_eq!(c("https://ar.io/"), None); + assert_eq!( + c("https://ugs6ntfppsulcv3l6ixq2huelmpsr2b56wxjnmtj6iwhfm6qvdyq.ar.io/"), + None + ); + assert_eq!(c("ar://graphql.ar/?after=x"), None); + assert_eq!(c("ar://Not A Name"), None); + assert!(Arweave::accepts(&url(&format!("ar://{TX}/")))); + assert!(Arweave::accepts(&url("ar://ardrive/"))); + assert!(Arweave::accepts(&url("ar://graphql.ar/?after=x"))); + assert!(!Arweave::accepts(&url("ar://some.other/"))); + assert!(Arweave::has_robots(&url("ar://ardrive/"))); + assert!(!Arweave::has_robots(&url(&format!("ar://{TX}/")))); + assert!(!Arweave::has_robots(&url("ar://graphql.ar/"))); + } + + #[test] + fn listings() { + let other = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"; + let ao = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; + let sw = "sssssssssssssssssssssssssssssssssssssssssss"; + let range = Range { + max: 1000, + size: 20, + after: None, + }; + // Items (not programs' messages), and the next page of the same blocks. + let body = format!( + "{{\"data\":{{\"transactions\":{{\"pageInfo\":{{\"hasNextPage\":true}},\"edges\":[\ + {{\"cursor\":\"c1\",\"node\":{{\"id\":\"{TX}\",\"tags\":[{{\"name\":\"Title\",\"value\":\"Hello\"}}]}}}},\ + {{\"cursor\":\"c2\",\"node\":{{\"id\":\"{ao}\",\"tags\":[{{\"name\":\"Data-Protocol\",\"value\":\"ao\"}}]}}}},\ + {{\"cursor\":\"c3\",\"node\":{{\"id\":\"{sw}\",\"tags\":[{{\"name\":\"App-Name\",\"value\":\"SmartWeaveAction\"}}]}}}},\ + {{\"cursor\":\"c2+/=\",\"node\":{{\"id\":\"{other}\",\"tags\":[]}}}}]}}}}}}" + ); + let parsed = listed(&range, body.as_bytes()); + assert!(parsed.skip); + assert_eq!( + parsed.links, + [ + Link { + url: format!("ar://{TX}/"), + label: Some("Hello".into()) + }, + Link { + url: format!("ar://{other}/"), + label: None + }, + Link { + url: "ar://graphql.ar/?max=1000&size=20&after=c2%2B%2F%3D".into(), + label: None + }, + ] + ); + // The last page of those blocks, sparse: the blocks before, twice as many. + let last = + "{\"data\":{\"transactions\":{\"pageInfo\":{\"hasNextPage\":false},\"edges\":[]}}}"; + let links: Vec = listed(&range, last.as_bytes()) + .links + .into_iter() + .map(|l| l.url) + .collect(); + assert_eq!(links, ["ar://graphql.ar/?max=980&size=40"]); + // Too much for the gateway: half as many blocks; one block, skipped. + let error = "{\"errors\":[{\"message\":\"Limit for rows or bytes to read exceeded\"}],\"data\":null}"; + let links: Vec = listed(&range, error.as_bytes()) + .links + .into_iter() + .map(|l| l.url) + .collect(); + assert_eq!(links, ["ar://graphql.ar/?max=1000&size=10"]); + let one = Range { + max: 1000, + size: 1, + after: None, + }; + let links: Vec = listed(&one, error.as_bytes()) + .links + .into_iter() + .map(|l| l.url) + .collect(); + assert_eq!(links, ["ar://graphql.ar/?max=999&size=1"]); + assert_eq!( + graphql_error(error.as_bytes()).as_deref(), + Some("Limit for rows or bytes to read exceeded") + ); + assert_eq!(graphql_error(last.as_bytes()), None); + } + + #[test] + fn ranges() { + let range = Range { + max: 1000, + size: 20, + after: Some("c2+/=".into()), + }; + assert_eq!(range.min(), 981); + let url = range.url(); + assert_eq!(url, "ar://graphql.ar/?max=1000&size=20&after=c2%2B%2F%3D"); + assert_eq!(Range::of(&Url::parse(&url).unwrap()), Some(range)); + assert_eq!(Range::of(&Url::parse("ar://graphql.ar/").unwrap()), None); + let start = Range { + max: 5, + size: 20, + after: None, + }; + assert_eq!(start.min(), 1); + assert_eq!( + Range::of(&Url::parse("ar://graphql.ar/?max=9&size=999999").unwrap()) + .unwrap() + .size, + MAX_BLOCKS + ); + } + + /// A gateway answering requests in turn with `answers`; returns the + /// requests (start line and body). + fn serve(answers: Vec) -> (SocketAddr, std::thread::JoinHandle>) { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let handle = std::thread::spawn(move || { + let mut requests = Vec::new(); + for answer in answers { + let (mut stream, _) = listener.accept().unwrap(); + let mut reader = BufReader::new(stream.try_clone().unwrap()); + let mut head = String::new(); + loop { + let mut line = String::new(); + if reader.read_line(&mut line).unwrap() == 0 || line == "\r\n" { + break; + } + head.push_str(&line); + } + let length: usize = head + .lines() + .find_map(|l| l.strip_prefix("Content-Length: ")) + .and_then(|n| n.trim().parse().ok()) + .unwrap_or(0); + let mut body = vec![0u8; length]; + reader.read_exact(&mut body).unwrap(); + let first = head.lines().next().unwrap_or("").to_string(); + requests.push(format!("{first} {}", String::from_utf8_lossy(&body))); + stream.write_all(answer.as_bytes()).unwrap(); + } + requests + }); + (addr, handle) + } + + #[test] + fn redirects() { + let page = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\nA"; + // Within the gateway: followed here. + let (addr, server) = serve(vec![ + format!("HTTP/1.1 302 Found\r\nLocation: /{TX}/about/\r\n\r\n"), + page.to_string(), + ]); + let gateway = Some(format!("http://{addr}")); + let arweave = Arweave::new(gateway, None, Some(Duration::from_secs(5))).unwrap(); + let response = arweave + .fetch(Route::Proxied, &url(&format!("ar://{TX}/about"))) + .unwrap(); + assert!( + matches!(&response, Response::Document { mime, .. } if mime == "text/html"), + "{response:?}" + ); + assert_eq!( + server.join().unwrap(), + [ + format!("GET /{TX}/about HTTP/1.1 "), + format!("GET /{TX}/about/ HTTP/1.1 "), + ] + ); + // Elsewhere: a link, written the one way. + let (addr, server) = serve(vec![format!( + "HTTP/1.1 301 Moved\r\nLocation: https://arweave.net/{TX}/x\r\n\r\n" + )]); + let gateway = Some(format!("http://{addr}")); + let arweave = Arweave::new(gateway, None, Some(Duration::from_secs(5))).unwrap(); + assert_eq!( + arweave.fetch(Route::Proxied, &url(&format!("ar://{TX}/"))), + Ok(Response::Redirect(format!("ar://{TX}/x"))) + ); + assert_eq!(server.join().unwrap(), [format!("GET /{TX} HTTP/1.1 ")]); + } + + /// Needs serde_json (the stand-in build here has none). + #[test] + fn through_the_gateway() { + let page = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\nA"; + // A name (looked up), then its page, which moved within the gateway. + let (addr, server) = serve(vec![ + format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n{{\"txId\":\"{TX}\"}}" + ), + format!("HTTP/1.1 302 Found\r\nLocation: /{TX}/about/\r\n\r\n"), + page.to_string(), + ]); + let arweave = Arweave::new( + Some(format!("http://{addr}")), + None, + Some(Duration::from_secs(5)), + ) + .unwrap(); + assert!( + arweave.proxied(), + "a gateway here: every item its own server" + ); + let response = arweave + .fetch(Route::Proxied, &url("ar://ardrive/about")) + .unwrap(); + assert!( + matches!(&response, Response::Document { mime, .. } if mime == "text/html"), + "{response:?}" + ); + let requests = server.join().unwrap(); + assert_eq!( + requests, + [ + "GET /ar-io/resolver/ardrive HTTP/1.1 ".to_string(), + format!("GET /{TX}/about HTTP/1.1 "), + format!("GET /{TX}/about/ HTTP/1.1 "), + ] + ); + + // The listing: a GraphQL query for the page after the cursor. + let (addr, server) = serve(vec![ + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n{\"data\":{}}".to_string(), + ]); + let arweave = Arweave::new( + Some(format!("http://{addr}")), + None, + Some(Duration::from_secs(5)), + ) + .unwrap(); + let response = arweave + .fetch( + Route::Proxied, + &url("ar://graphql.ar/?max=1000&size=20&after=c2%2B"), + ) + .unwrap(); + assert!(matches!(response, Response::Document { mime, .. } if mime == LISTING_TYPE)); + let requests = server.join().unwrap(); + assert!( + requests[0].starts_with("POST /graphql HTTP/1.1 {"), + "{requests:?}" + ); + assert!(requests[0].contains("\"after\":\"c2+\""), "{requests:?}"); + assert!(requests[0].contains("text/html"), "{requests:?}"); + assert!(requests[0].contains("\"min\":981"), "{requests:?}"); + assert!(requests[0].contains("\"max\":1000"), "{requests:?}"); + assert!( + requests[0].contains("block: {min: $min, max: $max}"), + "{requests:?}" + ); + + // A gateway elsewhere is one server for every item. + let remote = Arweave::new(Some("http://192.0.2.1:8080".into()), None, None).unwrap(); + assert!(!remote.proxied()); + assert!(Arweave::new(Some("ftp://x".into()), None, None).is_err()); + } +} diff --git a/src/spider/crawl.rs b/src/spider/crawl.rs index 0641039..698f6b6 100644 --- a/src/spider/crawl.rs +++ b/src/spider/crawl.rs @@ -90,9 +90,21 @@ pub trait Protocol: Send + Sync + 'static { url.scheme == Self::SCHEME } + /// The way pages are saved under for a link written another way (IPFS's + /// CIDs and gateway links, say); `None` to take it as it is. + fn rewrite(_link: &str) -> Option { + None + } + /// Where a host's robots.txt is. const ROBOTS: &'static str = "/robots.txt"; + /// Whether a host (its `/` URL) can have a robots.txt at all; one that + /// can't is crawled without asking (Arweave's single files, say). + fn has_robots(_origin: &Url) -> bool { + true + } + /// What robots.txt rules are matched against: the path and query. fn robots_path(url: &Url) -> String { match &url.query { @@ -164,6 +176,9 @@ pub fn port(url: &Url) -> u16 { url.port.unwrap_or(match url.scheme.as_str() { "gemini" => 1965, "gopher" => 70, + "spartan" => 300, + "nex" => 1900, + "finger" => 79, "https" => 443, _ => 80, }) @@ -220,6 +235,8 @@ pub struct Settings { /// title:/free bitcoin/ # … on the title /// body:/some phrase/ # … on the text /// 0.0.0.0 tracker.example # hosts-file lines work too +/// nostr:npub1… # a Nostr URI: what starts with it (and, +/// # for lost-spider nostr, an author's posts) /// ``` #[derive(Default)] pub struct Blacklist { @@ -270,7 +287,7 @@ impl Blacklist { let line = without_comment(line); let mut words = line.split_whitespace(); let Some(first) = words.next() else { continue }; - if first.contains("://") { + if first.contains("://") || first.to_ascii_lowercase().starts_with("nostr:") { // Written the way the spider writes URLs, so they compare. let prefix = match Url::parse(first) { Some(url) if first.ends_with('/') || url.path != "/" => url.to_string(), @@ -327,6 +344,16 @@ impl Blacklist { || self.urls.iter().any(|r| r.is_match(text)) } + /// What's listed by name: domains (and bare names) and URL beginnings, + /// as written down. A protocol can find its own kinds of names there + /// (Nostr's authors). + pub fn names(&self) -> impl Iterator { + self.domains + .iter() + .map(String::as_str) + .chain(self.prefixes.iter().map(String::as_str)) + } + fn blocks_page(&self, title: Option<&str>, body: &str) -> bool { title.is_some_and(|t| self.titles.iter().any(|r| r.is_match(t))) || self.bodies.iter().any(|r| r.is_match(body)) @@ -512,7 +539,7 @@ fn clean(mut url: Url) -> Url { /// What a search box is sent: everything but letters, digits and `-._~` /// percent-encoded. -fn encode_query(text: &str) -> String { +pub fn encode_query(text: &str) -> String { let mut out = String::with_capacity(text.len() * 3); for b in text.bytes() { if b.is_ascii_alphanumeric() || b"-._~".contains(&b) { @@ -595,17 +622,18 @@ pub fn looks_like_posting(text: &str) -> bool { /// name are two, each with its own robots.txt); punycode for an /// internationalized host, so both spellings are one host. pub fn host_key(url: &Url) -> String { - let host = lost::urls::ascii_host(&url.host).unwrap_or_else(|| url.host.clone()); + let host = lost::urls::wire_host(url).unwrap_or_else(|| url.host.clone()); match url.port { Some(port) => format!("{}://{host}:{port}", url.scheme), None => format!("{}://{host}", url.scheme), } } -/// The URL as it's sent: its host in punycode. +/// The URL as it's sent: its host in punycode (but an Arweave ID or an IPFS +/// CID as it is: lowercased, it's another one, or none). fn on_the_wire(url: &Url) -> Url { let mut url = url.clone(); - if let Some(ascii) = lost::urls::ascii_host(&url.host) { + if let Some(ascii) = lost::urls::wire_host(&url) { url.host = ascii; } url @@ -714,6 +742,7 @@ struct State { accepts: fn(&Url) -> bool, proxied: bool, robots_url: &'static str, + has_robots: fn(&Url) -> bool, robots_path: fn(&Url) -> String, is_file: fn(&Url) -> bool, untitled: fn(&Url) -> String, @@ -732,6 +761,7 @@ struct State { active: usize, /// Of which, host names being looked up. resolving: usize, + skipped: Skipped, } enum Task { @@ -767,6 +797,8 @@ enum Outcome { SlowDown(Duration), Retry, Nothing, + /// Not there (any more). + Gone, Failed, } @@ -801,7 +833,7 @@ impl State { hops: u8, settings: &Settings, ) -> bool { - if !(self.accepts)(&url) || lost::urls::ascii_host(&url.host).is_none() { + if !(self.accepts)(&url) || lost::urls::wire_host(&url).is_none() { return false; } let url = clean(url); @@ -836,23 +868,27 @@ impl State { } else { (None, String::new()) }; + let wants_robots = (self.has_robots)(&origin); if !done { - queue.push_back(Job { - url: robots, - label: None, - kind: Kind::Robots, - tries: 0, - hops: 0, - }); + if wants_robots { + queue.push_back(Job { + url: robots, + label: None, + kind: Kind::Robots, + tries: 0, + hops: 0, + }); + self.waiting += 1; + } if route.is_none() { self.unresolved.push_back(key.clone()); } - self.waiting += 1; } e.insert(Host { origin, queue, - robots: None, + // None to ask for: no rules. + robots: (!wants_robots).then(Robots::default), route, server, listed: false, @@ -1066,12 +1102,15 @@ impl State { .map_or(MAX_FAILURES, |host| host.failures); if failures >= MAX_FAILURES { self.retire(key); + self.skipped.unreachable += 1; } else if job.kind == Kind::Robots { // Asked again before anything else: nothing is fetched // there without its rules. self.requeue(key, job, true); } else if job.tries == 0 { self.requeue(key, Job { tries: 1, ..job }, false); + } else { + self.skipped.unreachable += 1; } } Outcome::Robots(robots) => { @@ -1086,6 +1125,11 @@ impl State { links, } => { let new = !blocked && content.is_none_or(|c| self.contents.insert(c)); + if blocked { + self.skipped.blocked += 1; + } else if !new { + self.skipped.copies += 1; + } if new { if let Some(host) = self.hosts.get_mut(key) { host.saved += 1; @@ -1163,6 +1207,7 @@ impl State { self.retire(key); } } + Outcome::Gone => self.skipped.gone += 1, Outcome::Nothing => {} } @@ -1179,6 +1224,26 @@ impl State { pub struct Stats { pub hosts: usize, pub waiting: u64, + pub skipped: Skipped, +} + +/// Pages fetched (or tried) but not saved, by why. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Skipped { + /// The same text as a page saved already. + pub copies: u64, + /// Not found, or not there any more. + pub gone: u64, + /// No answer, or errors only, after trying again. + pub unreachable: u64, + /// Kept out by the blacklist. + pub blocked: u64, +} + +impl Skipped { + pub fn total(&self) -> u64 { + self.copies + self.gone + self.unreachable + self.blocked + } } pub struct Crawler { @@ -1198,6 +1263,7 @@ impl Crawler

{ accepts: P::accepts, proxied: protocol.proxied(), robots_url: P::ROBOTS, + has_robots: P::has_robots, robots_path: P::robots_path, is_file: P::is_file, untitled: P::fallback_title, @@ -1212,6 +1278,7 @@ impl Crawler

{ waiting: 0, active: 0, resolving: 0, + skipped: Skipped::default(), }; // With --dont-crawl, the links are all there is to fetch: no seeds, // and nothing from the file's pages. @@ -1268,6 +1335,7 @@ impl Crawler

{ Stats { hosts: state.hosts.len(), waiting: state.waiting, + skipped: state.skipped, } } @@ -1436,7 +1504,7 @@ impl Crawler

{ } Response::SlowDown(wait) => Outcome::SlowDown(wait), Response::TryLater => Outcome::Retry, - Response::Gone => Outcome::Nothing, + Response::Gone => Outcome::Gone, } } } @@ -1632,6 +1700,10 @@ mod tests { } } + fn has_robots(origin: &Url) -> bool { + !origin.host.starts_with("norobots") + } + fn resolve(&self, url: &Url) -> Option { assert!(!self.proxied, "nothing is looked up through a proxy"); // Every *.test host lives on one server, except nowhere.test. @@ -1697,6 +1769,63 @@ mod tests { } } + #[test] + fn ids_are_sent_as_they_are() { + // An Arweave ID lowercased is another ID (one that isn't there). + let tx = "bNbA3TEQVL60xlgCcqdz4ZPHFZ711cZ3hmkpGttDt_U"; + let url = Url::parse(&format!("ar://{tx}/")).unwrap(); + assert_eq!(on_the_wire(&url).host, tx); + assert_eq!(host_key(&url), format!("ar://{tx}")); + let other = Url::parse(&format!("ar://{}/", tx.to_lowercase())).unwrap(); + assert_ne!(seen_key(&url), seen_key(&other)); + // Other hosts still go out lowercase, in punycode. + let mut url = Url::parse("gemini://café.example/").unwrap(); + url.host = "Café.Example".into(); + assert_eq!(on_the_wire(&url).host, "xn--caf-dma.example"); + } + + #[test] + fn what_wasnt_saved_and_why() { + let fake = Fake { + pages: [ + ( + "gemini://a.test/", + doc("Home =>/copy =>/missing =>gemini://down.test/"), + ), + ( + "gemini://a.test/copy", + doc("Home =>/copy =>/missing =>gemini://down.test/"), + ), + ("gemini://a.test/robots.txt", doc("")), + ] + .into_iter() + .map(|(u, r)| (u.to_string(), r)) + .collect(), + asked: Mutex::new(Vec::new()), + proxied: false, + }; + let crawler = Crawler::new( + fake, + settings(), + Known::default(), + &["gemini://a.test/".into()], + ); + let (tx, rx) = std::sync::mpsc::sync_channel(100); + crawler.start(tx); + assert_eq!(rx.iter().count(), 1); + let skipped = crawler.stats().skipped; + assert_eq!( + skipped, + Skipped { + copies: 1, + gone: 1, + unreachable: 1, + blocked: 0, + } + ); + assert_eq!(skipped.total(), 3); + } + #[test] fn a_small_crawl() { let pages = vec![ @@ -2016,4 +2145,32 @@ mod tests { <= MAX_FAILURES as usize ); } + + #[test] + fn hosts_without_robots_txt() { + // Not asked for a robots.txt it can't have; crawled all the same, + // looked up or through a proxy. + for proxied in [false, true] { + let pages = vec![ + ("gemini://norobots.test/", doc("Home =>/a")), + ("gemini://norobots.test/a", doc("A")), + ]; + let (saved, asked) = crawl_via( + proxied, + pages, + settings(), + Known::default(), + &["gemini://norobots.test/"], + ); + let urls: Vec<&str> = saved.iter().map(|p| p.url.as_str()).collect(); + assert_eq!( + urls, + ["gemini://norobots.test/", "gemini://norobots.test/a"] + ); + assert!( + !asked.iter().any(|u| u.ends_with("robots.txt")), + "{asked:?}" + ); + } + } } diff --git a/src/spider/finger.rs b/src/spider/finger.rs new file mode 100644 index 0000000..e46a367 --- /dev/null +++ b/src/spider/finger.rs @@ -0,0 +1,189 @@ +//! Finger (RFC 1288; its URLs: `finger://HOST/USER`): plain TCP on port +//! 79. A request is a user name (none: the server's own page) and CRLF; +//! the answer is text about them, their `.plan`, until the server closes +//! the connection. Links are the URLs written in it (ARCHITECTURE.md +//! §15.12). + +use std::time::Duration; + +use lost::urls::Url; + +use crate::crawl::{Parsed, Protocol, Response, Route, mentions, percent_decode, port}; +use crate::net::{self, Proxy}; + +/// Where a crawl starts: servers that host people's `.plan`s. +const SEEDS: &[&str] = &["finger://plan.cat/", "finger://happynetbox.com/"]; + +pub const TIMEOUT: Duration = Duration::from_secs(10); +/// Through a proxy (Tor adds a few seconds a hop). +pub const PROXIED_TIMEOUT: Duration = Duration::from_secs(30); +/// What's written in a `.plan` and kept as a link (only `finger://` ones +/// are followed). +const SCHEMES: &[&str] = &[ + "finger", "gemini", "gopher", "spartan", "nex", "https", "http", +]; + +pub struct Finger { + /// A SOCKS5 proxy (Finger can't go through an HTTP one). + proxy: Option, + timeout: Duration, +} + +impl Finger { + pub fn new(proxy: Option, timeout: Option) -> lost::Result { + if let Some(Proxy::Http(_)) = proxy { + return Err("Finger can only go through a SOCKS5 proxy (socks5://HOST:PORT)".into()); + } + let timeout = timeout.unwrap_or(match proxy { + Some(_) => PROXIED_TIMEOUT, + None => TIMEOUT, + }); + Ok(Finger { proxy, timeout }) + } +} + +/// The user a URL asks about: its path's first part, decoded. +fn user(url: &Url) -> String { + percent_decode( + url.path + .trim_start_matches('/') + .split('/') + .next() + .unwrap_or(""), + ) +} + +impl Protocol for Finger { + const NAME: &'static str = "finger"; + const SCHEME: &'static str = "finger"; + const SPACE: &'static str = "Fingerspace"; + const HOSTS: &'static str = "servers"; + + fn proxy(&self) -> Option<&Proxy> { + self.proxy.as_ref() + } + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + /// No such thing: a robots.txt would be a user called that. + fn has_robots(_origin: &Url) -> bool { + false + } + + /// `user@host`, or the host for its own page. + fn fallback_title(url: &Url) -> String { + match user(url) { + user if user.is_empty() => url.host.clone(), + user => format!("{user}@{}", url.host), + } + } + + fn fetch(&self, route: Route, url: &Url) -> Result { + let user = user(url); + // Only a name: no forwarding (`user@other.host`) or odd bytes. + if user.contains(['@', '\r', '\n', '\t', ' ']) { + return Ok(Response::Gone); + } + let answer = net::exchange( + route, + self.proxy.as_ref(), + &url.host, + port(url), + &format!("{user}\r\n"), + self.timeout, + lost::MAX_BODY_BYTES, + )?; + if answer.iter().all(u8::is_ascii_whitespace) { + return Ok(Response::Gone); + } + Ok(Response::Document { + mime: "text/plain".into(), + body: Some(answer), + }) + } + + fn parse(&self, url: &Url, _mime: &str, body: &[u8]) -> Parsed { + let text = String::from_utf8_lossy(body).replace('\r', ""); + let mut links = Vec::new(); + for scheme in SCHEMES { + for link in mentions(&text, scheme) { + if !links.iter().any(|l: &crate::crawl::Link| l.url == link.url) { + links.push(link); + } + } + } + Parsed { + title: Some(Self::fallback_title(url)), + body: text, + links, + skip: false, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn plans() { + use std::io::{Read, Write}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = std::thread::spawn(move || { + let mut asked = Vec::new(); + for answer in [ + &b"Login: alice\r\nPlan:\r\nWorking on finger://plan.cat/bob and https://example.org/.\r\n"[..], + b"\r\n", + ] { + let (mut s, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n") { + s.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + asked.push(String::from_utf8(request).unwrap()); + s.write_all(answer).unwrap(); + } + asked + }); + let finger = Finger::new(None, Some(Duration::from_secs(5))).unwrap(); + let address = std::net::SocketAddr::from(([127, 0, 0, 1], port)); + let url = Url::parse(&format!("finger://127.0.0.1:{port}/alice")).unwrap(); + let Ok(Response::Document { mime, body }) = finger.fetch(Route::Direct(address), &url) + else { + panic!("a plan"); + }; + let page = finger.parse(&url, &mime, &body.unwrap()); + assert_eq!(page.title, Some("alice@127.0.0.1".to_string())); + assert_eq!( + page.body, + "Login: alice\nPlan:\nWorking on finger://plan.cat/bob and https://example.org/.\n" + ); + let links: Vec<&str> = page.links.iter().map(|l| l.url.as_str()).collect(); + assert_eq!(links, ["finger://plan.cat/bob", "https://example.org/"]); + // Nothing to say: gone. + let url = Url::parse(&format!("finger://127.0.0.1:{port}/")).unwrap(); + assert_eq!( + finger.fetch(Route::Direct(address), &url), + Ok(Response::Gone) + ); + assert_eq!(server.join().unwrap(), ["alice\r\n", "\r\n"]); + // No forwarding to other hosts. + let url = Url::parse("finger://plan.cat/bob@elsewhere.example").unwrap(); + assert_eq!( + finger.fetch(Route::Direct(address), &url), + Ok(Response::Gone) + ); + assert_eq!( + Finger::fallback_title(&Url::parse("finger://plan.cat/").unwrap()), + "plan.cat" + ); + assert!(!Finger::has_robots( + &Url::parse("finger://plan.cat/").unwrap() + )); + } +} diff --git a/src/spider/gemini.rs b/src/spider/gemini.rs index 1974fe1..962c0ef 100644 --- a/src/spider/gemini.rs +++ b/src/spider/gemini.rs @@ -215,7 +215,7 @@ fn parse_status(line: &str) -> Option<(u8, &str)> { } /// `text/gemini; charset=utf-8; lang=en` → `text/gemini`. -fn mime_type(mime: &str) -> String { +pub(crate) fn mime_type(mime: &str) -> String { mime.split(';') .next() .unwrap_or("") @@ -227,7 +227,7 @@ fn is_text(mime: &str) -> bool { mime_type(mime).starts_with("text/") } -fn charset(mime: &str) -> Option { +pub(crate) fn charset(mime: &str) -> Option { mime.split(';').skip(1).find_map(|param| { let (name, value) = param.split_once('=')?; name.trim() @@ -238,7 +238,7 @@ fn charset(mime: &str) -> Option { /// UTF-8 unless the server says Latin-1; anything else is read as UTF-8, /// with what doesn't fit replaced. -fn decode(bytes: &[u8], charset: Option<&str>) -> String { +pub(crate) fn decode(bytes: &[u8], charset: Option<&str>) -> String { match charset { Some( "iso-8859-1" | "iso8859-1" | "latin1" | "latin-1" | "l1" | "windows-1252" | "cp1252", @@ -252,7 +252,7 @@ fn decode(bytes: &[u8], charset: Option<&str>) -> String { /// drops the markup (`=>`, `#`, `*`, `>`), and preformatted blocks stay /// as they are. Links are every `=>` line, plus `gemini://` URLs written /// in the text. -fn gemtext(base: &Url, text: &str) -> Parsed { +pub(crate) fn gemtext(base: &Url, text: &str) -> Parsed { let mut lines: Vec<&str> = Vec::new(); let mut links: Vec = Vec::new(); let mut title: Option<&str> = None; diff --git a/src/spider/http.rs b/src/spider/http.rs index 7a716a8..14f1d50 100644 --- a/src/spider/http.rs +++ b/src/spider/http.rs @@ -70,6 +70,40 @@ impl Http { } pub fn get(&self, route: Route, url: &Url) -> Result { + self.request(route, url, None, false, None) + } + + /// GET, the answer's body read whatever its type (an API's JSON, say). + pub fn get_data(&self, route: Route, url: &Url) -> Result { + self.request(route, url, None, true, None) + } + + /// GET asking for one type (`Accept`), the body read whatever its type: + /// a Nostr relay's information document, say. + pub fn get_as(&self, route: Route, url: &Url, accept: &str) -> Result { + self.request(route, url, None, true, Some(accept)) + } + + /// POSTs `body` (of type `content_type`); the answer's body is read + /// whatever its type (an API's JSON, say). + pub fn post( + &self, + route: Route, + url: &Url, + content_type: &str, + body: &[u8], + ) -> Result { + self.request(route, url, Some((content_type, body)), true, None) + } + + fn request( + &self, + route: Route, + url: &Url, + upload: Option<(&str, &[u8])>, + any_body: bool, + accept: Option<&str>, + ) -> Result { let https = url.scheme == "https"; let via_http_proxy = route == Route::Proxied && matches!(self.proxy, Some(Proxy::Http(_))); if https && via_http_proxy { @@ -100,14 +134,26 @@ impl Http { Some(port) => format!("{}:{port}", url.host), None => url.host.clone(), }; + let (method, extra) = match upload { + Some((content_type, body)) => ( + "POST", + format!( + "Content-Type: {content_type}\r\nContent-Length: {}\r\n", + body.len() + ), + ), + None => ("GET", String::new()), + }; + let accept = accept.unwrap_or("text/html, text/plain;q=0.9, */*;q=0.5"); let request = format!( - "GET {target} HTTP/1.1\r\nHost: {host_header}\r\nUser-Agent: lost-spider/{}\r\n\ - Accept: text/html, text/plain;q=0.9, */*;q=0.5\r\nAccept-Encoding: gzip\r\n\ - Connection: close\r\n\r\n", + "{method} {target} HTTP/1.1\r\nHost: {host_header}\r\nUser-Agent: lost-spider/{}\r\n\ + Accept: {accept}\r\nAccept-Encoding: gzip\r\n\ + {extra}Connection: close\r\n\r\n", env!("CARGO_PKG_VERSION") ); stream .write_all(request.as_bytes()) + .and_then(|()| stream.write_all(upload.map_or(&[][..], |(_, body)| body))) .and_then(|()| stream.flush()) .map_err(|e| e.to_string())?; @@ -145,7 +191,8 @@ impl Http { body: Vec::new(), }; - let limit = if (200..300).contains(&status) && is_text(&answer.content_type()) { + let wanted = any_body || is_text(&answer.content_type()); + let limit = if (200..300).contains(&status) && wanted { MAX_BODY } else if status >= 400 { MAX_ERROR_BODY @@ -483,6 +530,57 @@ mod tests { Url::parse(text).unwrap() } + #[test] + fn posting() { + // A server that reads the whole request, body too, as servers do. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let mut reader = BufReader::new(stream.try_clone().unwrap()); + let mut request = String::new(); + loop { + let mut line = String::new(); + if reader.read_line(&mut line).unwrap() == 0 || line == "\r\n" { + break; + } + request.push_str(&line); + } + let length: usize = request + .lines() + .find_map(|l| l.strip_prefix("Content-Length: ")) + .and_then(|n| n.trim().parse().ok()) + .unwrap_or(0); + let mut body = vec![0u8; length]; + reader.read_exact(&mut body).unwrap(); + stream + .write_all(b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n{\"data\":1}") + .unwrap(); + (request, body) + }); + let http = Http::new(None, Duration::from_secs(5)).unwrap(); + let answer = http + .post( + Route::Direct(addr), + &url("http://gateway.example/graphql"), + "application/json", + b"{\"query\":\"x\"}", + ) + .unwrap(); + // JSON isn't text, but what was asked for is read all the same. + assert_eq!(answer.body, b"{\"data\":1}"); + let (request, body) = server.join().unwrap(); + assert!( + request.starts_with("POST /graphql HTTP/1.1\r\n"), + "{request}" + ); + assert!( + request.contains("Content-Type: application/json\r\n"), + "{request}" + ); + assert_eq!(body, b"{\"query\":\"x\"}"); + } + #[test] fn plain_chunked_and_through_a_proxy() { let http = Http::new(None, Duration::from_secs(5)).unwrap(); diff --git a/src/spider/i2p.rs b/src/spider/i2p.rs index 879e08a..829d2d0 100644 --- a/src/spider/i2p.rs +++ b/src/spider/i2p.rs @@ -243,7 +243,8 @@ fn base64_i2p(text: &str) -> Option> { Some(out) } -fn base32(bytes: &[u8]) -> String { +/// RFC 4648 base32, lowercase, without padding (also IPFS's `b…` CIDs). +pub(crate) fn base32(bytes: &[u8]) -> String { const ALPHABET: &[u8; 32] = b"abcdefghijklmnopqrstuvwxyz234567"; let mut out = String::with_capacity(bytes.len() * 8 / 5 + 1); let (mut bits, mut count) = (0u32, 0); diff --git a/src/spider/ipfs.rs b/src/spider/ipfs.rs new file mode 100644 index 0000000..43455fe --- /dev/null +++ b/src/spider/ipfs.rs @@ -0,0 +1,792 @@ +//! IPFS: sites and files addressed by what's in them, through an IPFS +//! node's HTTP gateway — Kubo's path gateway, `http://127.0.0.1:8080`, by +//! default; `--gateway` names another (a public one, say). +//! +//! Pages are saved under `ipfs://CID/path` (content that never changes) or +//! `ipns://NAME/path` (a name for the latest version of something: an IPNS +//! key, a DNSLink domain such as `docs.ipfs.tech`, or an ENS name such as +//! `vitalik.eth`; the node looks them up). Every way of writing the same +//! address becomes one: CIDs as CIDv1 in base32 (`bafy…`), keys in base36 +//! (`k51…`), and links to gateways (`https://ipfs.io/ipfs/…`, +//! `https://….ipfs.dweb.link/…`) as `ipfs://` links (ARCHITECTURE.md §15.7). + +use std::net::{IpAddr, SocketAddr, TcpStream, ToSocketAddrs}; +use std::time::{Duration, Instant}; + +use lost::urls::Url; + +use crate::crawl::{Blacklist, Link, Parsed, Protocol, Response, Route, mentions}; +use crate::http::{self, Http}; +use crate::i2p::base32; +use crate::net::Proxy; + +/// Finding content on the network the first time can take a while. +pub const TIMEOUT: Duration = Duration::from_secs(60); +pub const GATEWAY: &str = "http://127.0.0.1:8080"; +/// How long to wait for a node that has just started to find content. +const WARM_UP: Duration = Duration::from_secs(10 * 60); + +/// Where a crawl starts: the IPFS project's own sites (DNSLink), and an +/// ENS name's. +const SEEDS: &[&str] = &[ + "ipns://docs.ipfs.tech/", + "ipns://blog.ipfs.tech/", + "ipns://specs.ipfs.tech/", + "ipns://ipfs.tech/", + "ipns://vitalik.eth/", +]; + +/// Multicodecs: UnixFS files and folders (what every CIDv0 is), and a +/// public key (an IPNS name). +const DAG_PB: u8 = 0x70; +const LIBP2P_KEY: u64 = 0x72; + +const BASE58: &[u8] = b"123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"; +const BASE36: &[u8] = b"0123456789abcdefghijklmnopqrstuvwxyz"; +/// Longer isn't a CID or a key (and isn't worth the arithmetic). +const MAX_ID: usize = 200; + +pub struct Ipfs { + http: Http, + /// The gateway's scheme, host and port. + gateway: Url, + /// Where it is, when it's reached directly (no `--proxy`). + address: Option, + /// It's on this computer: requests to it needn't wait for each other. + local: bool, +} + +impl Ipfs { + pub fn new( + gateway: Option, + proxy: Option, + timeout: Option, + ) -> lost::Result { + let text = gateway.unwrap_or_else(|| GATEWAY.to_string()); + let mut gateway = Url::parse(text.trim().trim_end_matches('/')) + .filter(|u| { + (u.scheme == "http" || u.scheme == "https") && u.path == "/" && u.query.is_none() + }) + .ok_or_else(|| { + format!("--gateway takes http://HOST:PORT or https://HOST (not \"{text}\")") + })?; + // Kubo's gateway at `localhost` sends every request on to a + // subdomain (`CID.ipfs.localhost`); at 127.0.0.1 it answers itself. + if gateway.host == "localhost" { + gateway.host = "127.0.0.1".into(); + } + let ip = gateway + .host + .trim_start_matches('[') + .trim_end_matches(']') + .parse::() + .ok(); + let local = proxy.is_none() && ip.is_some_and(|ip| ip.is_loopback()); + let address = match proxy { + Some(_) => None, + None => { + let port = gateway + .port + .unwrap_or(if gateway.scheme == "https" { 443 } else { 80 }); + let host = gateway.host.trim_start_matches('[').trim_end_matches(']'); + Some( + (host, port) + .to_socket_addrs() + .ok() + .and_then(|mut a| a.next()) + .ok_or_else(|| format!("can't find the gateway {}", gateway.host))?, + ) + } + }; + Ok(Ipfs { + http: Http::new(proxy, timeout.unwrap_or(TIMEOUT))?, + gateway, + address, + local, + }) + } + + fn route(&self) -> Route { + match self.address { + Some(address) => Route::Direct(address), + None => Route::Proxied, + } + } + + /// `ipfs://CID/path?query` as the gateway has it: `/ipfs/CID/path?query`. + fn on_gateway(&self, url: &Url) -> Url { + let mut at = self.gateway.clone(); + at.path = format!("/{}/{}{}", url.scheme, url.host, url.path); + at.query = url.query.clone(); + at.fragment = None; + at + } + + /// Where a redirect from the gateway goes (`/ipfs/CID/dir/`, or a whole + /// URL on it), as an `ipfs://` link. Other redirects stay as they are. + fn redirected(&self, location: &str) -> String { + let path = Url::parse(location) + .filter(|u| u.host == self.gateway.host && u.port == self.gateway.port) + .map(|u| u.path); + from_path(path.as_deref().unwrap_or(location)) + .or_else(|| canonical(location)) + .unwrap_or_else(|| location.to_string()) + } +} + +impl Protocol for Ipfs { + const NAME: &'static str = "ipfs"; + /// `--link docs.ipfs.tech` is a name: `ipns://docs.ipfs.tech/`. + const SCHEME: &'static str = "ipns"; + const SPACE: &'static str = "IPFS"; + const HOSTS: &'static str = "sites"; + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + fn proxy(&self) -> Option<&Proxy> { + self.http.proxy() + } + + /// Through a gateway on this computer, every site is a server of its + /// own; through any other, they all are that one server, asked one + /// thing at a time (`--delay`), as a public gateway should be. + fn proxied(&self) -> bool { + self.local + } + + fn resolve(&self, _url: &Url) -> Option { + Some( + self.address + .unwrap_or_else(|| SocketAddr::from(([0, 0, 0, 0], 0))), + ) + } + + fn accepts(url: &Url) -> bool { + (url.scheme == "ipfs" || url.scheme == "ipns") && !url.host.is_empty() + } + + fn rewrite(link: &str) -> Option { + canonical(link) + } + + fn fetch(&self, _route: Route, url: &Url) -> Result { + let answer = self.http.get(self.route(), &self.on_gateway(url))?; + if answer.status >= 500 && missing_path(&answer.body) { + // The content is there, but not this path in it (some gateways + // say so with a 500): not found, like a 404. + return Ok(Response::Gone); + } + if answer.status >= 500 { + // Not found on the network in time, or a name the node can't + // look up. + return Err(format!("not found on the network ({})", answer.status)); + } + Ok(match http::response(answer) { + Response::Redirect(location) => Response::Redirect(self.redirected(&location)), + other => other, + }) + } + + fn parse(&self, url: &Url, mime: &str, body: &[u8]) -> Parsed { + let mut parsed = http::parse(url, mime, body); + let written = mentions(&parsed.body, "ipfs") + .into_iter() + .chain(mentions(&parsed.body, "ipns")); + let mut links: Vec = Vec::new(); + for mut link in std::mem::take(&mut parsed.links).into_iter().chain(written) { + if let Some(url) = canonical(&link.url) { + link.url = url; + } + if !links.iter().any(|l| l.url == link.url) { + links.push(link); + } + } + parsed.links = links; + parsed + } + + /// Fails at once, with a message for people, if the gateway isn't there. + fn prepare(&self, _blacklist: &mut Blacklist) -> lost::Result<()> { + if let Some(address) = self.address { + TcpStream::connect_timeout(&address, Duration::from_secs(5)).map_err(|e| { + let hint = if self.local { + "; is your IPFS node running? (ipfs daemon)" + } else { + "" + }; + format!("can't reach the IPFS gateway at {address} ({e}){hint}") + })?; + } + // A node finds nothing until it's connected to other nodes, a few + // seconds to minutes after it starts; until then its gateway answers + // every request with an error (504, after its own timeout). So the + // first seed is asked for until it comes. + let probe = Url::parse(SEEDS[0]).expect("a valid URL"); + let started = Instant::now(); + let mut waiting = false; + loop { + let problem = match self.http.get(self.route(), &self.on_gateway(&probe)) { + Ok(answer) if answer.status < 500 => { + if waiting { + eprintln!("The IPFS node is finding content."); + } + return Ok(()); + } + Ok(answer) => format!( + "the gateway answered {}{}", + answer.status, + reason(&answer.body) + ), + Err(e) => e, + }; + if started.elapsed() >= WARM_UP { + return Err(format!( + "the IPFS node couldn't fetch {} in {} minutes ({problem}); \ + is it connected to other nodes? (ipfs swarm peers)", + SEEDS[0], + WARM_UP.as_secs() / 60 + ) + .into()); + } + if !waiting { + eprintln!("Waiting for the IPFS node to find content ({problem})…"); + waiting = true; + } + std::thread::sleep(Duration::from_secs(10)); + } + } +} + +/// What a gateway's error page says, if anything: `: its first line`. +fn reason(body: &[u8]) -> String { + let text = String::from_utf8_lossy(body); + let line = text.lines().map(str::trim).find(|l| !l.is_empty()); + match line { + Some(line) => format!(": {}", line.chars().take(200).collect::()), + None => String::new(), + } +} + +/// A gateway's error page about a path that isn't in the content (a +/// robots.txt asked of a file, a page a site doesn't have). +fn missing_path(body: &[u8]) -> bool { + let text = String::from_utf8_lossy(body).to_ascii_lowercase(); + ["no link named", "not a directory", "file does not exist"] + .iter() + .any(|w| text.contains(w)) +} + +/// The address pages of IPFS are saved under, for a link to IPFS written +/// any way: `ipfs://`/`ipns://` with any CID or key, a path on a gateway +/// (`https://ipfs.io/ipfs/CID/…`), or a subdomain gateway +/// (`https://CID.ipfs.dweb.link/…`, `https://docs-ipfs-tech.ipns.dweb.link/…`). +/// Queries (the gateways' `?filename=` and such) and fragments go. `None` +/// for anything else. +pub fn canonical(text: &str) -> Option { + let text = text.trim(); + let (scheme, rest) = text.split_once("://")?; + let scheme = scheme.to_ascii_lowercase(); + let rest = rest.split('#').next().unwrap_or(""); + match scheme.as_str() { + "ipfs" | "ipns" => { + let end = rest.find(['/', '?']).unwrap_or(rest.len()); + let (id, after) = rest.split_at(end); + let path = after.split('?').next().unwrap_or(""); + // `/ipfs/CID/…` resolved against a page, as if it were a site's + // root (how gateways' folder listings link). + from_path(path).or_else(|| address(&scheme, id, path)) + } + "http" | "https" => { + let url = Url::parse(text)?; + let labels: Vec<&str> = url.host.split('.').collect(); + if labels.len() >= 3 && (labels[1] == "ipfs" || labels[1] == "ipns") { + let id = if labels[1] == "ipns" { + inline_dnslink(labels[0]) + } else { + labels[0].to_string() + }; + if let Some(found) = address(labels[1], &id, &url.path) { + return Some(found); + } + } + from_path(&url.path) + } + _ => None, + } +} + +/// `/ipfs/CID/rest` or `/ipns/NAME/rest`, as an address. +fn from_path(path: &str) -> Option { + let rest = path.strip_prefix('/')?; + let (kind, rest) = rest.split_once('/')?; + if kind != "ipfs" && kind != "ipns" { + return None; + } + let end = rest.find('/').unwrap_or(rest.len()); + let (id, path) = rest.split_at(end); + address(kind, id, path) +} + +/// `ipfs://CID/path` or `ipns://NAME/path`, written the one way. A CID +/// given as a name is content all the same. +fn address(kind: &str, id: &str, path: &str) -> Option { + let path = if path.is_empty() { "/" } else { path }; + if kind == "ipns" + && let Some(name) = name(id) + { + return Some(format!("ipns://{name}{path}")); + } + content_id(id).map(|cid| format!("ipfs://{cid}{path}")) +} + +/// An IPNS name: a key (`k51…`), or a domain (DNSLink, ENS), lowercase. +fn name(text: &str) -> Option { + if let Some(key) = key(text) { + return Some(key); + } + let domain = text.to_ascii_lowercase(); + let valid = domain.contains('.') + && domain.split('.').all(|label| { + !label.is_empty() + && label + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-') + }); + valid.then_some(domain) +} + +/// `docs-ipfs-tech` (a DNSLink name made one DNS label, as subdomain +/// gateways write it) as `docs.ipfs.tech`: `-` was `.`, `--` was `-`. +fn inline_dnslink(label: &str) -> String { + label + .replace("--", "\0") + .replace('-', ".") + .replace('\0', "-") +} + +/// A content ID as CIDv1 in base32 (`bafy…`), from any CID: CIDv0 (`Qm…`), +/// or CIDv1 in base32, base36, base58 or hex. Keys aren't content. +fn content_id(text: &str) -> Option { + let bytes = if is_v0(text) { + let multihash = base_decode(text, BASE58)?; + if multihash.len() != 34 || multihash[..2] != [0x12, 0x20] { + return None; + } + let mut bytes = vec![1, DAG_PB]; + bytes.extend(multihash); + bytes + } else { + multibase(text)? + }; + let (version, rest) = varint(&bytes)?; + let (codec, multihash) = varint(rest)?; + if version != 1 || codec == LIBP2P_KEY || !is_multihash(multihash) { + return None; + } + Some(format!("b{}", base32(&bytes))) +} + +/// An IPNS key as CIDv1 in base36 (`k51…`), from a peer ID (`12D3Koo…`) +/// or a CIDv1 of a key in any base. (Old RSA peer IDs looked like CIDv0, +/// `Qm…`; those are taken as content, which is what `Qm…` almost always is.) +fn key(text: &str) -> Option { + let bytes = if text.starts_with("12D3Koo") { + let multihash = base_decode(text, BASE58)?; + if !is_multihash(&multihash) { + return None; + } + let mut bytes = vec![1, LIBP2P_KEY as u8]; + bytes.extend(multihash); + bytes + } else { + let bytes = multibase(text)?; + let (version, rest) = varint(&bytes)?; + let (codec, multihash) = varint(rest)?; + if version != 1 || codec != LIBP2P_KEY || !is_multihash(multihash) { + return None; + } + bytes + }; + Some(format!("k{}", base_encode(&bytes, BASE36))) +} + +fn is_v0(text: &str) -> bool { + text.len() == 46 && text.starts_with("Qm") +} + +/// A multihash: the hash's code, its length, and that many bytes. +fn is_multihash(bytes: &[u8]) -> bool { + let Some((_, rest)) = varint(bytes) else { + return false; + }; + let Some((length, digest)) = varint(rest) else { + return false; + }; + length > 0 && digest.len() as u64 == length +} + +/// The bytes of a multibase string: its first character says the base. +fn multibase(text: &str) -> Option> { + if text.len() > MAX_ID || !text.is_ascii() || text.len() < 2 { + return None; + } + let (prefix, rest) = text.split_at(1); + match prefix { + "b" | "B" => base32_decode(&rest.to_ascii_lowercase()), + "k" | "K" => base_decode(&rest.to_ascii_lowercase(), BASE36), + "z" => base_decode(rest, BASE58), + "f" | "F" => hex_decode(rest), + _ => None, + } +} + +/// An unsigned LEB128 number, and what follows it. +pub(crate) fn varint(bytes: &[u8]) -> Option<(u64, &[u8])> { + let mut value = 0u64; + for (i, &b) in bytes.iter().enumerate().take(9) { + value |= u64::from(b & 0x7f) << (7 * i); + if b & 0x80 == 0 { + return Some((value, &bytes[i + 1..])); + } + } + None +} + +/// Text in a base of `alphabet`'s length (base58, base36) as bytes; each +/// leading zero digit is a zero byte. +fn base_decode(text: &str, alphabet: &[u8]) -> Option> { + if text.len() > MAX_ID { + return None; + } + let base = alphabet.len() as u32; + // Little-endian bytes of the number so far. + let mut bytes: Vec = Vec::new(); + for c in text.bytes() { + let mut carry = alphabet.iter().position(|&a| a == c)? as u32; + for b in bytes.iter_mut() { + carry += u32::from(*b) * base; + *b = carry as u8; + carry >>= 8; + } + while carry > 0 { + bytes.push(carry as u8); + carry >>= 8; + } + } + let zeros = text.bytes().take_while(|&c| c == alphabet[0]).count(); + bytes.extend(std::iter::repeat_n(0, zeros)); + bytes.reverse(); + Some(bytes) +} + +/// Bytes as text in a base of `alphabet`'s length. +fn base_encode(bytes: &[u8], alphabet: &[u8]) -> String { + let base = alphabet.len() as u32; + // Little-endian digits of the number so far. + let mut digits: Vec = Vec::new(); + for &b in bytes { + let mut carry = u32::from(b); + for d in digits.iter_mut() { + carry += u32::from(*d) << 8; + *d = (carry % base) as u8; + carry /= base; + } + while carry > 0 { + digits.push((carry % base) as u8); + carry /= base; + } + } + let zeros = bytes.iter().take_while(|&&b| b == 0).count(); + std::iter::repeat_n(alphabet[0], zeros) + .chain(digits.iter().rev().map(|&d| alphabet[d as usize])) + .map(char::from) + .collect() +} + +/// RFC 4648 base32, lowercase, without padding. +pub(crate) fn base32_decode(text: &str) -> Option> { + let mut out = Vec::with_capacity(text.len() * 5 / 8); + let (mut bits, mut count) = (0u32, 0); + for c in text.bytes() { + let value = match c { + b'a'..=b'z' => c - b'a', + b'2'..=b'7' => c - b'2' + 26, + _ => return None, + }; + bits = bits << 5 | u32::from(value); + count += 5; + if count >= 8 { + count -= 8; + out.push((bits >> count) as u8); + bits &= (1 << count) - 1; + } + } + Some(out) +} + +fn hex_decode(text: &str) -> Option> { + if !text.len().is_multiple_of(2) { + return None; + } + (0..text.len()) + .step_by(2) + .map(|i| u8::from_str_radix(text.get(i..i + 2)?, 16).ok()) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{BufRead, BufReader, Write}; + + const V0: &str = "QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR"; + const V1: &str = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + const PEER: &str = "12D3KooWDpJ7As7BWAwRMfu1VU2WCqNjvq387JEYKDBj4kx6nXTN"; + const KEY: &str = "k51qzi5uqu5dhnwe629wdlncpql6frppdpwnz4wtlcw816aysd5wwlk63g4wmh"; + + #[test] + fn content_ids_and_keys() { + assert_eq!(content_id(V0).as_deref(), Some(V1)); + assert_eq!(content_id(V1).as_deref(), Some(V1)); + assert_eq!(content_id(&V1.to_ascii_uppercase()).as_deref(), Some(V1)); + assert_eq!( + content_id("zdj7Wic6KcJAfWz1c9o4M6kq9Lwd5BfbxkVafnrojaaGiSFxM").as_deref(), + Some(V1) + ); + assert_eq!( + content_id("f01701220c3c4733ec8affd06cf9e9ff50ffc6bcd2ec85a6170004bb709669c31de94391a") + .as_deref(), + Some(V1) + ); + // A key isn't content; nonsense is neither. + assert_eq!(content_id(KEY), None); + assert_eq!(content_id("QmShort"), None); + assert_eq!(content_id("hello"), None); + assert_eq!(content_id(&V0.to_ascii_lowercase()), None); + assert_eq!(key(PEER).as_deref(), Some(KEY)); + assert_eq!(key(KEY).as_deref(), Some(KEY)); + assert_eq!(key(V1), None); + // Base58 both ways. + let bytes = base_decode(V0, BASE58).unwrap(); + assert_eq!(base_encode(&bytes, BASE58), V0); + assert_eq!(base_encode(&[0, 0, 1], BASE58), "112"); + assert_eq!(base_decode("112", BASE58).unwrap(), [0, 0, 1]); + } + + #[test] + fn one_way_to_write_each_address() { + let c = |s: &str| canonical(s); + let file = format!("ipfs://{V1}/a/b.html"); + assert_eq!(c(&format!("ipfs://{V0}/a/b.html#top")), Some(file.clone())); + assert_eq!(c(&format!("IPFS://{V1}/a/b.html?x=1")), Some(file.clone())); + assert_eq!( + c(&format!( + "https://ipfs.io/ipfs/{V0}/a/b.html?filename=b.html" + )), + Some(file.clone()) + ); + assert_eq!( + c(&format!("https://{V1}.ipfs.dweb.link/a/b.html")), + Some(file) + ); + assert_eq!(c(&format!("ipfs://{V1}")), Some(format!("ipfs://{V1}/"))); + assert_eq!( + c("https://en-wikipedia--on--ipfs-org.ipns.dweb.link/wiki/"), + Some("ipns://en.wikipedia-on-ipfs.org/wiki/".into()) + ); + assert_eq!( + c("http://127.0.0.1:8080/ipns/docs.ipfs.tech/install/"), + Some("ipns://docs.ipfs.tech/install/".into()) + ); + assert_eq!( + c("ipns://Docs.IPFS.tech"), + Some("ipns://docs.ipfs.tech/".into()) + ); + assert_eq!(c(&format!("ipns://{V1}/x")), Some(format!("ipfs://{V1}/x"))); + assert_eq!( + c(&format!("ipns://{PEER}/x")), + Some(format!("ipns://{KEY}/x")) + ); + // A gateway's folder listing links from the root. + assert_eq!( + c(&format!("ipfs://{V1}/ipfs/{V0}/x")), + Some(format!("ipfs://{V1}/x")) + ); + // Not IPFS. + assert_eq!(c("https://example.com/ipfs/notacid"), None); + assert_eq!(c("https://example.com/page"), None); + assert_eq!(c("gemini://example.com/"), None); + assert_eq!(c("ipns://nodots"), None); + } + + /// A gateway answering one request with `answer`; returns the request. + fn serve(answer: &'static [u8]) -> (SocketAddr, std::thread::JoinHandle) { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let handle = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let mut request = String::new(); + let mut reader = BufReader::new(stream.try_clone().unwrap()); + loop { + let mut line = String::new(); + if reader.read_line(&mut line).unwrap() == 0 || line == "\r\n" { + break; + } + request.push_str(&line); + } + stream.write_all(answer).unwrap(); + request + }); + (addr, handle) + } + + fn url(text: &str) -> Url { + Url::parse(text).unwrap() + } + + #[test] + fn waiting_for_the_node() { + // The gateway's check, then the first seed, which comes. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let server = std::thread::spawn(move || { + let (check, _) = listener.accept().unwrap(); + drop(check); + let (mut stream, _) = listener.accept().unwrap(); + let mut reader = BufReader::new(stream.try_clone().unwrap()); + let mut first = String::new(); + reader.read_line(&mut first).unwrap(); + loop { + let mut line = String::new(); + if reader.read_line(&mut line).unwrap() == 0 || line == "\r\n" { + break; + } + } + stream + .write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n

Docs

") + .unwrap(); + first + }); + let ipfs = Ipfs::new( + Some(format!("http://{addr}")), + None, + Some(Duration::from_secs(5)), + ) + .unwrap(); + ipfs.prepare(&mut Blacklist::default()).unwrap(); + assert_eq!( + server.join().unwrap(), + "GET /ipns/docs.ipfs.tech/ HTTP/1.1\r\n" + ); + assert_eq!( + reason(b"\n ipfs resolve -r /ipns/x: context deadline exceeded\nmore"), + ": ipfs resolve -r /ipns/x: context deadline exceeded" + ); + assert_eq!(reason(b""), ""); + } + + #[test] + fn through_the_gateway() { + let answer: &'static [u8] = Box::leak( + format!( + "HTTP/1.1 301 Moved Permanently\r\nLocation: /ipfs/{V1}/dir/\r\nContent-Length: 0\r\n\r\n" + ) + .into_bytes() + .into_boxed_slice(), + ); + let (addr, server) = serve(answer); + let ipfs = Ipfs::new( + Some(format!("http://{addr}")), + None, + Some(Duration::from_secs(5)), + ) + .unwrap(); + assert!(ipfs.proxied(), "a gateway here: every site its own server"); + let response = ipfs + .fetch(Route::Proxied, &url(&format!("ipfs://{V1}/dir?x=1"))) + .unwrap(); + assert_eq!(response, Response::Redirect(format!("ipfs://{V1}/dir/"))); + let request = server.join().unwrap(); + assert!( + request.starts_with(&format!("GET /ipfs/{V1}/dir?x=1 HTTP/1.1\r\n")), + "{request}" + ); + assert!(request.contains(&format!("Host: {addr}\r\n")), "{request}"); + + // Not found in time: the site can't be reached. + let (addr, _server) = serve(b"HTTP/1.1 504 Gateway Timeout\r\nContent-Length: 0\r\n\r\n"); + let ipfs = Ipfs::new( + Some(format!("http://{addr}")), + None, + Some(Duration::from_secs(5)), + ) + .unwrap(); + assert!( + ipfs.fetch(Route::Proxied, &url("ipns://docs.ipfs.tech/")) + .is_err() + ); + + // A path the content doesn't have: not found, not unreachable. + let (addr, _server) = serve( + b"HTTP/1.1 500 Internal Server Error\r\nContent-Type: text/plain\r\n\r\n\ + failed to resolve /ipfs/bafy/robots.txt: no link named \"robots.txt\" under bafy", + ); + let ipfs = Ipfs::new( + Some(format!("http://{addr}")), + None, + Some(Duration::from_secs(5)), + ) + .unwrap(); + assert_eq!( + ipfs.fetch(Route::Proxied, &url(&format!("ipfs://{V1}/robots.txt"))), + Ok(Response::Gone) + ); + + // A gateway elsewhere is one server for every site; localhost is + // asked as 127.0.0.1. + let remote = Ipfs::new(Some("http://192.0.2.1:8080".into()), None, None).unwrap(); + assert!(!remote.proxied()); + let local = Ipfs::new(Some("http://localhost:8080/".into()), None, None).unwrap(); + assert_eq!(local.gateway.host, "127.0.0.1"); + assert!(Ipfs::new(Some("ftp://x".into()), None, None).is_err()); + assert!(Ipfs::new(Some("http://x.example/path".into()), None, None).is_err()); + } + + #[test] + fn pages_and_their_links() { + let ipfs = Ipfs::new(None, None, None).unwrap(); + let html = format!( + "HomeAbout\ + x\ + y\ + docs\ + web\ +

see ipfs://{V0}/z.

" + ); + let parsed = ipfs.parse( + &url(&format!("ipfs://{V1}/index.html")), + "text/html", + html.as_bytes(), + ); + assert_eq!(parsed.title.as_deref(), Some("Home")); + let links: Vec<&str> = parsed.links.iter().map(|l| l.url.as_str()).collect(); + for expected in [ + format!("ipfs://{V1}/about.html"), + format!("ipfs://{V1}/x"), + format!("ipfs://{V1}/y"), + "ipns://docs.ipfs.tech/".to_string(), + "https://example.com/".to_string(), + format!("ipfs://{V1}/z"), + ] { + assert!( + links.contains(&expected.as_str()), + "{expected} in {links:?}" + ); + } + assert!(Ipfs::accepts(&url(&format!("ipfs://{V1}/")))); + assert!(Ipfs::accepts(&url("ipns://docs.ipfs.tech/"))); + assert!(!Ipfs::accepts(&url("https://ipfs.io/"))); + } +} diff --git a/src/spider/main.rs b/src/spider/main.rs index c319dea..35d9015 100644 --- a/src/spider/main.rs +++ b/src/spider/main.rs @@ -5,13 +5,22 @@ //! LOST's library for the `.lost` format and your ID, and nothing in LOST //! knows it exists. +mod arweave; mod crawl; +mod finger; mod gemini; mod gopher; mod http; mod i2p; +mod ipfs; mod net; +mod nex; +mod nostr; mod onion; +mod spartan; +mod swarm; +mod ton; +mod websocket; use std::io::IsTerminal; use std::path::{Path, PathBuf}; @@ -25,12 +34,20 @@ use lost::lostfile::{Reader, Writer}; use lost::text::{format_bytes, thousands}; use lost::urls::Url; +use crate::arweave::Arweave; use crate::crawl::{Blacklist, Crawler, Known, Protocol, Settings}; +use crate::finger::Finger; use crate::gemini::Gemini; use crate::gopher::Gopher; use crate::i2p::I2p; +use crate::ipfs::Ipfs; use crate::net::Proxy; +use crate::nex::Nex; +use crate::nostr::Nostr; use crate::onion::Onion; +use crate::spartan::Spartan; +use crate::swarm::Swarm; +use crate::ton::Ton; const USAGE: &str = "\ lost-spider — LOST's crawler. It crawls a network and saves what it finds @@ -39,8 +56,19 @@ as a .lost file for LOST to import. Usage: lost-spider gemini [OPTIONS] Crawl Geminispace lost-spider gopher [OPTIONS] Crawl Gopherspace + lost-spider spartan [OPTIONS] Crawl Spartan capsules + lost-spider nex [OPTIONS] Crawl Nex servers + lost-spider finger [OPTIONS] Crawl Finger servers (people's .plans) lost-spider onion [OPTIONS] Crawl .onion sites, through Tor lost-spider i2p [OPTIONS] Crawl I2P sites, through an I2P router + lost-spider ipfs [OPTIONS] Crawl IPFS sites (and ENS names), through an + IPFS node's gateway + lost-spider arweave [OPTIONS] Crawl Arweave (the permaweb), through a gateway + lost-spider nostr [OPTIONS] Crawl Nostr: notes, articles and profiles, + from relays + lost-spider ton [OPTIONS] Crawl TON Sites, through a TON proxy + lost-spider swarm [OPTIONS] Crawl Swarm sites (and ENS names), through a + gateway Options: -o, --output PATH Where to save: a .lost file, or a folder to put one in @@ -60,21 +88,32 @@ Options: links: to add those pages to a file, and nothing else --search PHRASE Type PHRASE into the search boxes found, and crawl what they find (repeat for more). Only boxes that - look like searches; never comment or password boxes + look like searches; never comment or password boxes. + nostr: searched for on the relays that can search --searches FILE The same, one phrase a line --blacklist FILE Skip URLs and pages that FILE lists, in this crawl (domains, URL beginnings, /regex/, title:/regex/, - body:/regex/; # starts a comment) - --delay SECONDS Wait between two requests to the same server - (default 1; decimals allowed) + body:/regex/, nostr: npub1… for an author's posts; + # starts a comment) + --delay SECONDS Wait between two requests to the same server (or + Nostr relay) (default 1; decimals allowed) --host-limit N Save at most N pages from one host (default: no limit) --proxy URL Go through a proxy: socks5://HOST:PORT (any network; Tor's is socks5://127.0.0.1:9050) or http://HOST:PORT - (I2P only). onion uses Tor's by default, i2p - http://127.0.0.1:4444 + (I2P, TON, or an http:// IPFS gateway). onion uses + Tor's by default, i2p http://127.0.0.1:4444, ton + http://127.0.0.1:8080 + --gateway URL ipfs, arweave, swarm: the gateway to ask (defaults: + ipfs http://127.0.0.1:8080, Kubo's; arweave + https://turbo-gateway.com; swarm + https://api.gateway.ethswarm.org, or a Bee node's + http://127.0.0.1:1633). One that isn't on this + computer is asked one thing at a time -t, --timeout SECONDS How long to wait for a server to connect or send more (a whole page gets four times that). Defaults: gopher - 10, gemini 15, either through a proxy 30, onion 60, + and finger 10, gemini, spartan and nex 15, any of + those through a proxy 30, nostr 20, + arweave 30, onion 60, ipfs 60, ton 60, swarm 60, i2p 120 -h, --help Show this help -V, --version Show the version @@ -101,6 +140,7 @@ struct Options { delay: Duration, host_limit: Option, proxy: Option, + gateway: Option, timeout: Option, } @@ -129,12 +169,28 @@ fn run(args: Vec) -> lost::Result { println!("lost-spider {}", env!("CARGO_PKG_VERSION")); Ok(ExitCode::SUCCESS) } - "gemini" => start(&args[1..], Gemini::new), - "gopher" => start(&args[1..], Gopher::new), - "onion" => start(&args[1..], Onion::new), - "i2p" => start(&args[1..], I2p::new), + "gemini" => start(&args[1..], |o| Gemini::new(o.proxy.take(), o.timeout)), + "gopher" => start(&args[1..], |o| Gopher::new(o.proxy.take(), o.timeout)), + "spartan" => start(&args[1..], |o| Spartan::new(o.proxy.take(), o.timeout)), + "nex" => start(&args[1..], |o| Nex::new(o.proxy.take(), o.timeout)), + "finger" => start(&args[1..], |o| Finger::new(o.proxy.take(), o.timeout)), + "onion" => start(&args[1..], |o| Onion::new(o.proxy.take(), o.timeout)), + "i2p" => start(&args[1..], |o| I2p::new(o.proxy.take(), o.timeout)), + "ipfs" => start(&args[1..], |o| { + Ipfs::new(o.gateway.take(), o.proxy.take(), o.timeout) + }), + "arweave" => start(&args[1..], |o| { + Arweave::new(o.gateway.take(), o.proxy.take(), o.timeout) + }), + "swarm" => start(&args[1..], |o| { + Swarm::new(o.gateway.take(), o.proxy.take(), o.timeout) + }), + "ton" => start(&args[1..], |o| Ton::new(o.proxy.take(), o.timeout)), + "nostr" => start(&args[1..], |o| { + Nostr::new(o.proxy.take(), o.timeout, o.delay, o.searches.clone()) + }), other => Err(format!( - "unknown network \"{other}\" (lost-spider knows: gemini, gopher, onion, i2p)" + "unknown network \"{other}\" (lost-spider knows: gemini, gopher, onion, i2p, ipfs, arweave, nostr, ton, swarm, spartan, nex, finger)" ) .into()), } @@ -143,13 +199,13 @@ fn run(args: Vec) -> lost::Result { /// Reads the options for network `P` and crawls it. fn start( args: &[String], - make: fn(Option, Option) -> lost::Result

, + make: impl FnOnce(&mut Options) -> lost::Result

, ) -> lost::Result { let Some(mut options) = parse_options::

(args)? else { print!("{USAGE}"); return Ok(ExitCode::SUCCESS); }; - let protocol = make(options.proxy.take(), options.timeout)?; + let protocol = make(&mut options)?; crawl(protocol, options) } @@ -166,6 +222,7 @@ fn parse_options(args: &[String]) -> lost::Result> delay: Duration::from_secs(1), host_limit: None, proxy: None, + gateway: None, timeout: None, }; let mut args = args.iter(); @@ -214,6 +271,12 @@ fn parse_options(args: &[String]) -> lost::Result> options.host_limit = Some(limit); } "--proxy" => options.proxy = Some(Proxy::parse(&value()?)?), + "--gateway" if matches!(P::NAME, "ipfs" | "arweave" | "swarm") => { + options.gateway = Some(value()?) + } + "--gateway" => { + return Err("--gateway is for lost-spider ipfs, arweave and swarm".into()); + } "-t" | "--timeout" => { let text = value()?; let seconds: f64 = text @@ -239,11 +302,13 @@ fn parse_options(args: &[String]) -> lost::Result> /// network's scheme). fn link(text: &str) -> lost::Result { let text = text.trim(); - let full = if text.contains("://") { + // `nostr:npub1…` has no `//`. + let full = if text.contains("://") || Url::parse(text).is_some() { text.to_string() } else { format!("{}://{text}", P::SCHEME) }; + let full = P::rewrite(&full).unwrap_or(full); match Url::parse(&full) { Some(url) if P::accepts(&url) => Ok(url.to_string()), _ => Err(format!("\"{text}\" isn't a link lost-spider {} can follow", P::NAME).into()), @@ -272,6 +337,7 @@ fn crawl(protocol: P, options: Options) -> lost::Result { let hint = match P::NAME { "onion" => " (is Tor running?)", "i2p" => " (is the I2P router running?)", + "ton" => " (is the TON proxy running?)", _ => "", }; format!("{e}{hint}") @@ -345,12 +411,17 @@ fn crawl(protocol: P, options: Options) -> lost::Result { }; let status = |pages: u64| { let stats = crawler.stats(); + let skipped = match stats.skipped.total() { + 0 => String::new(), + n => format!(" · {} skipped", thousands(n)), + }; format!( - "{} {} for {}… {} pages saved · {} {} · {} links waiting", + "{} {} for {}… {} pages saved{} · {} {} · {} links waiting", verb, P::SPACE, duration(started.elapsed()), thousands(pages), + skipped, thousands(stats.hosts as u64), P::HOSTS, thousands(stats.waiting), @@ -428,10 +499,33 @@ fn crawl(protocol: P, options: Options) -> lost::Result { path.display() ); } + if let Some(line) = skipped_line(crawler.stats().skipped) { + eprintln!("{line}"); + } eprintln!("To search them: lost -d \"{}\"", path.display()); Ok(ExitCode::SUCCESS) } +/// Why pages weren't saved, when some weren't: "Not saved: 40 copies of +/// pages saved already, 3 not found." +fn skipped_line(skipped: crawl::Skipped) -> Option { + let parts: Vec = [ + ( + skipped.copies, + "copy of a page saved already", + "copies of pages saved already", + ), + (skipped.gone, "not found", "not found"), + (skipped.unreachable, "unreachable", "unreachable"), + (skipped.blocked, "blacklisted", "blacklisted"), + ] + .into_iter() + .filter(|(n, _, _)| *n > 0) + .map(|(n, one, many)| format!("{} {}", thousands(n), if n == 1 { one } else { many })) + .collect(); + (!parts.is_empty()).then(|| format!("Not saved: {}.", parts.join(", "))) +} + /// "6 hours, 15 minutes, 56 seconds": the units from the largest one there /// is, so "56 seconds", "3 minutes, 0 seconds", "2 days, 0 hours, …". fn duration(elapsed: Duration) -> String { @@ -637,6 +731,27 @@ mod tests { assert!(parse_options::(&args(&["--link", "example.org"])).is_err()); assert!(parse_options::(&args(&["--proxy", "localhost:9050"])).is_err()); assert!(parse_options::(&args(&["-t", "0"])).is_err()); + // Nostr's links: a URI, a bare entity, a relay. + let npub = "npub10elfcs4fr0l0r8af98jlmgdh9c8tcxjvz9qkw038js35mp4dma8qzvjptg"; + let o = parse_options::(&args(&[ + "--link", + &format!("nostr:{npub}"), + "--link", + npub, + "--link", + "wss://relay.example", + ])) + .unwrap() + .unwrap(); + assert_eq!( + o.links, + [ + format!("nostr:{npub}"), + format!("nostr:{npub}"), + "wss://relay.example/".to_string() + ] + ); + assert!(parse_options::(&args(&["--link", "https://example.org/"])).is_err()); assert!(parse_options::(&args(&["-c"])).is_err()); assert!(parse_options::(&args(&["--dont-crawl"])).is_err()); let o = parse_options::(&args(&["--dont-crawl", "--link", "a.org"])); @@ -651,6 +766,30 @@ mod tests { assert!(parse_options::(&args(&["--link", "https://x.org/"])).is_err()); assert!(parse_options::(&args(&["--delay", "-1"])).is_err()); assert!(parse_options::(&args(&["--bogus"])).is_err()); + // IPFS: a gateway, and links written the one way. + let o = parse_options::(&args(&[ + "--gateway", + "https://ipfs.io", + "--link", + "docs.ipfs.tech", + "--link", + "QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR", + "--link", + "https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR/a", + ])) + .unwrap() + .unwrap(); + assert_eq!(o.gateway.as_deref(), Some("https://ipfs.io")); + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + assert_eq!( + o.links, + [ + "ipns://docs.ipfs.tech/".to_string(), + format!("ipfs://{cid}/"), + format!("ipfs://{cid}/a"), + ] + ); + assert!(parse_options::(&args(&["--gateway", "http://x:1"])).is_err()); assert!( parse_options::(&args(&["--help"])) .unwrap() diff --git a/src/spider/net.rs b/src/spider/net.rs index 27c8ff5..f81f9dd 100644 --- a/src/spider/net.rs +++ b/src/spider/net.rs @@ -98,6 +98,50 @@ pub fn connect( Ok(stream) } +/// One request on a plain connection, the way the small protocols go +/// (Spartan, Nex, Finger): `request` sent, then everything until the +/// server closes, at most `max` bytes, the whole of it within four times +/// `timeout`. A server that resets the connection after its answer has +/// still answered. +pub fn exchange( + route: Route, + proxy: Option<&Proxy>, + host: &str, + port: u16, + request: &str, + timeout: Duration, + max: usize, +) -> Result, String> { + let host = host.trim_start_matches('[').trim_end_matches(']'); + let mut stream = connect(route, proxy, host, port, timeout)?; + stream + .write_all(request.as_bytes()) + .map_err(|e| e.to_string())?; + let started = std::time::Instant::now(); + let mut body = Vec::new(); + let mut buffer = [0u8; 16 * 1024]; + while body.len() < max { + if started.elapsed() > timeout * 4 { + return Err("too slow".into()); + } + match stream.read(&mut buffer) { + Ok(0) => break, + Ok(n) => body.extend_from_slice(&buffer[..n]), + Err(e) + if matches!( + e.kind(), + std::io::ErrorKind::ConnectionReset | std::io::ErrorKind::ConnectionAborted + ) => + { + break; + } + Err(e) => return Err(e.to_string()), + } + } + body.truncate(max); + Ok(body) +} + /// SOCKS5 (RFC 1928), no authentication, connecting by name. fn socks5(stream: &mut TcpStream, host: &str, port: u16) -> Result<(), String> { let io = |e: std::io::Error| format!("proxy: {e}"); diff --git a/src/spider/nex.rs b/src/spider/nex.rs new file mode 100644 index 0000000..a851f80 --- /dev/null +++ b/src/spider/nex.rs @@ -0,0 +1,183 @@ +//! Nex (nex://nightfall.city/nex/info/specification.txt): plain TCP on +//! port 1900. A request is a path and CRLF; the answer is the file itself, +//! until the server closes the connection, with no status or type. A path +//! ending in `/` is a directory: text whose `=> URL label` lines are links, +//! as in gemtext (ARCHITECTURE.md §15.12). + +use std::time::Duration; + +use lost::urls::Url; + +use crate::crawl::{Parsed, Protocol, Response, Route, mentions, percent_decode, port}; +use crate::gemini::gemtext; +use crate::net::{self, Proxy}; + +/// Where a crawl starts: the protocol's home, Nightfall City. +const SEEDS: &[&str] = &["nex://nightfall.city/"]; + +pub const TIMEOUT: Duration = Duration::from_secs(15); +/// Through a proxy (Tor adds a few seconds a hop). +pub const PROXIED_TIMEOUT: Duration = Duration::from_secs(30); + +pub struct Nex { + /// A SOCKS5 proxy (Nex can't go through an HTTP one). + proxy: Option, + timeout: Duration, +} + +impl Nex { + pub fn new(proxy: Option, timeout: Option) -> lost::Result { + if let Some(Proxy::Http(_)) = proxy { + return Err("Nex can only go through a SOCKS5 proxy (socks5://HOST:PORT)".into()); + } + let timeout = timeout.unwrap_or(match proxy { + Some(_) => PROXIED_TIMEOUT, + None => TIMEOUT, + }); + Ok(Nex { proxy, timeout }) + } +} + +impl Protocol for Nex { + const NAME: &'static str = "nex"; + const SCHEME: &'static str = "nex"; + const SPACE: &'static str = "Nexspace"; + const HOSTS: &'static str = "servers"; + + fn proxy(&self) -> Option<&Proxy> { + self.proxy.as_ref() + } + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + fn fetch(&self, route: Route, url: &Url) -> Result { + let request = format!("{}\r\n", percent_decode(&url.path)); + let answer = net::exchange( + route, + self.proxy.as_ref(), + &url.host, + port(url), + &request, + self.timeout, + lost::MAX_BODY_BYTES, + )?; + if answer.is_empty() { + return Ok(Response::Gone); + } + // Nothing says what it is: a directory by its `/`, gemtext by its + // name, anything with a NUL in it isn't text. + let mime = if url.path.ends_with('/') { + "text/nex-directory" + } else if url.path.ends_with(".gmi") || url.path.ends_with(".gemini") { + "text/gemini" + } else if answer.contains(&0) { + return Ok(Response::Document { + mime: "application/octet-stream".into(), + body: None, + }); + } else { + "text/plain" + }; + Ok(Response::Document { + mime: mime.into(), + body: Some(answer), + }) + } + + fn parse(&self, url: &Url, mime: &str, body: &[u8]) -> Parsed { + let text = String::from_utf8_lossy(body).into_owned(); + let mut parsed = match mime { + "text/nex-directory" | "text/gemini" => gemtext(url, &text), + _ => Parsed { + body: text.clone(), + ..Parsed::default() + }, + }; + for link in mentions(&text, Self::SCHEME) { + if !parsed.links.iter().any(|l| l.url == link.url) { + parsed.links.push(link); + } + } + parsed + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A server giving `answers` in turn; what it was asked comes back. + fn serve(answers: Vec<&'static [u8]>) -> (u16, std::thread::JoinHandle>) { + use std::io::{Read, Write}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = std::thread::spawn(move || { + let mut asked = Vec::new(); + for answer in answers { + let (mut s, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n") { + s.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + asked.push(String::from_utf8(request).unwrap()); + s.write_all(answer).unwrap(); + } + asked + }); + (port, server) + } + + #[test] + fn directories_and_files() { + let (port, server) = serve(vec![ + b"Welcome to my nex\n=> notes/ My notes\n=> about.txt\n=> nex://other.example/\n", + b"Just text, mentioning nex://third.example/ once.\n", + b"\x89PNG\x00\x00", + b"", + ]); + let nex = Nex::new(None, Some(Duration::from_secs(5))).unwrap(); + let address = std::net::SocketAddr::from(([127, 0, 0, 1], port)); + let fetch = |path: &str| { + let url = Url::parse(&format!("nex://127.0.0.1:{port}{path}")).unwrap(); + (nex.fetch(Route::Direct(address), &url).unwrap(), url) + }; + let (Response::Document { mime, body }, url) = fetch("/") else { + panic!("a directory"); + }; + let page = nex.parse(&url, &mime, &body.unwrap()); + let links: Vec<&str> = page.links.iter().map(|l| l.url.as_str()).collect(); + let base = format!("nex://127.0.0.1:{port}"); + assert_eq!( + links, + [ + format!("{base}/notes/").as_str(), + format!("{base}/about.txt").as_str(), + "nex://other.example/" + ] + ); + assert_eq!(page.links[0].label.as_deref(), Some("My notes")); + let (Response::Document { mime, body }, url) = fetch("/about%20me.txt") else { + panic!("a file"); + }; + assert_eq!(mime, "text/plain"); + let page = nex.parse(&url, &mime, &body.unwrap()); + assert_eq!(page.links[0].url, "nex://third.example/"); + assert_eq!( + fetch("/cat.png").0, + Response::Document { + mime: "application/octet-stream".into(), + body: None + } + ); + assert_eq!(fetch("/missing").0, Response::Gone); + // The path, decoded, and CRLF. + assert_eq!( + server.join().unwrap(), + ["/\r\n", "/about me.txt\r\n", "/cat.png\r\n", "/missing\r\n"] + ); + } +} diff --git a/src/spider/nostr.rs b/src/spider/nostr.rs new file mode 100644 index 0000000..6e08cb3 --- /dev/null +++ b/src/spider/nostr.rs @@ -0,0 +1,2467 @@ +//! Nostr: posts ("events") signed by their authors and kept by relays, +//! servers anyone can run, reached over WebSocket (`wss://`). The same post +//! is usually on many relays, and none of them is its home: a post is named +//! by itself (NIP-19, NIP-21), `nostr:note1…` (by its ID), `nostr:naddr1…` +//! (an article or wiki entry, by author, kind and name: the newest version) +//! or `nostr:npub1…` (a profile, by its key). +//! +//! What's saved: notes (kind 1) and comments (1111), long-form articles +//! (30023), wiki entries (30818) and profiles (0, with the author's newest +//! posts and relays as its links). Each post's signature is checked, so no +//! relay can make up a post in someone's name; posts that ask to expire +//! (NIP-40) aren't saved at all. +//! +//! How it's crawled: each relay is listed, newest first, 200 posts at a +//! time, as pages that are followed, not saved: `wss://RELAY/?kinds=1`, then +//! `…&until=TIME` for the posts before. A listing's posts are kept here until +//! the crawl gets to them, so they're fetched once. Authors lead to their +//! profiles, profiles to their relays (NIP-65), posts to what they mention +//! and answer (ARCHITECTURE.md §15.9). + +use std::collections::{HashMap, HashSet, VecDeque}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; +use std::time::{Duration, Instant}; + +use k256::schnorr::{Signature, VerifyingKey}; +use k256::sha2::{Digest, Sha256}; +use lost::urls::Url; +use rustls::ClientConfig; +use serde_json::{Value, json}; + +use crate::crawl::{ + Blacklist, Link, Parsed, Protocol, Response, Route, encode_query, mentions, percent_decode, +}; +use crate::http::Http; +use crate::net::{self, Proxy}; +use crate::websocket::WebSocket; + +pub const TIMEOUT: Duration = Duration::from_secs(20); + +/// Where a crawl starts: big, general relays. +const SEEDS: &[&str] = &[ + "wss://relay.damus.io/", + "wss://nos.lol/", + "wss://relay.primal.net/", + "wss://relay.snort.social/", + "wss://nostr.mom/", + "wss://purplepag.es/", +]; +/// With `--search`: the seeds, and relays known to search (NIP-50). Any +/// other relay that says it can (its NIP-11 document) is searched too. +const SEEDS_AND_SEARCH: &[&str] = &[ + "wss://relay.damus.io/", + "wss://nos.lol/", + "wss://relay.primal.net/", + "wss://relay.snort.social/", + "wss://nostr.mom/", + "wss://purplepag.es/", + "wss://search.nos.today/", + "wss://relay.nostr.band/", + "wss://relay.ditto.pub/", + "wss://relay.noswhere.com/", +]; + +const PROFILE: u32 = 0; +const NOTE: u32 = 1; +const DELETION: u32 = 5; +const REPOST: u32 = 6; +const COMMENT: u32 = 1111; +const RELAY_LIST: u32 = 10002; +const ARTICLE: u32 = 30023; +const WIKI: u32 = 30818; + +/// What each relay is listed for, one listing apiece: page-like posts +/// (rarer, so not drowned among notes), and notes with the requests to +/// delete them (NIP-09: newer than what they delete, so listed first). +const LISTS: &[&str] = &["30023,30818", "1,5"]; +/// What a search asks for: profiles, notes, articles, wiki entries. +const SEARCHED: &str = "0,1,30023,30818"; +/// The kinds a listing can ask for. +const LISTABLE: &[u32] = &[PROFILE, NOTE, DELETION, ARTICLE, WIKI]; +const LISTING_SIZE: usize = 200; +/// Search results, a page at a time (relays give fewer than for listings). +const SEARCH_SIZE: usize = 100; +/// Deletion requests read from a profile's author. +const DELETIONS_PER_PROFILE: usize = 200; +/// A profile's newest posts, of each sort, that become its links. +const POSTS_PER_PROFILE: usize = 50; + +/// What the listing pages say they are; and the posts, worked out already. +const LISTING_TYPE: &str = "application/x-lost-spider-listing"; +const PAGE_TYPE: &str = "application/x-lost-spider-nostr"; + +/// Posts from listings, kept until they're crawled (the oldest go first +/// past this: they're asked for again then). +const MAX_KEPT: usize = 50_000; +/// Names and where posts were seen, remembered up to this many. +const MAX_REMEMBERED: usize = 500_000; +/// Relays asked for a post before it's taken as gone. +const RELAYS_TRIED: usize = 3; +/// A relay that can't be reached this many times in a row isn't asked again. +const MAX_RELAY_FAILURES: u32 = 3; +/// An answer stops here, whatever was asked. +const MAX_EVENTS: usize = 1_000; +/// Notes with fewer letters and digits than this ("gm", "🤙", a link) aren't +/// saved; what they link to is still followed. +const MIN_NOTE_CHARS: usize = 10; +/// A note's title: its author, and its start up to this many characters. +const TITLE_CHARS: usize = 60; + +pub struct Nostr { + proxy: Option, + timeout: Duration, + /// Between two requests to the same relay (`--delay`). + delay: Duration, + tls: OnceLock, String>>, + relays: Mutex>>, + kept: Mutex, + /// Authors' names, by public key (hex). + names: Mutex>, + /// The relay a post or profile was last seen on, by its `nostr:` URI. + hints: Mutex>, + subscriptions: AtomicU64, + /// `--search`'s phrases, asked of the relays that can search. + searches: Vec, + /// For relays' information documents (NIP-11), over HTTP. + http: Http, + /// Authors whose posts aren't saved (`--blacklist`), public keys (hex). + blocked: Mutex>, + /// Posts their authors asked to delete (NIP-09): IDs and addresses + /// (`KIND:PUBKEY:NAME`), each with who asked (`TARGET PUBKEY`): only the + /// author's own request counts. + deleted: Mutex>, +} + +#[derive(Default)] +struct Relay { + state: Mutex, + /// What it says about itself (NIP-11), asked once. + info: OnceLock, +} + +/// What matters of a relay's information document. +#[derive(Debug, Default, Clone, Copy, PartialEq)] +struct Info { + /// It searches (NIP-50). + searches: bool, + /// Nothing can be read without logging in or paying. + closed: bool, +} + +impl Info { + fn of(document: &Value) -> Info { + let searches = document["supported_nips"] + .as_array() + .is_some_and(|nips| nips.iter().any(|n| n.as_u64() == Some(50))); + let limits = &document["limitation"]; + let closed = limits["auth_required"].as_bool() == Some(true) + || limits["payment_required"].as_bool() == Some(true); + Info { searches, closed } + } +} + +struct RelayState { + socket: Option, + /// When it may be asked again. + next: Instant, + /// Connections that failed in a row. + failures: u32, +} + +impl Default for RelayState { + fn default() -> Self { + RelayState { + socket: None, + next: Instant::now(), + failures: 0, + } + } +} + +/// Posts from listings, by URI, with the relay they came from; the oldest +/// first out. +#[derive(Default)] +struct Kept { + events: HashMap, + order: VecDeque, +} + +impl Kept { + fn keep(&mut self, uri: String, event: Event, relay: &str) { + if self.events.contains_key(&uri) { + return; + } + // Those taken already leave their place behind: cleared out now + // and then. + if self.order.len() >= 2 * MAX_KEPT { + let events = &self.events; + self.order.retain(|u| events.contains_key(u)); + } + while self.events.len() >= MAX_KEPT { + let Some(oldest) = self.order.pop_front() else { + break; + }; + self.events.remove(&oldest); + } + self.order.push_back(uri.clone()); + self.events.insert(uri, (event, relay.to_string())); + } + + fn take(&mut self, uri: &str) -> Option<(Event, String)> { + // Its place in `order` goes when it comes up: nothing's there then. + self.events.remove(uri) + } +} + +/// A relay's answer: the events it sent, and why it stopped early if it did +/// (`CLOSED`: "rate-limited: …", "auth-required: …"). +struct Answer { + events: Vec, + closed: Option, +} + +impl Nostr { + pub fn new( + proxy: Option, + timeout: Option, + delay: Duration, + searches: Vec, + ) -> lost::Result { + if let Some(Proxy::Http(_)) = proxy { + return Err("lost-spider nostr goes through socks5:// proxies only".into()); + } + let timeout = timeout.unwrap_or(TIMEOUT); + Ok(Nostr { + http: Http::new(proxy.clone(), timeout)?, + searches, + blocked: Mutex::new(HashSet::new()), + deleted: Mutex::new(HashSet::new()), + proxy, + timeout, + delay, + tls: OnceLock::new(), + relays: Mutex::new(HashMap::new()), + kept: Mutex::new(Kept::default()), + names: Mutex::new(HashMap::new()), + hints: Mutex::new(HashMap::new()), + subscriptions: AtomicU64::new(0), + }) + } + + /// What the relay at `key` says about itself (NIP-11: its address over + /// HTTP, asking for `application/nostr+json`); nothing if it won't say. + fn info(&self, key: &str) -> Info { + *self.relay(key).info.get_or_init(|| { + let Some(mut url) = Url::parse(key) else { + return Info::default(); + }; + url.scheme = if url.scheme == "ws" { "http" } else { "https" }.into(); + let route = match &self.proxy { + Some(_) => Some(Route::Proxied), + None => { + use std::net::ToSocketAddrs; + let port = url + .port + .unwrap_or(if url.scheme == "https" { 443 } else { 80 }); + (url.host.as_str(), port) + .to_socket_addrs() + .ok() + .and_then(|mut a| a.next()) + .map(Route::Direct) + } + }; + let answer = route + .and_then(|route| self.http.get_as(route, &url, "application/nostr+json").ok()); + match answer { + Some(answer) if answer.status == 200 => serde_json::from_slice(&answer.body) + .map(|document: Value| Info::of(&document)) + .unwrap_or_default(), + _ => Info::default(), + } + }) + } + + /// Takes the authors named in `blacklist`; how many there are. + fn block_authors(&self, blacklist: &Blacklist) -> usize { + let mut blocked = self.blocked.lock().unwrap_or_else(|e| e.into_inner()); + for name in blacklist.names() { + let entity = name.strip_prefix("nostr:").unwrap_or(name); + if let Some(Entity::Profile { pubkey, .. }) = Entity::decode(entity) { + blocked.insert(hex(&pubkey)); + } + } + blocked.len() + } + + /// A post that may be saved: signed, not expiring, not by a blocked + /// author, not deleted by its author. + fn wanted(&self, event: &Event) -> bool { + if !usable(event) { + return false; + } + if self + .blocked + .lock() + .unwrap_or_else(|e| e.into_inner()) + .contains(&event.pubkey) + { + return false; + } + let deleted = self.deleted.lock().unwrap_or_else(|e| e.into_inner()); + let by_author = |target: &str| deleted.contains(&format!("{target} {}", event.pubkey)); + !(by_author(&event.id) || event.coordinate().is_some_and(|a| by_author(&a))) + } + + /// Takes in the deletion requests among `events` (checked: signed). + fn note_deletions(&self, events: &[Event]) { + let mut deleted = self.deleted.lock().unwrap_or_else(|e| e.into_inner()); + for event in events.iter().filter(|e| e.kind == DELETION && usable(e)) { + if deleted.len() >= MAX_REMEMBERED { + deleted.clear(); + } + for tag in &event.tags { + if let (Some(name), Some(value)) = (tag.first(), tag.get(1)) + && (name == "e" || name == "a") + { + deleted.insert(format!("{value} {}", event.pubkey)); + } + } + } + } + + fn relay(&self, key: &str) -> Arc { + let mut relays = self.relays.lock().unwrap_or_else(|e| e.into_inner()); + relays.entry(key.to_string()).or_default().clone() + } + + fn connect(&self, key: &str) -> Result { + let url = Url::parse(key).ok_or_else(|| format!("\"{key}\" isn't a relay"))?; + let tls = match url.scheme.as_str() { + "wss" => Some(self.tls.get_or_init(net::any_certificate).clone()?), + _ => None, + }; + WebSocket::connect(&url, self.proxy.as_ref(), self.timeout, tls.as_ref()) + } + + /// Asks the relay at `key` for the events matching `filters`, once + /// it's had `--delay` since the last time. A connection that was open + /// already and fails is opened again, once. + fn ask(&self, key: &str, filters: &[Value]) -> Result { + let relay = self.relay(key); + let mut state = relay.state.lock().unwrap_or_else(|e| e.into_inner()); + if state.failures >= MAX_RELAY_FAILURES { + return Err("the relay can't be reached".into()); + } + let wait = state.next.saturating_duration_since(Instant::now()); + if !wait.is_zero() { + std::thread::sleep(wait); + } + let result = loop { + let reused = state.socket.is_some(); + if !reused { + match self.connect(key) { + Ok(socket) => { + state.socket = Some(socket); + state.failures = 0; + } + Err(e) => { + state.failures += 1; + break Err(e); + } + } + } + let socket = state.socket.as_mut().expect("connected"); + match self.subscribe(socket, filters) { + Ok(answer) => break Ok(answer), + Err(e) => { + state.socket = None; + if !reused { + break Err(e); + } + } + } + }; + state.next = Instant::now() + self.delay; + result + } + + /// One subscription: `REQ`, the events up to `EOSE` (or `CLOSED`), + /// `CLOSE`. + fn subscribe(&self, socket: &mut WebSocket, filters: &[Value]) -> Result { + let id = format!("lost{}", self.subscriptions.fetch_add(1, Ordering::Relaxed)); + let mut request = vec![json!("REQ"), json!(id)]; + request.extend(filters.iter().cloned()); + socket.send(&Value::Array(request).to_string())?; + let deadline = Instant::now() + self.timeout * 4; + let mut events = Vec::new(); + loop { + let text = socket.receive(deadline)?; + let Ok(Value::Array(message)) = serde_json::from_str::(&text) else { + continue; + }; + let ours = message.get(1).and_then(Value::as_str) == Some(id.as_str()); + match message.first().and_then(Value::as_str) { + Some("EVENT") if ours => { + if let Some(event) = message.get(2).and_then(Event::from_json) { + events.push(event); + } + if events.len() >= MAX_EVENTS { + socket.send(&json!(["CLOSE", id]).to_string())?; + return Ok(Answer { + events, + closed: None, + }); + } + } + Some("EOSE") if ours => { + socket.send(&json!(["CLOSE", id]).to_string())?; + return Ok(Answer { + events, + closed: None, + }); + } + Some("CLOSED") if ours => { + let reason = message.get(2).and_then(Value::as_str).unwrap_or(""); + return Ok(Answer { + events, + closed: Some(reason.to_string()), + }); + } + // NOTICE, AUTH (no logging in here), answers to others. + _ => {} + } + } + } + + /// Relays to ask for `uri`: where it was seen, those it names, and some + /// of the seeds (a different few for each, to share the asking out). + fn relays_for(&self, uri: &str, named: &[String]) -> Vec { + let mut out: Vec = Vec::new(); + let seen = self + .hints + .lock() + .unwrap_or_else(|e| e.into_inner()) + .get(uri) + .cloned(); + let start = uri.bytes().map(usize::from).sum::() % SEEDS.len(); + let seeds = (0..SEEDS.len()).map(|i| SEEDS[(start + i) % SEEDS.len()].to_string()); + for relay in seen + .into_iter() + .chain(named.iter().filter_map(|r| relay_key(r))) + .chain(seeds) + { + if !out.contains(&relay) { + out.push(relay); + } + } + out.truncate(RELAYS_TRIED); + out + } + + /// The newest event matching `filters` on the relays for `uri`, and the + /// relay it came from; `None` if none of them has one. + fn look_up( + &self, + uri: &str, + named: &[String], + filters: &[Value], + ) -> Result, String> { + let mut problem = None; + let mut answered = false; + for relay in self.relays_for(uri, named) { + match self.ask(&relay, filters) { + Ok(answer) => { + answered = true; + self.note_deletions(&answer.events); + let best = answer + .events + .into_iter() + .filter(|e| e.kind != DELETION && self.wanted(e)) + .max_by_key(|e| e.created_at); + if let Some(event) = best { + return Ok(Some((event, relay))); + } + } + Err(e) => problem = Some(e), + } + } + match problem { + Some(e) if !answered => Err(e), + _ => Ok(None), + } + } + + fn remember_name(&self, event: &Event) { + if event.kind != PROFILE { + return; + } + let Some(name) = Profile::of(event).name() else { + return; + }; + let mut names = self.names.lock().unwrap_or_else(|e| e.into_inner()); + if names.len() >= MAX_REMEMBERED { + names.clear(); + } + names.insert(event.pubkey.clone(), name); + } + + fn remember_hint(&self, uri: &str, relay: &str) { + let mut hints = self.hints.lock().unwrap_or_else(|e| e.into_inner()); + if hints.len() >= MAX_REMEMBERED { + hints.clear(); + } + hints.insert(uri.to_string(), relay.to_string()); + } + + fn name_of(&self, pubkey: &str) -> Option { + self.names + .lock() + .unwrap_or_else(|e| e.into_inner()) + .get(pubkey) + .cloned() + } + + /// `wss://RELAY/`: its listings, and its searches for `--search`'s + /// phrases if it can search. `wss://RELAY/?kinds=…&until=…` (and + /// `&search=…`): a page of one, newest first. + fn listing(&self, url: &Url) -> Result { + let Some(relay) = relay_key(&url.to_string()) else { + return Ok(Response::Gone); + }; + let Some(query) = &url.query else { + let info = self.info(&relay); + if info.closed { + return Ok(Response::Gone); + } + let mut links: Vec = LISTS + .iter() + .map(|kinds| Link { + url: format!("{relay}?kinds={kinds}"), + label: None, + }) + .collect(); + if info.searches { + for phrase in &self.searches { + links.push(Link { + url: format!("{relay}?kinds={SEARCHED}&search={}", encode_query(phrase)), + label: None, + }); + } + } + return Ok(page(LISTING_TYPE, None, String::new(), links, true)); + }; + let Some(list) = List::of(query) else { + return Ok(Response::Gone); + }; + let size = if list.search.is_some() { + SEARCH_SIZE + } else { + LISTING_SIZE + }; + let mut filter = json!({ "kinds": list.kinds, "limit": size }); + if let Some(until) = list.until { + filter["until"] = json!(until); + } + if let Some(search) = &list.search { + filter["search"] = json!(search); + } + let answer = self.ask(&relay, &[filter])?; + if let Some(reason) = &answer.closed + && answer.events.is_empty() + { + return Ok(refused(reason)); + } + let oldest = answer.events.iter().map(|e| e.created_at).min(); + self.note_deletions(&answer.events); + let events: Vec = answer + .events + .into_iter() + .filter(|e| e.kind != DELETION && self.wanted(e)) + .collect(); + + // Names for the notes' titles: those not known yet, in one go. + let unknown: HashSet<&str> = events + .iter() + .map(|e| e.pubkey.as_str()) + .filter(|p| self.name_of(p).is_none()) + .collect(); + if !unknown.is_empty() { + let authors: Vec<&str> = unknown.into_iter().collect(); + let filter = json!({ "kinds": [PROFILE], "authors": authors }); + if let Ok(answer) = self.ask(&relay, &[filter]) { + for profile in answer.events.iter().filter(|e| usable(e)) { + self.remember_name(profile); + } + } + } + + let mut links = Vec::new(); + { + let mut kept = self.kept.lock().unwrap_or_else(|e| e.into_inner()); + for event in events { + let Some(uri) = event.uri() else { + continue; + }; + self.remember_hint(&uri, &relay); + // A profile (found by a search) is asked for whole later. + if event.kind == PROFILE { + self.remember_name(&event); + let label = self.name_of(&event.pubkey); + links.push(Link { url: uri, label }); + continue; + } + let label = event.title_tag().map(str::to_string); + kept.keep(uri.clone(), event, &relay); + links.push(Link { url: uri, label }); + } + } + if let Some(oldest) = oldest { + // Posts of the same second as the last page's oldest may not + // all have fit: that second again, unless it's all there was. + let until = if list.until == Some(oldest) { + oldest.checked_sub(1) + } else { + Some(oldest) + }; + if let Some(until) = until { + links.push(Link { + url: format!( + "{relay}?{}", + List { + until: Some(until), + ..list + } + .query() + ), + label: None, + }); + } + } + Ok(page(LISTING_TYPE, None, String::new(), links, true)) + } + + /// A profile: who it is, and its newest posts and relays as links. + fn profile(&self, uri: &str, pubkey: &str, named: &[String]) -> Result { + let filters = [ + json!({ "kinds": [PROFILE, RELAY_LIST], "authors": [pubkey] }), + json!({ "kinds": [ARTICLE, WIKI], "authors": [pubkey], "limit": POSTS_PER_PROFILE }), + json!({ "kinds": [NOTE], "authors": [pubkey], "limit": POSTS_PER_PROFILE }), + json!({ "kinds": [DELETION], "authors": [pubkey], "limit": DELETIONS_PER_PROFILE }), + ]; + let mut problem = None; + let mut best: Option<(Vec, String)> = None; + for relay in self.relays_for(uri, named) { + match self.ask(&relay, &filters) { + Ok(answer) => { + self.note_deletions(&answer.events); + let events: Vec = answer + .events + .into_iter() + .filter(|e| e.pubkey == pubkey && e.kind != DELETION && self.wanted(e)) + .collect(); + let has_profile = events.iter().any(|e| e.kind == PROFILE); + let better = best.as_ref().is_none_or(|(b, _)| events.len() > b.len()); + if has_profile || better { + best = Some((events, relay)); + } + if has_profile { + break; + } + } + Err(e) => problem = Some(e), + } + } + let Some((events, relay)) = best else { + return match problem { + Some(e) => Err(e), + None => Ok(Response::Gone), + }; + }; + if events.is_empty() { + return Ok(Response::Gone); + } + let newest = |kind: u32| { + events + .iter() + .filter(|e| e.kind == kind) + .max_by_key(|e| e.created_at) + }; + let profile = newest(PROFILE).map(Profile::of).unwrap_or_default(); + if let Some(event) = newest(PROFILE) { + self.remember_name(event); + } + let mut links = Vec::new(); + if let Some(website) = profile.website() { + links.push(Link { + url: website, + label: None, + }); + } + if let Some(list) = newest(RELAY_LIST) { + for tag in &list.tags { + if tag.first().map(String::as_str) == Some("r") + && let Some(relay) = tag.get(1).and_then(|r| relay_key(r)) + { + links.push(Link { + url: relay, + label: None, + }); + } + } + } + { + let mut kept = self.kept.lock().unwrap_or_else(|e| e.into_inner()); + let mut posts: Vec<&Event> = events + .iter() + .filter(|e| matches!(e.kind, NOTE | ARTICLE | WIKI)) + .collect(); + posts.sort_by_key(|e| std::cmp::Reverse(e.created_at)); + for post in posts { + let Some(post_uri) = post.uri() else { + continue; + }; + self.remember_hint(&post_uri, &relay); + links.push(Link { + url: post_uri.clone(), + label: post.title_tag().map(str::to_string), + }); + kept.keep(post_uri, post.clone(), &relay); + } + } + let Some(name) = profile.name() else { + // No profile: only its posts, followed. + return Ok(page(PAGE_TYPE, None, String::new(), links, true)); + }; + Ok(page(PAGE_TYPE, Some(name), profile.text(), links, false)) + } + + /// A post (seen on `relay`) as a page. + fn post(&self, event: &Event, relay: &str) -> Response { + let author = self + .name_of(&event.pubkey) + .unwrap_or_else(|| short(&npub(&event.pubkey))); + let author_uri = npub_uri(&event.pubkey); + // Its author's profile is looked for where the post was. + self.remember_hint(&author_uri, relay); + let mut links = vec![Link { + url: author_uri, + label: Some(author.clone()), + }]; + for (uri, named) in tag_links(event) { + if let Some(relay) = named { + self.remember_hint(&uri, &relay); + } + if !links.iter().any(|l| l.url == uri) { + links.push(Link { + url: uri, + label: None, + }); + } + } + // URLs are read from what's shown (not an article's image sources). + let (title, body, skip) = match event.kind { + NOTE | COMMENT => { + let text = self.with_names(&event.content); + let start = lost::text::single_line(&text); + let title = format!("{author}: {}", cut(&start, TITLE_CHARS)); + let words = strip_references(&event.content) + .chars() + .filter(|c| c.is_alphanumeric()) + .count(); + (title, text, words < MIN_NOTE_CHARS) + } + ARTICLE => { + let (text, written) = markdown_to_text(&event.content); + links.extend(written); + let title = event + .title_tag() + .map(str::to_string) + .unwrap_or_else(|| cut(&lost::text::single_line(&text), TITLE_CHARS)); + let body = match event.tag("summary").filter(|s| !s.trim().is_empty()) { + Some(summary) => format!("{}\n\n{text}", summary.trim()), + None => text, + }; + (title, with_topics(body, event), false) + } + WIKI => { + let text = asciidoc_to_text(&event.content); + let title = event + .title_tag() + .or_else(|| event.tag("d")) + .map(str::to_string) + .unwrap_or_else(|| cut(&lost::text::single_line(&text), TITLE_CHARS)); + (title, with_topics(text, event), false) + } + // A repost: what it reposts. + REPOST => (String::new(), String::new(), true), + _ => return Response::Gone, + }; + // References that say where to look: remembered. + for (_, _, entity) in references(&event.content) { + let named = match &entity { + Entity::Profile { relays, .. } + | Entity::Event { relays, .. } + | Entity::Address { relays, .. } => relays.iter().find_map(|r| relay_key(r)), + }; + if let Some(relay) = named { + self.remember_hint(&entity.uri(), &relay); + } + } + let mut found = nostr_links(&event.content); + let shown = if event.kind == ARTICLE { + &body + } else { + &event.content + }; + found.extend(mentions(shown, "https")); + found.extend(mentions(shown, "http")); + for link in found { + if !links.iter().any(|l| l.url == link.url) { + links.push(link); + } + } + if skip { + return page(PAGE_TYPE, None, String::new(), links, true); + } + page(PAGE_TYPE, Some(title), body, links, false) + } + + /// `nostr:npub1…` mentions written as `@name`, for the names known. + fn with_names(&self, text: &str) -> String { + let mut out = String::with_capacity(text.len()); + let mut last = 0; + for (start, end, entity) in references(text) { + if let Entity::Profile { pubkey, .. } = entity + && let Some(name) = self.name_of(&hex(&pubkey)) + { + out.push_str(&text[last..start]); + out.push('@'); + out.push_str(&name); + last = end; + } + } + out.push_str(&text[last..]); + out + } +} + +/// A listing's page: what it asks for. +#[derive(Debug, PartialEq)] +struct List { + kinds: Vec, + until: Option, + /// A search's words (NIP-50). + search: Option, +} + +impl List { + fn of(query: &str) -> Option { + let mut kinds = None; + let mut until = None; + let mut search = None; + for pair in query.split('&') { + match pair.split_once('=') { + Some(("search", value)) => { + search = Some(percent_decode(value)).filter(|s| !s.trim().is_empty()); + search.as_ref()?; + } + Some(("kinds", value)) => { + let list: Option> = value + .split([',', '+']) + .map(|k| k.parse().ok().filter(|k| LISTABLE.contains(k))) + .collect(); + kinds = Some(list?); + } + Some(("until", value)) => until = Some(value.parse().ok()?), + _ => return None, + } + } + let kinds = kinds.filter(|k| !k.is_empty())?; + Some(List { + kinds, + until, + search, + }) + } + + /// Written as a query again: `kinds=…`, then `search=…`, `until=…`. + fn query(&self) -> String { + let kinds: Vec = self.kinds.iter().map(u32::to_string).collect(); + let mut query = format!("kinds={}", kinds.join(",")); + if let Some(search) = &self.search { + query.push_str(&format!("&search={}", encode_query(search))); + } + if let Some(until) = self.until { + query.push_str(&format!("&until={until}")); + } + query + } +} + +/// What a relay's `CLOSED` means for the page that asked. +fn refused(reason: &str) -> Response { + if reason.starts_with("rate-limited") { + Response::SlowDown(Duration::from_secs(60)) + } else { + Response::Gone + } +} + +/// A page worked out here, for [`Protocol::parse`] to read back. +fn page(mime: &str, title: Option, body: String, links: Vec, skip: bool) -> Response { + let links: Vec = links.into_iter().map(|l| json!([l.url, l.label])).collect(); + let page = json!({ "title": title, "body": body, "links": links, "skip": skip }); + Response::Document { + mime: mime.to_string(), + body: Some(page.to_string().into_bytes()), + } +} + +impl Protocol for Nostr { + const NAME: &'static str = "nostr"; + /// `--link npub1…`: `nostr:npub1…`. + const SCHEME: &'static str = "nostr"; + const SPACE: &'static str = "Nostr"; + const HOSTS: &'static str = "items"; + + fn seeds(&self) -> &'static [&'static str] { + if self.searches.is_empty() { + SEEDS + } else { + SEEDS_AND_SEARCH + } + } + + fn proxy(&self) -> Option<&Proxy> { + self.proxy.as_ref() + } + + /// Every post, profile and relay is a server of its own to the crawl: + /// posts are mostly kept here from listings already, and relays are + /// asked one thing at a time here (`--delay`). + fn proxied(&self) -> bool { + true + } + + fn accepts(url: &Url) -> bool { + match url.scheme.as_str() { + "ws" | "wss" => !url.host.is_empty(), + "nostr" => Entity::decode(&url.host).is_some_and(|e| e.uri() == url.to_string()), + _ => false, + } + } + + /// Relays are servers for programs, and posts aren't sites. + fn has_robots(_origin: &Url) -> bool { + false + } + + /// `nostr:nevent1…`, `nprofile1…`, a bare `npub1…`: the way posts and + /// profiles are saved (`note1…`, `npub1…`, `naddr1…` without relays). + fn rewrite(link: &str) -> Option { + let url = Url::parse(link)?; + if url.scheme != "nostr" { + return None; + } + Entity::decode(&url.host).map(|e| e.uri()) + } + + fn fallback_title(url: &Url) -> String { + short(&url.host) + } + + fn fetch(&self, _route: Route, url: &Url) -> Result { + if url.scheme == "ws" || url.scheme == "wss" { + return self.listing(url); + } + let Some(entity) = Entity::decode(&url.host) else { + return Ok(Response::Gone); + }; + let uri = entity.uri(); + if uri != url.to_string() { + return Ok(Response::Redirect(uri)); + } + match &entity { + Entity::Profile { pubkey, relays } => self.profile(&uri, &hex(pubkey), relays), + Entity::Event { id, relays, .. } => { + let kept = self + .kept + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take(&uri); + let found = match kept { + // Its author may have asked to delete it since. + Some((event, _)) if !self.wanted(&event) => None, + Some(found) => Some(found), + None => self.look_up( + &uri, + relays, + &[ + json!({ "ids": [hex(id)] }), + json!({ "kinds": [DELETION], "#e": [hex(id)] }), + ], + )?, + }; + let Some((event, relay)) = found else { + return Ok(Response::Gone); + }; + // Saved under its own name: an article's, a profile's. + match event.uri() { + Some(own) if own != uri => { + self.kept.lock().unwrap_or_else(|e| e.into_inner()).keep( + own.clone(), + event, + &relay, + ); + Ok(Response::Redirect(own)) + } + Some(_) => Ok(self.post(&event, &relay)), + None => Ok(Response::Gone), + } + } + Entity::Address { + kind, + author, + identifier, + relays, + } => { + let kept = self + .kept + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take(&uri); + let coordinate = format!("{kind}:{}:{identifier}", hex(author)); + let found = match kept { + Some((event, _)) if !self.wanted(&event) => None, + Some(found) => Some(found), + None => self.look_up( + &uri, + relays, + &[ + json!({ "kinds": [kind], "authors": [hex(author)], "#d": [identifier] }), + json!({ "kinds": [DELETION], "#a": [coordinate] }), + ], + )?, + }; + match found { + Some((event, relay)) => Ok(self.post(&event, &relay)), + None => Ok(Response::Gone), + } + } + } + } + + fn parse(&self, _url: &Url, mime: &str, body: &[u8]) -> Parsed { + if mime != PAGE_TYPE && mime != LISTING_TYPE { + return Parsed::default(); + } + let Ok(page) = serde_json::from_slice::(body) else { + return Parsed::default(); + }; + let links = page["links"] + .as_array() + .map(|links| { + links + .iter() + .filter_map(|link| { + Some(Link { + url: link[0].as_str()?.to_string(), + label: link[1].as_str().map(str::to_string), + }) + }) + .collect() + }) + .unwrap_or_default(); + Parsed { + title: page["title"].as_str().map(str::to_string), + body: page["body"].as_str().unwrap_or("").to_string(), + links, + skip: page["skip"].as_bool().unwrap_or(true), + } + } + + /// Some relay must answer. Authors in `--blacklist` (`npub1…`, + /// `nostr:npub1…`, `nprofile1…`) have their posts skipped. + fn prepare(&self, blacklist: &mut Blacklist) -> lost::Result<()> { + match self.block_authors(blacklist) { + 0 => {} + 1 => eprintln!("Skipping the posts of 1 blacklisted author."), + n => eprintln!("Skipping the posts of {n} blacklisted authors."), + } + if !self.searches.is_empty() { + eprintln!( + "Searching for {} on the relays that can search (NIP-50).", + match self.searches.len() { + 1 => "1 phrase".to_string(), + n => format!("{n} phrases"), + } + ); + } + let reached = std::thread::scope(|scope| { + let tries: Vec<_> = SEEDS + .iter() + .map(|seed| scope.spawn(move || (seed, self.connect(seed)))) + .collect(); + let mut reached = 0; + let mut problem = None; + for attempt in tries { + let Ok((seed, result)) = attempt.join() else { + continue; + }; + match result { + Ok(socket) => { + let relay = self.relay(seed); + relay.state.lock().unwrap_or_else(|e| e.into_inner()).socket = Some(socket); + reached += 1; + } + Err(e) => problem = Some(format!("{seed}: {e}")), + } + } + (reached, problem) + }); + match reached { + (0, problem) => Err(format!( + "can't reach any Nostr relay ({})", + problem.unwrap_or_default() + ) + .into()), + (n, _) => { + eprintln!("Reached {n} of {} relays.", SEEDS.len()); + Ok(()) + } + } + } +} + +// ── Posts ─────────────────────────────────────────────────────────────── + +/// A post (NIP-01). +#[derive(Debug, Clone, PartialEq)] +struct Event { + /// Hex. + id: String, + pubkey: String, + created_at: u64, + kind: u32, + tags: Vec>, + content: String, + sig: String, +} + +impl Event { + fn from_json(value: &Value) -> Option { + let text = |name: &str| value[name].as_str().map(str::to_string); + let tags = value["tags"] + .as_array()? + .iter() + .map(|tag| { + tag.as_array()? + .iter() + .map(|v| v.as_str().map(str::to_string)) + .collect::>>() + }) + .collect::>>>()?; + Some(Event { + id: text("id")?, + pubkey: text("pubkey")?, + created_at: value["created_at"].as_u64()?, + kind: u32::try_from(value["kind"].as_u64()?).ok()?, + tags, + content: text("content")?, + sig: text("sig")?, + }) + } + + /// Whether the ID is the post's (the SHA-256 of what NIP-01 says to + /// write out) and its author signed it (BIP-340). + fn is_signed(&self) -> bool { + let (Some(id), Some(pubkey), Some(sig)) = ( + unhex::<32>(&self.id), + unhex::<32>(&self.pubkey), + unhex::<64>(&self.sig), + ) else { + return false; + }; + let written = json!([ + 0, + self.pubkey, + self.created_at, + self.kind, + self.tags, + self.content + ]) + .to_string(); + if Sha256::digest(written.as_bytes()).as_slice() != id.as_slice() { + return false; + } + let (Ok(key), Ok(signature)) = ( + VerifyingKey::from_slice(&pubkey), + Signature::from_slice(&sig), + ) else { + return false; + }; + key.verify_raw(&id, &signature).is_ok() + } + + /// The first value of the first tag named `name`. + fn tag(&self, name: &str) -> Option<&str> { + self.tags + .iter() + .find(|t| t.first().map(String::as_str) == Some(name)) + .and_then(|t| t.get(1)) + .map(String::as_str) + } + + /// An article's or wiki entry's address, `KIND:PUBKEY:NAME`, as + /// deletion requests write it. + fn coordinate(&self) -> Option { + (30_000..40_000).contains(&self.kind).then(|| { + format!( + "{}:{}:{}", + self.kind, + self.pubkey, + self.tag("d").unwrap_or("") + ) + }) + } + + fn title_tag(&self) -> Option<&str> { + self.tag("title").filter(|t| !t.trim().is_empty()) + } + + /// The post's `nostr:` URI, as it's saved: a profile's key, an + /// article's or wiki entry's address (kind, author, name), any other + /// post's ID. + fn uri(&self) -> Option { + let pubkey = unhex::<32>(&self.pubkey)?; + let entity = if self.kind == PROFILE { + Entity::Profile { + pubkey, + relays: Vec::new(), + } + } else if (30_000..40_000).contains(&self.kind) { + Entity::Address { + kind: self.kind, + author: pubkey, + identifier: self.tag("d").unwrap_or("").to_string(), + relays: Vec::new(), + } + } else { + Entity::Event { + id: unhex::<32>(&self.id)?, + relays: Vec::new(), + author: None, + kind: None, + } + }; + // An address whose name is too long for one (over 255 bytes) has none. + let uri = entity.uri(); + Entity::decode(uri.strip_prefix("nostr:")?).map(|_| uri) + } +} + +/// Signed, and not asking to expire (NIP-40: whatever the date, it was +/// meant to go away, so it isn't kept in a file for good). +fn usable(event: &Event) -> bool { + event.tag("expiration").is_none() && event.is_signed() +} + +/// A profile's fields (kind 0's content, JSON). +#[derive(Debug, Default)] +struct Profile { + name: Option, + display_name: Option, + about: Option, + website: Option, + nip05: Option, +} + +impl Profile { + fn of(event: &Event) -> Profile { + let Ok(fields) = serde_json::from_str::(&event.content) else { + return Profile::default(); + }; + let field = |name: &str| { + fields[name] + .as_str() + .map(str::trim) + .filter(|v| !v.is_empty()) + .map(str::to_string) + }; + Profile { + name: field("name"), + display_name: field("display_name").or_else(|| field("displayName")), + about: field("about"), + website: field("website"), + nip05: field("nip05"), + } + } + + fn name(&self) -> Option { + self.display_name + .clone() + .or_else(|| self.name.clone()) + .map(|n| lost::text::single_line(&n)) + .filter(|n| !n.is_empty()) + } + + fn website(&self) -> Option { + let site = self.website.as_deref()?; + let site = if site.contains("://") { + site.to_string() + } else { + format!("https://{site}") + }; + Url::parse(&site) + .filter(|u| u.scheme == "http" || u.scheme == "https") + .map(|u| u.to_string()) + } + + fn text(&self) -> String { + let mut text = self.about.clone().unwrap_or_default(); + for extra in [&self.name, &self.nip05, &self.website] + .into_iter() + .flatten() + { + if !text.contains(extra.as_str()) { + if !text.is_empty() { + text.push('\n'); + } + text.push_str(extra); + } + } + text + } +} + +/// What a post's tags point at: profiles (`p`), posts (`e`, `q`) and +/// addresses (`a`), with the relay named there, if any. +fn tag_links(event: &Event) -> Vec<(String, Option)> { + let mut out: Vec<(String, Option)> = Vec::new(); + for tag in &event.tags { + let (Some(name), Some(value)) = (tag.first(), tag.get(1)) else { + continue; + }; + let relay = tag.get(2).and_then(|r| relay_key(r)); + let entity = match name.as_str() { + "p" => unhex::<32>(value).map(|pubkey| Entity::Profile { + pubkey, + relays: Vec::new(), + }), + "e" | "q" => unhex::<32>(value).map(|id| Entity::Event { + id, + relays: Vec::new(), + author: None, + kind: None, + }), + "a" => address(value), + _ => None, + }; + // A `q` can be an address too. + let entity = entity.or_else(|| (name == "q").then(|| address(value)).flatten()); + if let Some(uri) = entity.map(|e| e.uri()) + && !out.iter().any(|(u, _)| *u == uri) + { + out.push((uri, relay)); + } + } + out +} + +/// `KIND:PUBKEY:NAME`, an `a` tag's address. +fn address(text: &str) -> Option { + let mut parts = text.splitn(3, ':'); + let kind = parts.next()?.parse().ok()?; + let author = unhex::<32>(parts.next()?)?; + let identifier = parts.next()?.to_string(); + Some(Entity::Address { + kind, + author, + identifier, + relays: Vec::new(), + }) +} + +/// A relay's address as it's kept: `wss://HOST[:PORT]/PATH`, nothing after. +fn relay_key(text: &str) -> Option { + let mut url = Url::parse(text.trim())?; + if url.scheme != "wss" && url.scheme != "ws" { + return None; + } + url.query = None; + url.fragment = None; + Some(url.to_string()) +} + +// ── Text ──────────────────────────────────────────────────────────────── + +/// `nostr:` references written in `text`: where each is, and what it names. +fn references(text: &str) -> Vec<(usize, usize, Entity)> { + let mut out = Vec::new(); + let lower = text.to_ascii_lowercase(); + let mut from = 0; + while let Some(i) = lower[from..].find("nostr:") { + let start = from + i; + let entity_start = start + "nostr:".len(); + let end = entity_start + + lower[entity_start..] + .find(|c: char| !c.is_ascii_alphanumeric()) + .unwrap_or(lower.len() - entity_start); + if let Some(entity) = Entity::decode(&lower[entity_start..end]) { + out.push((start, end, entity)); + } + from = end.max(entity_start); + } + out +} + +/// The posts and profiles `text` refers to, as links. +fn nostr_links(text: &str) -> Vec { + let mut links: Vec = Vec::new(); + for (_, _, entity) in references(text) { + let url = entity.uri(); + if !links.iter().any(|l| l.url == url) { + links.push(Link { url, label: None }); + } + } + links +} + +/// `text` without its `nostr:` references and URLs: what's left to read. +fn strip_references(text: &str) -> String { + text.split_whitespace() + .filter(|w| { + let w = w.to_ascii_lowercase(); + !w.contains("nostr:") && !w.contains("://") + }) + .collect::>() + .join(" ") +} + +/// Markdown as text to read, and its links: `[text](url)` keeps its text, +/// an image its description; `#`s, `**`, `` ` ``s and the like go. +fn markdown_to_text(markdown: &str) -> (String, Vec) { + let mut links: Vec = Vec::new(); + let mut lines = Vec::new(); + for line in markdown.lines() { + let trimmed = line.trim_start(); + if trimmed.starts_with("```") || trimmed.starts_with("~~~") { + continue; + } + let line = trimmed.trim_start_matches('#').trim_start_matches('>'); + let line = if line.len() < trimmed.len() { + line.trim_start() + } else { + line + }; + let mut out = String::with_capacity(line.len()); + let mut rest = line; + while let Some(open) = rest.find('[') { + let image = open > 0 && rest[..open].ends_with('!'); + let Some(close) = rest[open..].find("](").map(|c| open + c) else { + break; + }; + let Some(end) = rest[close + 2..].find(')').map(|e| close + 2 + e) else { + break; + }; + out.push_str(&rest[..if image { open - 1 } else { open }]); + out.push_str(&rest[open + 1..close]); + let target = rest[close + 2..end].split_whitespace().next().unwrap_or(""); + if !image + && (target.starts_with("http://") + || target.starts_with("https://") + || target.starts_with("nostr:")) + { + let url = match Url::parse(target) { + Some(u) if u.scheme == "nostr" => Entity::decode(&u.host).map(|e| e.uri()), + Some(_) => Some(target.to_string()), + None => None, + }; + if let Some(url) = url + && !links.iter().any(|l| l.url == url) + { + links.push(Link { + url, + label: Some(rest[open + 1..close].to_string()).filter(|l| !l.is_empty()), + }); + } + } + rest = &rest[end + 1..]; + } + out.push_str(rest); + let out = out.replace("**", "").replace("__", "").replace("~~", ""); + lines.push(out.replace('`', "")); + } + (lines.join("\n").trim().to_string(), links) +} + +/// AsciiDoc (wiki entries) as text to read: `=` headings and `[[links]]` +/// as their words. +fn asciidoc_to_text(text: &str) -> String { + text.lines() + .map(|line| { + let trimmed = line.trim_start(); + let line = if trimmed.starts_with('=') { + trimmed.trim_start_matches('=').trim_start() + } else { + line + }; + let mut out = String::with_capacity(line.len()); + let mut rest = line; + while let Some(open) = rest.find("[[") { + let Some(close) = rest[open..].find("]]").map(|c| open + c) else { + break; + }; + out.push_str(&rest[..open]); + let inside = &rest[open + 2..close]; + out.push_str(inside.rsplit('|').next().unwrap_or(inside)); + rest = &rest[close + 2..]; + } + out.push_str(rest); + out + }) + .collect::>() + .join("\n") + .trim() + .to_string() +} + +/// `text`, and its post's topics (`t` tags) as `#hashtags` at the end, those +/// it doesn't have already: an article's topics are often only there. +fn with_topics(mut text: String, event: &Event) -> String { + let lower = text.to_lowercase(); + let mut topics: Vec = Vec::new(); + for tag in &event.tags { + if let (Some("t"), Some(topic)) = (tag.first().map(String::as_str), tag.get(1)) { + let topic = topic.trim().trim_start_matches('#'); + let hashtag = format!("#{topic}"); + if !topic.is_empty() + && !topic.contains(char::is_whitespace) + && !lower.contains(&hashtag.to_lowercase()) + && !topics.contains(&hashtag) + { + topics.push(hashtag); + } + } + } + if !topics.is_empty() { + if !text.is_empty() { + text.push_str("\n\n"); + } + text.push_str(&topics.join(" ")); + } + text +} + +/// `text` cut to `max` characters, at a word, with "…" if anything went. +fn cut(text: &str, max: usize) -> String { + if text.chars().count() <= max { + return text.to_string(); + } + let end = text.char_indices().nth(max).map_or(text.len(), |(i, _)| i); + let head = &text[..end]; + let head = match head.rfind(' ') { + Some(i) if i > end / 2 => &head[..i], + _ => head, + }; + format!("{}…", head.trim_end()) +} + +/// `npub1abcdefgh…wxyz`: an entity short enough for a title. +fn short(entity: &str) -> String { + let chars: Vec = entity.chars().collect(); + if chars.len() <= 20 { + return entity.to_string(); + } + let head: String = chars[..12].iter().collect(); + let tail: String = chars[chars.len() - 4..].iter().collect(); + format!("{head}…{tail}") +} + +fn npub(pubkey: &str) -> String { + unhex::<32>(pubkey).map_or_else(|| pubkey.to_string(), |k| bech32_encode("npub", &k)) +} + +fn npub_uri(pubkey: &str) -> String { + format!("nostr:{}", npub(pubkey)) +} + +// ── Names: NIP-19 ─────────────────────────────────────────────────────── + +/// What a `nostr:` URI names. Relays are only hints of where to look. +#[derive(Debug, Clone, PartialEq)] +enum Entity { + /// `npub1…` / `nprofile1…`. + Profile { + pubkey: [u8; 32], + relays: Vec, + }, + /// `note1…` / `nevent1…`. + Event { + id: [u8; 32], + relays: Vec, + author: Option<[u8; 32]>, + kind: Option, + }, + /// `naddr1…`: the newest of an author's posts of a kind under a name. + Address { + kind: u32, + author: [u8; 32], + identifier: String, + relays: Vec, + }, +} + +impl Entity { + /// From bech32 (no `nostr:`). Secret keys (`nsec1…`) and anything else + /// are `None`. + fn decode(text: &str) -> Option { + let (hrp, data) = bech32_decode(text)?; + match hrp.as_str() { + "npub" => Some(Entity::Profile { + pubkey: data.try_into().ok()?, + relays: Vec::new(), + }), + "note" => Some(Entity::Event { + id: data.try_into().ok()?, + relays: Vec::new(), + author: None, + kind: None, + }), + "nprofile" => { + let tlv = Tlv::read(&data)?; + Some(Entity::Profile { + pubkey: tlv.special.try_into().ok()?, + relays: tlv.relays, + }) + } + "nevent" => { + let tlv = Tlv::read(&data)?; + Some(Entity::Event { + id: tlv.special.try_into().ok()?, + relays: tlv.relays, + author: tlv.author, + kind: tlv.kind, + }) + } + "naddr" => { + let tlv = Tlv::read(&data)?; + Some(Entity::Address { + kind: tlv.kind?, + author: tlv.author?, + identifier: String::from_utf8(tlv.special).ok()?, + relays: tlv.relays, + }) + } + _ => None, + } + } + + /// The URI it's saved under: `nostr:npub1…`, `nostr:note1…`, or + /// `nostr:naddr1…` with kind, author and name only. + fn uri(&self) -> String { + let text = match self { + Entity::Profile { pubkey, .. } => bech32_encode("npub", pubkey), + Entity::Event { id, .. } => bech32_encode("note", id), + Entity::Address { + kind, + author, + identifier, + .. + } => { + let name = identifier.as_bytes(); + // Too long a name can't be written: no URI decodes to this. + let Ok(length) = u8::try_from(name.len()) else { + return "nostr:".to_string(); + }; + let mut data = Vec::with_capacity(name.len() + 42); + data.extend_from_slice(&[0, length]); + data.extend_from_slice(name); + data.extend_from_slice(&[2, 32]); + data.extend_from_slice(author); + data.extend_from_slice(&[3, 4]); + data.extend_from_slice(&kind.to_be_bytes()); + bech32_encode("naddr", &data) + } + }; + format!("nostr:{text}") + } +} + +/// NIP-19's type-length-value fields. +#[derive(Default)] +struct Tlv { + /// Type 0: a key, an ID or a name. + special: Vec, + relays: Vec, + author: Option<[u8; 32]>, + kind: Option, +} + +impl Tlv { + fn read(data: &[u8]) -> Option { + let mut tlv = Tlv::default(); + let mut special = None; + let mut rest = data; + while !rest.is_empty() { + let [kind, length, ..] = *rest else { + return None; + }; + let value = rest.get(2..2 + usize::from(length))?; + match kind { + 0 if special.is_none() => special = Some(value.to_vec()), + 1 => { + if let Ok(relay) = std::str::from_utf8(value) { + tlv.relays.push(relay.to_string()); + } + } + 2 => tlv.author = value.try_into().ok(), + 3 => tlv.kind = Some(u32::from_be_bytes(value.try_into().ok()?)), + _ => {} + } + rest = &rest[2 + usize::from(length)..]; + } + tlv.special = special?; + Some(tlv) + } +} + +const BECH32: &[u8; 32] = b"qpzry9x8gf2tvdw0s3jn54khce6mua7l"; + +fn polymod(values: impl IntoIterator) -> u32 { + const GENERATOR: [u32; 5] = [0x3b6a57b2, 0x26508e6d, 0x1ea119fa, 0x3d4233dd, 0x2a1462b3]; + let mut check: u32 = 1; + for value in values { + let top = check >> 25; + check = ((check & 0x1ff_ffff) << 5) ^ u32::from(value); + for (i, g) in GENERATOR.iter().enumerate() { + if (top >> i) & 1 == 1 { + check ^= g; + } + } + } + check +} + +fn expand(hrp: &str) -> Vec { + hrp.bytes() + .map(|b| b >> 5) + .chain([0]) + .chain(hrp.bytes().map(|b| b & 31)) + .collect() +} + +/// Bech32 (BIP-173): the prefix and the bytes. No length limit: NIP-19's +/// longer entities go past BIP-173's 90 characters. +fn bech32_decode(text: &str) -> Option<(String, Vec)> { + if text.bytes().any(|b| b.is_ascii_uppercase()) && text.bytes().any(|b| b.is_ascii_lowercase()) + { + return None; + } + let text = text.to_ascii_lowercase(); + let split = text.rfind('1')?; + let (hrp, data) = (&text[..split], &text[split + 1..]); + if hrp.is_empty() || data.len() < 6 { + return None; + } + let values: Vec = data + .bytes() + .map(|c| BECH32.iter().position(|&b| b == c).map(|p| p as u8)) + .collect::>()?; + if polymod(expand(hrp).into_iter().chain(values.iter().copied())) != 1 { + return None; + } + let bytes = regroup(&values[..values.len() - 6], 5, 8, false)?; + Some((hrp.to_string(), bytes)) +} + +fn bech32_encode(hrp: &str, bytes: &[u8]) -> String { + let values = regroup(bytes, 8, 5, true).expect("padding allowed"); + let check = polymod( + expand(hrp) + .into_iter() + .chain(values.iter().copied()) + .chain([0; 6]), + ) ^ 1; + let mut out = String::with_capacity(hrp.len() + 1 + values.len() + 6); + out.push_str(hrp); + out.push('1'); + for v in values { + out.push(BECH32[usize::from(v)] as char); + } + for i in 0..6 { + out.push(BECH32[((check >> (5 * (5 - i))) & 31) as usize] as char); + } + out +} + +/// Bits regrouped (`from`-bit values to `to`-bit ones). +fn regroup(data: &[u8], from: u32, to: u32, pad: bool) -> Option> { + let mut acc: u32 = 0; + let mut bits: u32 = 0; + let max = (1u32 << to) - 1; + let mut out = Vec::with_capacity(data.len() * from as usize / to as usize + 1); + for &value in data { + let value = u32::from(value); + if value >> from != 0 { + return None; + } + acc = (acc << from) | value; + bits += from; + while bits >= to { + bits -= to; + out.push(((acc >> bits) & max) as u8); + } + } + if pad { + if bits > 0 { + out.push(((acc << (to - bits)) & max) as u8); + } + } else if bits >= from || ((acc << (to - bits)) & max) != 0 { + return None; + } + Some(out) +} + +fn hex(bytes: &[u8]) -> String { + bytes.iter().map(|b| format!("{b:02x}")).collect() +} + +fn unhex(text: &str) -> Option<[u8; N]> { + if text.len() != N * 2 { + return None; + } + let mut out = [0u8; N]; + for (i, byte) in out.iter_mut().enumerate() { + *byte = u8::from_str_radix(text.get(i * 2..i * 2 + 2)?, 16).ok()?; + } + Some(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Signed with BIP-340's first test key (made with its reference code). + const NOTE_JSON: &str = r##"{"id": "d44d44ca126877188068e2150f78f459fadd681e71df1247b4dc1e094af2ae26", "pubkey": "dff1d77f2a671c5f36183726db2341be58feae1da2deced843240f7b502ba659", "created_at": 1760000000, "kind": 1, "tags": [["t", "lost"], ["p", "dff1d77f2a671c5f36183726db2341be58feae1da2deced843240f7b502ba659"]], "content": "Hello from LOST\n\"quoted\" \\ back — ünïcødé ✓\ttab", "sig": "54734a4b6d5f2cc512ca8cb3b367e89eaaac444221162d6f2102e9f21cba256db6bb98402e211b20c7c4eb2c31f9328ff65e5654430b1895d5281462fd8fcf3d"}"##; + const ARTICLE_JSON: &str = r##"{"id": "64807c365425d5093ef6cae9ac6c02399993fad0cc170eea12b7203be284da73", "pubkey": "dff1d77f2a671c5f36183726db2341be58feae1da2deced843240f7b502ba659", "created_at": 1760000100, "kind": 30023, "tags": [["d", "hello"], ["title", "Hello, Nostr"], ["summary", "A test article."], ["published_at", "1760000000"]], "content": "# Hello\n\nSome **bold** text and a [link](https://example.org/).\n\n![a cat](https://example.org/cat.png)", "sig": "9195f881de527b0e8677a859d60f555a6135cad91712151dfc64f6751e0ee102512ae0c3afce397a65ca843b8903fa9ec5e9d67b894aa51995d3710d4ada6248"}"##; + const AUTHOR: &str = "dff1d77f2a671c5f36183726db2341be58feae1da2deced843240f7b502ba659"; + /// BIP-340's second test key. + const OTHER: &str = "dd308afec5777e13121fa72b9cc1b7cc0139715309b086c960e18fd969774eb8"; + /// The note's author asks to delete it; then someone else does. + const DELETION_JSON: &str = r##"{"id": "9ef55d8b3ecb18c6bafc3ca45427d419a70befeb5af57e458d180205dec89273", "pubkey": "dff1d77f2a671c5f36183726db2341be58feae1da2deced843240f7b502ba659", "created_at": 1760000200, "kind": 5, "tags": [["e", "d44d44ca126877188068e2150f78f459fadd681e71df1247b4dc1e094af2ae26"]], "content": "", "sig": "bc55e8c55454333439886e0b28ad70b47314c4b0fe02cb664442f44d24a58028aab604b605e1c902f4ca3573b8e2f8a06a9f32d2cba516f649ed54fc7515ca19"}"##; + const OTHERS_DELETION_JSON: &str = r##"{"id": "2447055bbfd15be374f6ad5080974b8122577bfc8fb6b75c489b336e5177fa65", "pubkey": "dd308afec5777e13121fa72b9cc1b7cc0139715309b086c960e18fd969774eb8", "created_at": 1760000300, "kind": 5, "tags": [["e", "d44d44ca126877188068e2150f78f459fadd681e71df1247b4dc1e094af2ae26"]], "content": "", "sig": "0fabedf8bd06af44fb04e0543b45dfecf3239956ef30e687627724f56319dd5dbb085764ec8649ee247fb64ece495e59f2b8934c8053166a703cbad4d2e58f4a"}"##; + + fn event(json: &str) -> Event { + Event::from_json(&serde_json::from_str(json).unwrap()).unwrap() + } + + fn nostr() -> Nostr { + Nostr::new(None, None, Duration::ZERO, Vec::new()).unwrap() + } + + #[test] + fn nip19() { + // NIP-19's own example. + let npub = "npub10elfcs4fr0l0r8af98jlmgdh9c8tcxjvz9qkw038js35mp4dma8qzvjptg"; + let key = "7e7e9c42a91bfef19fa929e5fda1b72e0ebc1a4c1141673e2794234d86addf4e"; + assert_eq!( + Entity::decode(npub), + Some(Entity::Profile { + pubkey: unhex(key).unwrap(), + relays: Vec::new() + }) + ); + assert_eq!(bech32_encode("npub", &unhex::<32>(key).unwrap()), npub); + assert_eq!( + Entity::decode(&npub.to_uppercase()).unwrap().uri(), + format!("nostr:{npub}") + ); + // A changed character, mixed case, a secret key: none. + assert_eq!(Entity::decode(&npub.replace("0elf", "0elg")), None); + assert_eq!( + Entity::decode("npub10ELFcs4fr0l0r8af98jlmgdh9c8tcxjvz9qkw038js35mp4dma8qzvjptg"), + None + ); + let nsec = bech32_encode("nsec", &[7; 32]); + assert_eq!(Entity::decode(&nsec), None); + // From nostr-tools' tests: addresses, TLVs in either order. + let naddr = "naddr1qq98yetxv4ex2mnrv4esygrl54h466tz4v0re4pyuavvxqptsejl0vxcmnhfl60z3rth2xkpjspsgqqqw4rsf34vl5"; + let Some(Entity::Address { + kind, + author, + identifier, + .. + }) = Entity::decode(naddr) + else { + panic!("an address"); + }; + assert_eq!(kind, 30023); + assert_eq!( + hex(&author), + "7fa56f5d6962ab1e3cd424e758c3002b8665f7b0d8dcee9fe9e288d7751ac194" + ); + assert_eq!(identifier, "references"); + let other = "naddr1qqrxyctwv9hxzq3q80cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsxpqqqp65wqfwwaehxw309aex2mrp0yhxummnw3ezuetcv9khqmr99ekhjer0d4skjm3wv4uxzmtsd3jjucm0d5q3vamnwvaz7tmwdaehgu3wvfskuctwvyhxxmmd0zfmwx"; + let entity = Entity::decode(other).unwrap(); + let Entity::Address { relays, .. } = &entity else { + panic!("an address"); + }; + assert!(relays.contains(&"wss://nostr.banana.com".to_string())); + // Saved without its relays, and that decodes the same. + let uri = entity.uri(); + assert!(uri.len() < other.len()); + let Some(Entity::Address { + kind, + identifier, + relays, + .. + }) = Entity::decode(uri.strip_prefix("nostr:").unwrap()) + else { + panic!("an address"); + }; + assert_eq!( + (kind, identifier.as_str(), relays.len()), + (30023, "banana", 0) + ); + // An event with relays: saved as its note. + let nevent = "nevent1qqst8cujky046negxgwwm5ynqwn53t8aqjr6afd8g59nfqwxpdhylpcpzamhxue69uhhyetvv9ujuetcv9khqmr99e3k7mg8arnc9"; + let entity = Entity::decode(nevent).unwrap(); + let Entity::Event { relays, .. } = &entity else { + panic!("an event"); + }; + assert_eq!(relays.len(), 1); + assert!(entity.uri().starts_with("nostr:note1")); + // Malformed TLVs don't loop or panic. + assert_eq!(Entity::decode("nevent1qq5v4rkz"), None); + } + + #[test] + fn uris_and_links() { + let npub = "npub10elfcs4fr0l0r8af98jlmgdh9c8tcxjvz9qkw038js35mp4dma8qzvjptg"; + let url = |s: &str| Url::parse(s).unwrap(); + assert!(Nostr::accepts(&url(&format!("nostr:{npub}")))); + assert!(Nostr::accepts(&url("wss://relay.damus.io/"))); + assert!(Nostr::accepts(&url( + "wss://relay.damus.io/?kinds=1&until=5" + ))); + assert!(!Nostr::accepts(&url("https://relay.damus.io/"))); + let nprofile = bech32_encode("nprofile", &{ + let mut d = vec![0, 32]; + d.extend( + unhex::<32>("7e7e9c42a91bfef19fa929e5fda1b72e0ebc1a4c1141673e2794234d86addf4e") + .unwrap(), + ); + d.extend([1, 21]); + d.extend(b"wss://relay.example/x"); + d + }); + // Not the way it's saved: rewritten, and not crawled as it is. + assert!(!Nostr::accepts(&url(&format!("nostr:{nprofile}")))); + assert_eq!( + Nostr::rewrite(&format!("nostr:{nprofile}")), + Some(format!("nostr:{npub}")) + ); + assert_eq!( + Nostr::rewrite(&format!("nostr://{npub}")), + Some(format!("nostr:{npub}")) + ); + assert_eq!(Nostr::rewrite("nostr:nsec1xyz"), None); + assert!(!Nostr::accepts(&url(&format!( + "nostr:{}", + bech32_encode("nsec", &[1; 32]) + )))); + assert_eq!( + Nostr::fallback_title(&url(&format!("nostr:{npub}"))), + "npub10elfcs4…jptg" + ); + assert_eq!( + relay_key("wss://Relay.Example:443/path?x=1"), + Some("wss://relay.example/path".into()) + ); + assert_eq!(relay_key("https://relay.example/"), None); + // References written in text. + let text = format!("hi nostr:{npub}, see NOSTR:{nprofile}. and nostr:nope"); + let links: Vec = nostr_links(&text).into_iter().map(|l| l.url).collect(); + assert_eq!(links, [format!("nostr:{npub}")]); + assert_eq!(strip_references(&text), "hi see and"); + } + + #[test] + fn signatures() { + let note = event(NOTE_JSON); + assert!(note.is_signed()); + assert!(usable(¬e)); + assert!(event(ARTICLE_JSON).is_signed()); + // Anything changed, and the ID isn't its own. + let mut changed = note.clone(); + changed.content.push('!'); + assert!(!changed.is_signed()); + let mut changed = note.clone(); + changed.created_at += 1; + assert!(!changed.is_signed()); + // Asking to expire: not kept. + let mut expiring = note.clone(); + expiring + .tags + .push(vec!["expiration".into(), "9999999999".into()]); + assert!(!usable(&expiring)); + // Not a post. + assert_eq!(Event::from_json(&json!({ "id": "x" })), None); + assert_eq!( + Event::from_json(&json!({ "id": "", "pubkey": "", "created_at": 1, "kind": 1, + "tags": [["p", 5]], "content": "", "sig": "" })), + None + ); + // Another post's signature, by the same key: the ID is the post's, + // the signature isn't for it. + let mut changed = note.clone(); + changed.sig = event(ARTICLE_JSON).sig; + assert!(!changed.is_signed()); + } + + #[test] + fn posts_as_pages() { + let nostr = nostr(); + let note = event(NOTE_JSON); + assert_eq!(note.uri().as_deref().map(|u| &u[..11]), Some("nostr:note1")); + let parse = |r: Response| match r { + Response::Document { mime, body } => { + nostr.parse(&Url::parse("nostr:x").unwrap(), &mime, &body.unwrap()) + } + other => panic!("{other:?}"), + }; + let page = parse(nostr.post(¬e, "wss://relay.example/")); + assert!(!page.skip); + assert_eq!( + page.title, + Some(format!( + "{}: Hello from LOST \"quoted\" \\ back — ünïcødé ✓ tab", + short(&npub(AUTHOR)) + )) + ); + assert_eq!(page.body, note.content); + let links: Vec<&str> = page.links.iter().map(|l| l.url.as_str()).collect(); + assert_eq!(links, [npub_uri(AUTHOR)]); + // With the author's name known, and a mention of them. + nostr + .names + .lock() + .unwrap() + .insert(AUTHOR.into(), "Satoshi".into()); + let mut mention = note.clone(); + mention.content = format!("Thanks nostr:{} for everything", npub(AUTHOR)); + let page = parse(nostr.post(&mention, "wss://relay.example/")); + assert_eq!( + page.title.as_deref(), + Some("Satoshi: Thanks @Satoshi for everything") + ); + assert_eq!(page.body, "Thanks @Satoshi for everything"); + // Too little to read: followed, not saved. + let mut short_note = note.clone(); + short_note.content = "gm 🤙 https://example.org/a-long-link".into(); + assert!(parse(nostr.post(&short_note, "wss://relay.example/")).skip); + + let article = event(ARTICLE_JSON); + assert!(article.uri().unwrap().starts_with("nostr:naddr1")); + let page = parse(nostr.post(&article, "wss://relay.example/")); + assert_eq!(page.title.as_deref(), Some("Hello, Nostr")); + assert_eq!( + page.body, + "A test article.\n\nHello\n\nSome bold text and a link.\n\na cat" + ); + let links: Vec<(&str, Option<&str>)> = page + .links + .iter() + .map(|l| (l.url.as_str(), l.label.as_deref())) + .collect(); + assert_eq!( + links, + [ + (npub_uri(AUTHOR).as_str(), Some("Satoshi")), + ("https://example.org/", Some("link")), + ] + ); + } + + #[test] + fn tags_and_text() { + let mut note = event(NOTE_JSON); + let other = "7e7e9c42a91bfef19fa929e5fda1b72e0ebc1a4c1141673e2794234d86addf4e"; + note.tags = vec![ + vec![ + "e".into(), + other.into(), + "wss://hint.example".into(), + "reply".into(), + ], + vec!["a".into(), format!("30023:{other}:my-article")], + vec!["p".into(), other.into()], + vec!["p".into(), other.into()], + vec!["t".into(), "hashtag".into()], + vec!["e".into(), "not hex".into()], + ]; + let links = tag_links(¬e); + assert_eq!(links.len(), 3); + assert!(links[0].0.starts_with("nostr:note1")); + assert_eq!(links[0].1.as_deref(), Some("wss://hint.example/")); + assert!(links[1].0.starts_with("nostr:naddr1")); + assert_eq!(links[2].0, npub_uri(other)); + + assert_eq!( + asciidoc_to_text("= Bitcoin\n\nSee [[Satoshi Nakamoto]] and [[cryptography|crypto]]."), + "Bitcoin\n\nSee Satoshi Nakamoto and crypto." + ); + assert_eq!(cut("one two three four", 9), "one two…"); + assert_eq!(cut("short", 9), "short"); + assert_eq!(cut("ünïcødéünïcødé", 5), "ünïcø…"); + let profile = Profile { + name: Some("satoshi".into()), + display_name: Some(" Satoshi\nNakamoto ".into()), + about: Some("Bitcoin's creator.".into()), + website: Some("bitcoin.org".into()), + nip05: Some("_@bitcoin.org".into()), + }; + assert_eq!(profile.name().as_deref(), Some("Satoshi Nakamoto")); + assert_eq!(profile.website().as_deref(), Some("https://bitcoin.org/")); + // What's in the text already isn't added again. + assert_eq!(profile.text(), "Bitcoin's creator.\nsatoshi\n_@bitcoin.org"); + } + + #[test] + fn listings() { + assert_eq!( + List::of("kinds=30023,30818&until=1700000000"), + Some(List { + kinds: vec![30023, 30818], + until: Some(1_700_000_000), + search: None, + }) + ); + assert_eq!( + List::of("kinds=1").map(|l| l.query()).as_deref(), + Some("kinds=1") + ); + // A search, and the same written again. + let query = "kinds=0,1,30023,30818&search=bitcoin%20mining%3F&until=5"; + let list = List::of(query).unwrap(); + assert_eq!(list.search.as_deref(), Some("bitcoin mining?")); + assert_eq!(list.query(), query); + assert_eq!(List::of("kinds=1&search="), None); + // Other kinds, other questions: not listings. + assert_eq!(List::of("kinds=4"), None); + assert_eq!(List::of("kinds=1&authors=x"), None); + assert_eq!(List::of("until=5"), None); + assert_eq!(List::of("kinds=1&until=soon"), None); + // A relay's own page: its listings, without asking it anything (but + // for what it says about itself, known here already). + let nostr = nostr(); + let _ = nostr + .relay("wss://relay.example/") + .info + .set(Info::default()); + let url = Url::parse("wss://relay.example/").unwrap(); + let Ok(Response::Document { mime, body }) = nostr.listing(&url) else { + panic!("a listing"); + }; + let parsed = nostr.parse(&url, &mime, &body.unwrap()); + assert!(parsed.skip); + let links: Vec = parsed.links.into_iter().map(|l| l.url).collect(); + assert_eq!( + links, + [ + "wss://relay.example/?kinds=30023,30818", + "wss://relay.example/?kinds=1,5" + ] + ); + assert_eq!( + refused("rate-limited: slow down"), + Response::SlowDown(Duration::from_secs(60)) + ); + assert_eq!(refused("auth-required: members only"), Response::Gone); + + // What relays say about themselves. + assert_eq!( + Info::of( + &json!({ "supported_nips": [1, 11, 50], "limitation": { "auth_required": false } }) + ), + Info { + searches: true, + closed: false + } + ); + assert_eq!( + Info::of(&json!({ "supported_nips": [1], "limitation": { "payment_required": true } })), + Info { + searches: false, + closed: true + } + ); + assert_eq!(Info::of(&json!("not a document")), Info::default()); + // A relay that searches gets a listing per phrase; one that's + // closed, none at all. + let nostr = Nostr::new(None, None, Duration::ZERO, vec!["bitcoin mining".into()]).unwrap(); + assert_eq!(nostr.seeds(), SEEDS_AND_SEARCH); + let links = |nostr: &Nostr, key: &str| match nostr.listing(&Url::parse(key).unwrap()) { + Ok(Response::Document { mime, body }) => nostr + .parse(&Url::parse(key).unwrap(), &mime, &body.unwrap()) + .links + .into_iter() + .map(|l| l.url) + .collect::>(), + other => panic!("{other:?}"), + }; + let _ = nostr.relay("wss://search.example/").info.set(Info { + searches: true, + closed: false, + }); + assert_eq!( + links(&nostr, "wss://search.example/"), + [ + "wss://search.example/?kinds=30023,30818", + "wss://search.example/?kinds=1,5", + "wss://search.example/?kinds=0,1,30023,30818&search=bitcoin%20mining", + ] + ); + let _ = nostr.relay("wss://paid.example/").info.set(Info { + searches: true, + closed: true, + }); + assert_eq!( + nostr.listing(&Url::parse("wss://paid.example/").unwrap()), + Ok(Response::Gone) + ); + } + + #[test] + fn blocked_and_deleted() { + let nostr = nostr(); + let note = event(NOTE_JSON); + assert!(nostr.wanted(¬e)); + // Blacklisted authors, written either way. + let blacklist = Blacklist::parse(&format!( + "nostr:{}\n{}\nexample.org\nnostr:{}\n", + npub(AUTHOR), + npub(OTHER), + npub(OTHER) + )) + .unwrap(); + assert_eq!(nostr.block_authors(&blacklist), 2); + assert!(!nostr.wanted(¬e)); + // Deleted: only by its author's request, whoever else asks after. + let nostr = super::tests::nostr(); + nostr.note_deletions(&[event(OTHERS_DELETION_JSON)]); + assert!(nostr.wanted(¬e)); + nostr.note_deletions(&[event(DELETION_JSON), event(OTHERS_DELETION_JSON)]); + assert!(!nostr.wanted(¬e)); + // An article, by its address. + let article = event(ARTICLE_JSON); + assert_eq!( + article.coordinate().as_deref(), + Some(format!("30023:{AUTHOR}:hello").as_str()) + ); + assert!(nostr.wanted(&article)); + nostr + .deleted + .lock() + .unwrap() + .insert(format!("{} {AUTHOR}", article.coordinate().unwrap())); + assert!(!nostr.wanted(&article)); + assert_eq!(note.coordinate(), None); + // Topics written as hashtags, unless they're there. + let mut tagged = article.clone(); + tagged.tags.extend([ + vec!["t".into(), "nostr".into()], + vec!["t".into(), "#Bitcoin".into()], + vec!["t".into(), "nostr".into()], + vec!["t".into(), "two words".into()], + ]); + assert_eq!( + with_topics("About #bitcoin".into(), &tagged), + "About #bitcoin\n\n#nostr" + ); + assert_eq!(with_topics(String::new(), &tagged), "#nostr #Bitcoin"); + } + + #[test] + fn kept_posts() { + let mut kept = Kept::default(); + let note = event(NOTE_JSON); + for i in 0..MAX_KEPT + 10 { + kept.keep(format!("nostr:{i}"), note.clone(), "wss://r/"); + } + assert_eq!(kept.events.len(), MAX_KEPT); + assert!(kept.take("nostr:0").is_none()); + assert!(kept.take(&format!("nostr:{}", MAX_KEPT + 9)).is_some()); + assert!(kept.take(&format!("nostr:{}", MAX_KEPT + 9)).is_none()); + // Kept and taken, over and over: what's left to do stays small. + let mut kept = Kept::default(); + for i in 0..MAX_KEPT * 5 { + let uri = format!("nostr:{i}"); + kept.keep(uri.clone(), note.clone(), "wss://r/"); + assert!(kept.take(&uri).is_some()); + } + assert!(kept.events.is_empty()); + assert!(kept.order.len() <= 2 * MAX_KEPT); + } + + /// A relay on this computer that answers every `REQ` with `events`, + /// then `EOSE`; what it was asked comes back at the end. + fn relay(events: Vec<&'static str>) -> (String, std::thread::JoinHandle>) { + use std::io::{Read, Write}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let address = format!("ws://{}/", listener.local_addr().unwrap()); + let server = std::thread::spawn(move || { + let (mut s, _) = listener.accept().unwrap(); + let mut head = Vec::new(); + let mut byte = [0u8; 1]; + while !head.ends_with(b"\r\n\r\n") { + s.read_exact(&mut byte).unwrap(); + head.push(byte[0]); + } + s.write_all(b"HTTP/1.1 101 Switching Protocols\r\n\r\n") + .unwrap(); + let send = |s: &mut std::net::TcpStream, text: &str| { + let mut frame = vec![0x81]; + if text.len() < 126 { + frame.push(text.len() as u8); + } else { + frame.push(126); + frame.extend_from_slice(&(text.len() as u16).to_be_bytes()); + } + frame.extend_from_slice(text.as_bytes()); + s.write_all(&frame).unwrap(); + }; + let mut asked = Vec::new(); + loop { + let mut h = [0u8; 2]; + if s.read_exact(&mut h).is_err() { + break; + } + let length = match h[1] & 0x7f { + 126 => { + let mut b = [0u8; 2]; + s.read_exact(&mut b).unwrap(); + usize::from(u16::from_be_bytes(b)) + } + n => usize::from(n), + }; + let mut mask = [0u8; 4]; + s.read_exact(&mut mask).unwrap(); + let mut payload = vec![0u8; length]; + s.read_exact(&mut payload).unwrap(); + for (i, b) in payload.iter_mut().enumerate() { + *b ^= mask[i % 4]; + } + if h[0] & 0x0f == 0x8 { + break; + } + let message: Value = serde_json::from_slice(&payload).unwrap(); + if message[0] == "REQ" { + let id = message[1].as_str().unwrap().to_string(); + send(&mut s, &json!(["NOTICE", "welcome"]).to_string()); + for event in &events { + let event: Value = serde_json::from_str(event).unwrap(); + send(&mut s, &json!(["EVENT", id, event]).to_string()); + } + send(&mut s, &json!(["EOSE", id]).to_string()); + asked.push(message); + } + } + asked + }); + (address, server) + } + + #[test] + fn deleted_while_listed() { + // The note's author asked to delete it, someone else too: it's + // left out, and so are the requests. + let (address, server) = relay(vec![DELETION_JSON, OTHERS_DELETION_JSON, NOTE_JSON]); + let nostr = nostr(); + let url = Url::parse(&format!("{address}?kinds=1,5")).unwrap(); + let Ok(Response::Document { mime, body }) = nostr.fetch(Route::Proxied, &url) else { + panic!("a listing"); + }; + let links: Vec = nostr + .parse(&url, &mime, &body.unwrap()) + .links + .into_iter() + .map(|l| l.url) + .collect(); + assert_eq!(links, [format!("{address}?kinds=1,5&until=1760000000")]); + drop(nostr); + server.join().unwrap(); + } + + #[test] + fn searching_a_relay() { + let (address, server) = relay(vec![ARTICLE_JSON]); + let nostr = Nostr::new(None, None, Duration::ZERO, vec!["hello".into()]).unwrap(); + let url = Url::parse(&format!("{address}?kinds=0,1,30023,30818&search=hello")).unwrap(); + let Ok(Response::Document { mime, body }) = nostr.fetch(Route::Proxied, &url) else { + panic!("a listing"); + }; + let links: Vec = nostr + .parse(&url, &mime, &body.unwrap()) + .links + .into_iter() + .map(|l| l.url) + .collect(); + assert_eq!( + links, + [ + event(ARTICLE_JSON).uri().unwrap(), + format!("{address}?kinds=0,1,30023,30818&search=hello&until=1760000100"), + ] + ); + drop(nostr); + let asked = server.join().unwrap(); + assert_eq!( + asked[0][2], + json!({ "kinds": [0, 1, 30023, 30818], "limit": 100, "search": "hello" }) + ); + } + + #[test] + fn listing_a_relay() { + let (address, server) = relay(vec![NOTE_JSON, ARTICLE_JSON]); + let nostr = nostr(); + let url = Url::parse(&format!("{address}?kinds=1,30023&until=1760000100")).unwrap(); + let Ok(Response::Document { mime, body }) = nostr.fetch(Route::Proxied, &url) else { + panic!("a listing"); + }; + let parsed = nostr.parse(&url, &mime, &body.unwrap()); + assert!(parsed.skip); + let note_uri = event(NOTE_JSON).uri().unwrap(); + let article_uri = event(ARTICLE_JSON).uri().unwrap(); + let links: Vec<(String, Option)> = + parsed.links.into_iter().map(|l| (l.url, l.label)).collect(); + assert_eq!( + links, + [ + (note_uri.clone(), None), + (article_uri.clone(), Some("Hello, Nostr".into())), + (format!("{address}?kinds=1,30023&until=1760000000"), None), + ] + ); + // The posts listed are there already: no relay is asked. + let Ok(Response::Document { mime, body }) = + nostr.fetch(Route::Proxied, &Url::parse(&article_uri).unwrap()) + else { + panic!("a page"); + }; + let page = nostr.parse(&url, &mime, &body.unwrap()); + assert_eq!(page.title.as_deref(), Some("Hello, Nostr")); + drop(nostr); + let asked = server.join().unwrap(); + // The listing, and the authors' names. + assert_eq!(asked.len(), 2); + assert_eq!( + asked[0][2], + json!({ "kinds": [1, 30023], "limit": 200, "until": 1760000100 }) + ); + assert_eq!(asked[1][2], json!({ "kinds": [0], "authors": [AUTHOR] })); + } +} diff --git a/src/spider/spartan.rs b/src/spider/spartan.rs new file mode 100644 index 0000000..23e699e --- /dev/null +++ b/src/spider/spartan.rs @@ -0,0 +1,195 @@ +//! Spartan (spartan://mozz.us/specification.gmi): Gemini's sibling without +//! TLS, plain TCP on port 300. A request is `HOST PATH LENGTH` and CRLF +//! (the length of data sent after it: 0 here); an answer is a one-digit +//! status, a space, a "meta" line, CRLF and, for status 2, the document, +//! mostly gemtext (ARCHITECTURE.md §15.12). + +use std::time::Duration; + +use lost::urls::Url; + +use crate::crawl::{Parsed, Protocol, Response, Route, mentions, port}; +use crate::gemini::{charset, decode, gemtext, mime_type}; +use crate::net::{self, Proxy}; + +/// Where a crawl starts: the protocol's home, and a capsule that lists +/// others. +const SEEDS: &[&str] = &["spartan://mozz.us/", "spartan://spartan.mozz.us/"]; + +pub const TIMEOUT: Duration = Duration::from_secs(15); +/// Through a proxy (Tor adds a few seconds a hop). +pub const PROXIED_TIMEOUT: Duration = Duration::from_secs(30); + +pub struct Spartan { + /// A SOCKS5 proxy (Spartan can't go through an HTTP one). + proxy: Option, + timeout: Duration, +} + +impl Spartan { + pub fn new(proxy: Option, timeout: Option) -> lost::Result { + if let Some(Proxy::Http(_)) = proxy { + return Err("Spartan can only go through a SOCKS5 proxy (socks5://HOST:PORT)".into()); + } + let timeout = timeout.unwrap_or(match proxy { + Some(_) => PROXIED_TIMEOUT, + None => TIMEOUT, + }); + Ok(Spartan { proxy, timeout }) + } +} + +impl Protocol for Spartan { + const NAME: &'static str = "spartan"; + const SCHEME: &'static str = "spartan"; + const SPACE: &'static str = "Spartan"; + const HOSTS: &'static str = "capsules"; + + fn proxy(&self) -> Option<&Proxy> { + self.proxy.as_ref() + } + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + fn fetch(&self, route: Route, url: &Url) -> Result { + let host = url.host.trim_start_matches('[').trim_end_matches(']'); + let request = format!("{host} {} 0\r\n", url.path); + let answer = net::exchange( + route, + self.proxy.as_ref(), + &url.host, + port(url), + &request, + self.timeout, + lost::MAX_BODY_BYTES, + )?; + Ok(response(&answer)) + } + + fn parse(&self, url: &Url, mime: &str, body: &[u8]) -> Parsed { + let text = decode(body, charset(mime).as_deref()); + let mut parsed = if mime_type(mime) == "text/gemini" { + gemtext(url, &text) + } else { + Parsed { + body: text.clone(), + ..Parsed::default() + } + }; + for link in mentions(&text, Self::SCHEME) { + if !parsed.links.iter().any(|l| l.url == link.url) { + parsed.links.push(link); + } + } + parsed + } +} + +/// An answer, read: `2 text/gemini` and the document, `3 /path` (moved, +/// on the same host), `4 …` (no such thing), `5 …` (the server failed). +fn response(answer: &[u8]) -> Response { + let Some(end) = answer.iter().position(|&b| b == b'\n') else { + return Response::Gone; + }; + let line = String::from_utf8_lossy(&answer[..end]); + let line = line.trim_end_matches('\r'); + let (status, meta) = match line.split_once(' ') { + Some((status, meta)) => (status, meta.trim()), + None => (line, ""), + }; + match status { + "2" => { + let mime = if meta.is_empty() { + "text/gemini; charset=utf-8".to_string() + } else { + meta.to_string() + }; + let body = mime_type(&mime) + .starts_with("text/") + .then(|| answer[end + 1..].to_vec()); + Response::Document { mime, body } + } + "3" if !meta.is_empty() => Response::Redirect(meta.to_string()), + "5" => Response::TryLater, + _ => Response::Gone, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn answers() { + assert_eq!( + response(b"2 text/gemini\r\n# Hi\n=> /about About\n"), + Response::Document { + mime: "text/gemini".into(), + body: Some(b"# Hi\n=> /about About\n".to_vec()) + } + ); + // No type: gemtext. + assert!(matches!( + response(b"2\r\nhello"), + Response::Document { mime, body: Some(_) } if mime.starts_with("text/gemini") + )); + // Not text: its type only. + assert_eq!( + response(b"2 image/png\r\n\x89PNG"), + Response::Document { + mime: "image/png".into(), + body: None + } + ); + assert_eq!(response(b"3 /new/\r\n"), Response::Redirect("/new/".into())); + assert_eq!(response(b"4 Not found\r\n"), Response::Gone); + assert_eq!(response(b"5 Oops\r\n"), Response::TryLater); + assert_eq!(response(b"garbage"), Response::Gone); + } + + #[test] + fn a_capsule() { + use std::io::{Read, Write}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = std::thread::spawn(move || { + let (mut s, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n") { + s.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + s.write_all( + b"2 text/gemini\r\n# Mozz\n=> /about.gmi About\n=> gemini://mozz.us/ Gemini\n\ + See spartan://other.example/ too.\n", + ) + .unwrap(); + String::from_utf8(request).unwrap() + }); + let spartan = Spartan::new(None, Some(Duration::from_secs(5))).unwrap(); + let url = Url::parse(&format!("spartan://127.0.0.1:{port}/a/")).unwrap(); + let address = std::net::SocketAddr::from(([127, 0, 0, 1], port)); + let Ok(Response::Document { mime, body }) = spartan.fetch(Route::Direct(address), &url) + else { + panic!("a page"); + }; + let page = spartan.parse(&url, &mime, &body.unwrap()); + assert_eq!(page.title.as_deref(), Some("Mozz")); + let links: Vec<&str> = page.links.iter().map(|l| l.url.as_str()).collect(); + assert_eq!( + links, + [ + format!("spartan://127.0.0.1:{port}/about.gmi").as_str(), + "gemini://mozz.us/", + "spartan://other.example/" + ] + ); + // The host (no port), the path, no data. + assert_eq!(server.join().unwrap(), "127.0.0.1 /a/ 0\r\n"); + assert!(Spartan::accepts(&Url::parse("spartan://mozz.us/").unwrap())); + assert_eq!(Url::parse("spartan://mozz.us:300/").unwrap().port, None); + } +} diff --git a/src/spider/swarm.rs b/src/spider/swarm.rs new file mode 100644 index 0000000..0d61fd7 --- /dev/null +++ b/src/spider/swarm.rs @@ -0,0 +1,478 @@ +//! Swarm: Ethereum's storage network, where a site is a manifest (its files +//! by path) under the hash of its content, fetched over HTTP from a gateway +//! (`https://api.gateway.ethswarm.org` by default; `--gateway` names +//! another, a Bee node on this computer say: `http://127.0.0.1:1633`). +//! +//! Pages are saved under `bzz://HASH/path` (64 hex digits), or +//! `bzz://NAME.eth/path` for ENS names, which the gateway looks up. Links to +//! gateways (`https://NAME.bzz.link/`, `https://CID.bzz.link/`, +//! `…/bzz/HASH/…`) and CIDs (`bah5…`) are written that way too. Swarm has no +//! index to list, so the crawl follows links only (ARCHITECTURE.md §15.11). + +use std::net::{IpAddr, SocketAddr, ToSocketAddrs}; +use std::time::Duration; + +use lost::urls::Url; + +use crate::crawl::{Blacklist, Parsed, Protocol, Response, Route, mentions}; +use crate::http::{self, Answer, Http}; +use crate::ipfs::{base32_decode, varint}; +use crate::net::Proxy; + +pub const TIMEOUT: Duration = Duration::from_secs(60); +pub const GATEWAY: &str = "https://api.gateway.ethswarm.org"; + +/// Where a crawl starts: Swarm's own site, the Fair Data Society's, and a +/// site its gateway guide links to by CID. +const SEEDS: &[&str] = &[ + "bzz://swarm.eth/", + "bzz://fairdataprotocol.eth/", + "bzz://fdfd170f73953bc262d936d3a5329b787980335dc0547032bb2a6239ebe95a76/", +]; + +/// Gateways, whose links are links to Swarm: `/bzz/HASH/…` on them, or a +/// name or CID as a `bzz.link` subdomain. +const GATEWAYS: &[&str] = &["bzz.link", "gateway.ethswarm.org"]; + +/// Multicodecs of Swarm's CIDs: a manifest, a feed. +const MANIFEST: u64 = 0xfa; +const FEED: u64 = 0xfb; +/// Keccak-256, as a multihash. +const KECCAK_256: u64 = 0x1b; + +pub struct Swarm { + http: Http, + /// The gateway's scheme, host and port. + gateway: Url, + /// Where it is, for telling servers apart (none through a proxy). + address: Option, + /// It's on this computer: requests to it needn't wait for each other. + local: bool, +} + +impl Swarm { + pub fn new( + gateway: Option, + proxy: Option, + timeout: Option, + ) -> lost::Result { + let text = gateway.unwrap_or_else(|| GATEWAY.to_string()); + let gateway = Url::parse(text.trim().trim_end_matches('/')) + .filter(|u| { + (u.scheme == "http" || u.scheme == "https") && u.path == "/" && u.query.is_none() + }) + .ok_or_else(|| { + format!("--gateway takes http://HOST:PORT or https://HOST (not \"{text}\")") + })?; + let ip = gateway + .host + .trim_start_matches('[') + .trim_end_matches(']') + .parse::() + .ok(); + let local = proxy.is_none() + && (gateway.host == "localhost" || ip.is_some_and(|ip| ip.is_loopback())); + let address = match proxy { + Some(_) => None, + None => Some(lookup(&gateway)?), + }; + Ok(Swarm { + http: Http::new(proxy, timeout.unwrap_or(TIMEOUT))?, + gateway, + address, + local, + }) + } + + fn route(&self) -> Route { + match self.address { + Some(address) => Route::Direct(address), + None => Route::Proxied, + } + } + + /// The gateway's answer for `path`, its own redirects followed (a + /// directory's `/` added, say). + fn get(&self, path: &str) -> Result { + let mut url = self.gateway.clone(); + url.path = path.to_string(); + let mut answer = self.http.get(self.route(), &url)?; + for _ in 0..3 { + let location = match answer.status { + 301 | 302 | 303 | 307 | 308 => answer.header("location").map(str::to_string), + _ => None, + }; + let Some(location) = location else { + break; + }; + // Only the gateway's own paths: anywhere else is a link. + let path = match Url::parse(&location) { + Some(target) if target.host == self.gateway.host => target.path, + Some(_) => break, + None if location.starts_with('/') => { + location.split('?').next().unwrap_or("/").to_string() + } + None => break, + }; + // The gateway refusing the site: said as it is, not followed. + if path.starts_with("/forbidden") { + break; + } + url.path = path; + answer = self.http.get(self.route(), &url)?; + } + Ok(answer) + } +} + +/// The address of a host, looked up here. +fn lookup(url: &Url) -> Result { + let port = url + .port + .unwrap_or(if url.scheme == "https" { 443 } else { 80 }); + let host = url.host.trim_start_matches('[').trim_end_matches(']'); + (host, port) + .to_socket_addrs() + .ok() + .and_then(|mut a| a.next()) + .ok_or_else(|| format!("can't find {}", url.host)) +} + +impl Protocol for Swarm { + const NAME: &'static str = "swarm"; + /// `--link swarm.eth` is a name: `bzz://swarm.eth/`. + const SCHEME: &'static str = "bzz"; + const SPACE: &'static str = "Swarm"; + const HOSTS: &'static str = "sites"; + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + fn proxy(&self) -> Option<&Proxy> { + self.http.proxy() + } + + /// Through a gateway on this computer, every site is a server of its + /// own; through any other, they all are that one server, asked one + /// thing at a time (`--delay`). + fn proxied(&self) -> bool { + self.local + } + + fn resolve(&self, _url: &Url) -> Option { + Some( + self.address + .unwrap_or_else(|| SocketAddr::from(([0, 0, 0, 0], 0))), + ) + } + + fn accepts(url: &Url) -> bool { + url.scheme == "bzz" && (is_hash(&url.host) || is_ens(&url.host)) + } + + fn rewrite(link: &str) -> Option { + canonical(link) + } + + /// `GET /bzz/HASH/path`. A gateway that won't serve a site (public ones + /// only serve the sites they allow by hash) sends the crawler to its + /// `/forbidden` page: the site is as good as gone. + fn fetch(&self, _route: Route, url: &Url) -> Result { + let answer = self.get(&format!("/bzz/{}{}", url.host, url.path))?; + if answer.status >= 500 { + return Err(format!("unreachable ({})", answer.status)); + } + Ok(match http::response(answer) { + Response::Redirect(location) => { + let forbidden = Url::parse(&location) + .map_or(location.clone(), |u| u.path) + .starts_with("/forbidden"); + if forbidden { + Response::Gone + } else { + Response::Redirect(canonical(&location).unwrap_or(location)) + } + } + other => other, + }) + } + + fn parse(&self, url: &Url, mime: &str, body: &[u8]) -> Parsed { + let mut parsed = http::parse(url, mime, body); + let written = mentions(&parsed.body, "bzz"); + let mut links = Vec::new(); + for mut link in std::mem::take(&mut parsed.links).into_iter().chain(written) { + if let Some(url) = canonical(&link.url) { + link.url = url; + } + if !links.iter().any(|l: &crate::crawl::Link| l.url == link.url) { + links.push(link); + } + } + parsed.links = links; + parsed + } + + /// The gateway must answer; a public one serves only some sites. + fn prepare(&self, _blacklist: &mut Blacklist) -> lost::Result<()> { + self.get("/bzz/swarm.eth/").map_err(|e| { + format!( + "can't reach the Swarm gateway at {} ({e})", + self.gateway.host + ) + })?; + if !self.local { + eprintln!( + "Through {}: public gateways serve only the sites they allow; a Bee node \ + of your own reaches all of Swarm (--gateway http://127.0.0.1:1633).", + self.gateway.host + ); + } + Ok(()) + } +} + +/// The one way a Swarm address is saved: `bzz://HASH/path` (lowercase) or +/// `bzz://NAME.eth/path`, from a `bzz://` link (with a hash, a CID or a +/// name) or a gateway's (`https://NAME.bzz.link/…`, `https://CID.bzz.link/…`, +/// `https://GATEWAY/bzz/HASH/…`). Queries and fragments go. +pub fn canonical(text: &str) -> Option { + let text = text.trim(); + let (scheme, rest) = text.split_once("://")?; + let rest = rest.split(['#', '?']).next().unwrap_or(""); + let address = |host: &str, path: &str| -> Option { + let path = if path.is_empty() { "/" } else { path }; + let host = host.to_ascii_lowercase(); + let host = if is_hash(&host) || is_ens(&host) { + host + } else { + cid_hash(&host)? + }; + Some(format!("bzz://{host}{path}")) + }; + match scheme.to_ascii_lowercase().as_str() { + "bzz" => { + let end = rest.find('/').unwrap_or(rest.len()); + let (host, path) = rest.split_at(end); + // `/bzz/HASH/…` from a page made for gateways, read as if it + // were on one. + if let Some(inner) = path.strip_prefix("/bzz/") + && let Some(link) = canonical(&format!("bzz://{inner}")) + { + return Some(link); + } + address(host, path) + } + "http" | "https" => { + let url = Url::parse(text)?; + let gateway = GATEWAYS.iter().find(|g| { + url.host == **g + || url + .host + .strip_suffix(**g) + .is_some_and(|sub| sub.ends_with('.')) + })?; + // `/bzz/HASH/…`, on the gateway or a subdomain of it. + if let Some(rest) = url.path.strip_prefix("/bzz/") { + let end = rest.find('/').unwrap_or(rest.len()); + let (host, path) = rest.split_at(end); + return address(host, path); + } + // `NAME.bzz.link` (`NAME.eth`), `CID.bzz.link`. + let label = url.host.strip_suffix(*gateway)?.strip_suffix('.')?; + if label.contains('.') || label == "api" || label == "www" { + return None; + } + match cid_hash(label) { + Some(hash) => address(&hash, &url.path), + None => address(&format!("{label}.eth"), &url.path), + } + } + _ => None, + } +} + +/// A reference: 32 bytes, as 64 hex digits. +fn is_hash(text: &str) -> bool { + text.len() == 64 && text.bytes().all(|b| b.is_ascii_hexdigit()) +} + +/// An ENS name: `.eth`, with labels of letters, digits and `-`. +fn is_ens(text: &str) -> bool { + let Some(name) = text.strip_suffix(".eth") else { + return false; + }; + !name.is_empty() + && text.len() <= 253 + && name.split('.').all(|label| { + (1..=63).contains(&label.len()) + && label + .bytes() + .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-') + && !label.starts_with('-') + && !label.ends_with('-') + }) +} + +/// A Swarm CID's hash, hex: base32 (`b…`), version 1, a manifest's or a +/// feed's codec, a Keccak-256 multihash. +fn cid_hash(text: &str) -> Option { + let bytes = base32_decode(text.strip_prefix('b')?)?; + let (1, rest) = varint(&bytes)? else { + return None; + }; + let (codec, rest) = varint(rest)?; + let (hash, rest) = varint(rest)?; + let (length, digest) = varint(rest)?; + if !matches!(codec, MANIFEST | FEED) || hash != KECCAK_256 || length != 32 || digest.len() != 32 + { + return None; + } + Some(digest.iter().map(|b| format!("{b:02x}")).collect()) +} + +#[cfg(test)] +mod tests { + use super::*; + + const HASH: &str = "6c45eae389b3bffce21443316d0bd47c4101545092b7c72c313a33ee7d003475"; + + #[test] + fn one_way_to_write_each_address() { + let c = |s: &str| canonical(s); + let site = format!("bzz://{HASH}/"); + assert_eq!(c(&format!("bzz://{HASH}")), Some(site.clone())); + assert_eq!( + c(&format!("bzz://{}/a.html?x=1#y", HASH.to_uppercase())), + Some(format!("bzz://{HASH}/a.html")) + ); + assert_eq!( + c("bzz://Swarm.ETH/build"), + Some("bzz://swarm.eth/build".into()) + ); + // Gateways' links. + assert_eq!( + c(&format!( + "https://api.gateway.ethswarm.org/bzz/{HASH}/docs/" + )), + Some(format!("bzz://{HASH}/docs/")) + ); + assert_eq!( + c(&format!("https://gateway.ethswarm.org/bzz/{HASH}")), + Some(site.clone()) + ); + assert_eq!( + c("https://swarm.bzz.link/build"), + Some("bzz://swarm.eth/build".into()) + ); + // CIDs: a feed's and a manifest's, as their hashes. + assert_eq!( + c("https://bah5qcgza7x6rod3tsu54eywzg3j2kmu3pb4yam25ybkhamv3fjrdt27jlj3a.bzz.link/"), + Some("bzz://fdfd170f73953bc262d936d3a5329b787980335dc0547032bb2a6239ebe95a76/".into()) + ); + assert_eq!( + c("bzz://bah5acgzafqsrk5aadnqrurohacfyn6hayqnunelko275sxj2zaf4quhogvia/x"), + Some("bzz://2c251574001b611a45c7008b86f8e0c41b46916a76bfd95d3ac80bc850ee3550/x".into()) + ); + // Not Swarm's. + assert_eq!(c("https://example.org/bzz/x"), None); + assert_eq!(c("https://bzz.link/"), None); + assert_eq!(c("https://a.b.bzz.link/"), None); + assert_eq!(c("bzz://not-a-hash/"), None); + assert_eq!(c("bzz://-bad.eth/"), None); + // An IPFS CID isn't a Swarm one. + assert_eq!( + c("bzz://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi/"), + None + ); + let url = |s: &str| Url::parse(s).unwrap(); + assert!(Swarm::accepts(&url(&site))); + assert!(Swarm::accepts(&url("bzz://swarm.eth/"))); + assert!(Swarm::accepts(&url("bzz://docs.swarm.eth/a"))); + assert!(!Swarm::accepts(&url("bzz://example.org/"))); + assert!(!Swarm::accepts(&url(&format!("https://{HASH}/")))); + assert_eq!( + Swarm::rewrite("https://swarm.bzz.link/"), + Some("bzz://swarm.eth/".into()) + ); + } + + /// A gateway on this computer giving `answers` in turn; what it was + /// asked comes back at the end. + fn serve(answers: Vec) -> (String, std::thread::JoinHandle>) { + use std::io::{Read, Write}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let address = format!("http://{}", listener.local_addr().unwrap()); + let server = std::thread::spawn(move || { + let mut asked = Vec::new(); + for answer in answers { + let (mut s, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + s.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + let request = String::from_utf8(request).unwrap(); + asked.push(request.lines().next().unwrap().to_string()); + s.write_all(answer.as_bytes()).unwrap(); + } + asked + }); + (address, server) + } + + #[test] + fn through_the_gateway() { + let page = format!( + "SwarmFDP \ + About" + ); + let (gateway, server) = serve(vec![ + // A directory without its `/`: the gateway adds it. + format!("HTTP/1.1 308 Permanent Redirect\r\nLocation: /bzz/{HASH}/docs/\r\n\r\n"), + format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: {}\r\n\r\n{page}", + page.len() + ), + // A site the gateway won't serve. + "HTTP/1.1 302 Found\r\nLocation: /forbidden?hash=abc\r\n\r\n".into(), + "HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n".into(), + "HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n".into(), + ]); + let swarm = Swarm::new(Some(gateway), None, Some(Duration::from_secs(5))).unwrap(); + assert!(swarm.proxied()); + let url = Url::parse(&format!("bzz://{HASH}/docs")).unwrap(); + let Ok(Response::Document { mime, body }) = swarm.fetch(Route::Proxied, &url) else { + panic!("a page"); + }; + let parsed = swarm.parse(&url, &mime, &body.unwrap()); + assert_eq!(parsed.title.as_deref(), Some("Swarm")); + let links: Vec<&str> = parsed.links.iter().map(|l| l.url.as_str()).collect(); + assert_eq!( + links, + [ + "bzz://fairdataprotocol.eth/".to_string(), + format!("bzz://{HASH}/about") + ] + ); + assert_eq!(swarm.fetch(Route::Proxied, &url), Ok(Response::Gone)); + assert_eq!(swarm.fetch(Route::Proxied, &url), Ok(Response::Gone)); + assert_eq!( + swarm.fetch(Route::Proxied, &url), + Err("unreachable (502)".to_string()) + ); + let asked = server.join().unwrap(); + assert_eq!( + asked, + [ + format!("GET /bzz/{HASH}/docs HTTP/1.1"), + format!("GET /bzz/{HASH}/docs/ HTTP/1.1"), + format!("GET /bzz/{HASH}/docs HTTP/1.1"), + format!("GET /bzz/{HASH}/docs HTTP/1.1"), + format!("GET /bzz/{HASH}/docs HTTP/1.1"), + ] + ); + } +} diff --git a/src/spider/ton.rs b/src/spider/ton.rs new file mode 100644 index 0000000..33f7931 --- /dev/null +++ b/src/spider/ton.rs @@ -0,0 +1,188 @@ +//! TON Sites: the web over TON's network, through a TON proxy's HTTP proxy +//! (`rldp-http-proxy`, or Tonutils Proxy; `127.0.0.1:8080` by default). +//! The proxy looks up `.ton` names in TON DNS, reaches `.adnl` addresses +//! and `.bag` storage directly, and sends anything else to the ordinary +//! web, so only those three are crawled (ARCHITECTURE.md §15.10). + +use std::time::{Duration, Instant}; + +use lost::urls::Url; + +use crate::crawl::{Parsed, Protocol, Response, Route}; +use crate::http::{self, Http}; +use crate::net::Proxy; + +/// Where a crawl starts: the TON Foundation's site, and a few others that +/// link around. +const SEEDS: &[&str] = &[ + "http://foundation.ton/", + "http://storage.ton/", + "http://howtorunsite.ton/", + "http://tondnsdomain.ton/", +]; + +/// How long to wait for a proxy that has just started (it reads TON's +/// state before it can look up names). +const WARM_UP: Duration = Duration::from_secs(5 * 60); + +pub const TIMEOUT: Duration = Duration::from_secs(60); +pub const PROXY: &str = "http://127.0.0.1:8080"; + +/// What the proxy takes as TON's: anything else would go to the web. +const DOMAINS: &[&str] = &[".ton", ".adnl", ".bag"]; + +pub struct Ton { + http: Http, +} + +impl Ton { + pub fn new(proxy: Option, timeout: Option) -> lost::Result { + let proxy = match proxy { + Some(proxy) => proxy, + None => Proxy::parse(PROXY)?, + }; + Ok(Ton { + http: Http::new(Some(proxy), timeout.unwrap_or(TIMEOUT))?, + }) + } +} + +impl Protocol for Ton { + const NAME: &'static str = "ton"; + const SCHEME: &'static str = "http"; + const SPACE: &'static str = "TON Sites"; + const HOSTS: &'static str = "sites"; + + fn seeds(&self) -> &'static [&'static str] { + SEEDS + } + + fn proxy(&self) -> Option<&Proxy> { + self.http.proxy() + } + + /// The proxy answers for sites it can't reach (5xx): those count as the + /// site being unreachable, not as its pages. + fn fetch(&self, route: Route, url: &Url) -> Result { + let answer = self.http.get(route, url)?; + if answer.status >= 500 { + return Err(format!("unreachable ({})", answer.status)); + } + Ok(http::response(answer)) + } + + fn parse(&self, url: &Url, mime: &str, body: &[u8]) -> Parsed { + http::parse(url, mime, body) + } + + /// Waits for the proxy to reach a site (it can take a while after it + /// starts). + fn prepare(&self, _blacklist: &mut crate::crawl::Blacklist) -> lost::Result<()> { + let url = Url::parse(SEEDS[0]).expect("a valid URL"); + let started = Instant::now(); + let mut said = false; + loop { + match self.fetch(Route::Proxied, &url) { + Ok(_) => { + if said { + eprintln!("The TON proxy is ready."); + } + return Ok(()); + } + Err(_) if started.elapsed() < WARM_UP => { + if !said { + eprintln!( + "Waiting for the TON proxy to be able to reach sites \ + (a minute or two after it starts)…" + ); + said = true; + } + std::thread::sleep(Duration::from_secs(10)); + } + Err(e) => { + return Err(format!( + "the TON proxy couldn't reach {} in {} minutes ({e}); is it connected?", + url.host, + WARM_UP.as_secs() / 60 + ) + .into()); + } + } + } + } + + fn accepts(url: &Url) -> bool { + url.scheme == "http" && DOMAINS.iter().any(|d| url.host.ends_with(d)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn addresses() { + let accepts = |s: &str| Ton::accepts(&Url::parse(s).unwrap()); + assert!(accepts("http://foundation.ton/")); + assert!(accepts("http://blog.foundation.ton/a")); + assert!(accepts( + "http://vcqmha5j3ceve35ammfrhqty46rkhi455otydstv66pk2tmf7rl25f3.adnl/" + )); + assert!(accepts( + "http://45061c1d4ec44a937d0318589e13c73d151d1cef5d3c0e53afbcf56a6c2fe2bd.bag/" + )); + // The proxy would send these to the ordinary web. + assert!(!accepts("http://ton.org/")); + assert!(!accepts("http://foundation.ton.website/")); + assert!(!accepts("http://example.tonx/")); + assert!(!accepts("https://foundation.ton/")); + assert!(!accepts("gemini://foundation.ton/")); + // A bare name is a .ton site. + assert_eq!(Ton::SCHEME, "http"); + } + + #[test] + fn through_the_proxy() { + use std::io::{Read, Write}; + // A proxy that has the site, then one that can't reach it. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let proxy = Proxy::Http(listener.local_addr().unwrap().to_string()); + let server = std::thread::spawn(move || { + let mut asked = Vec::new(); + let page = "TONStorage"; + for answer in [ + format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: {}\r\n\r\n{page}", + page.len() + ), + "HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n".to_string(), + ] { + let (mut s, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + s.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + asked.push(String::from_utf8(request).unwrap()); + s.write_all(answer.as_bytes()).unwrap(); + } + asked + }); + let ton = Ton::new(Some(proxy), Some(Duration::from_secs(5))).unwrap(); + let url = Url::parse("http://foundation.ton/").unwrap(); + let Ok(Response::Document { mime, body }) = ton.fetch(Route::Proxied, &url) else { + panic!("a page"); + }; + let parsed = ton.parse(&url, &mime, &body.unwrap()); + assert_eq!(parsed.title.as_deref(), Some("TON")); + assert_eq!(parsed.links[0].url, "http://storage.ton/"); + assert_eq!( + ton.fetch(Route::Proxied, &url), + Err("unreachable (502)".to_string()) + ); + let asked = server.join().unwrap(); + // The whole URL, to the proxy. + assert!(asked[0].starts_with("GET http://foundation.ton/ HTTP/1.1\r\n")); + } +} diff --git a/src/spider/websocket.rs b/src/spider/websocket.rs new file mode 100644 index 0000000..7b24855 --- /dev/null +++ b/src/spider/websocket.rs @@ -0,0 +1,447 @@ +//! WebSocket (RFC 6455), as much as a Nostr client needs: a connection to a +//! relay (`ws://` or `wss://`, straight or through a SOCKS5 proxy) and text +//! messages both ways. No extensions (so no compression) and no +//! subprotocols; pings are answered. + +use std::io::{ErrorKind, Read, Write}; +use std::net::ToSocketAddrs; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use lost::urls::Url; +use rustls::pki_types::ServerName; +use rustls::{ClientConfig, ClientConnection, StreamOwned}; + +use crate::net::{self, Proxy, Route}; + +/// A message longer than this ends the connection: no Nostr event is that +/// big (relays refuse them far sooner). +const MAX_MESSAGE: usize = 16 * 1024 * 1024; +const MAX_HEADERS: usize = 16 * 1024; + +trait Stream: Read + Write + Send {} +impl Stream for T {} + +pub struct WebSocket { + stream: Box, + /// Read, not yet taken. + buffer: Vec, +} + +impl WebSocket { + /// Connects to `url` and asks for a WebSocket. `tls` is for `wss://`. + pub fn connect( + url: &Url, + proxy: Option<&Proxy>, + timeout: Duration, + tls: Option<&Arc>, + ) -> Result { + let secure = match url.scheme.as_str() { + "wss" => true, + "ws" => false, + other => return Err(format!("{other}:// isn't a WebSocket")), + }; + let host = url.host.trim_start_matches('[').trim_end_matches(']'); + let port = url.port.unwrap_or(if secure { 443 } else { 80 }); + let route = match proxy { + Some(Proxy::Socks5(_)) => Route::Proxied, + Some(Proxy::Http(_)) => { + return Err("a relay can't be reached through an HTTP proxy".into()); + } + None => Route::Direct( + (host, port) + .to_socket_addrs() + .ok() + .and_then(|mut a| a.next()) + .ok_or_else(|| format!("can't find {host}"))?, + ), + }; + let tcp = net::connect(route, proxy, host, port, timeout)?; + let stream: Box = if secure { + let tls = tls.ok_or("no TLS for wss://")?; + let name = ServerName::try_from(host.to_string()).map_err(|e| e.to_string())?; + let connection = ClientConnection::new(tls.clone(), name).map_err(|e| e.to_string())?; + Box::new(StreamOwned::new(connection, tcp)) + } else { + Box::new(tcp) + }; + let mut socket = WebSocket { + stream, + buffer: Vec::new(), + }; + socket.handshake(url, Instant::now() + timeout * 2)?; + Ok(socket) + } + + /// A WebSocket over `stream`, already open (for tests). + #[cfg(test)] + fn over(stream: impl Read + Write + Send + 'static) -> WebSocket { + WebSocket { + stream: Box::new(stream), + buffer: Vec::new(), + } + } + + fn handshake(&mut self, url: &Url, deadline: Instant) -> Result<(), String> { + let mut key = [0u8; 16]; + getrandom::fill(&mut key).map_err(|e| e.to_string())?; + let host = match url.port { + Some(port) => format!("{}:{port}", url.host), + None => url.host.clone(), + }; + let request = format!( + "GET {} HTTP/1.1\r\nHost: {host}\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\ + Sec-WebSocket-Key: {}\r\nSec-WebSocket-Version: 13\r\n\ + User-Agent: lost-spider/{}\r\n\r\n", + url.path, + base64(&key), + env!("CARGO_PKG_VERSION"), + ); + self.stream + .write_all(request.as_bytes()) + .and_then(|()| self.stream.flush()) + .map_err(|e| e.to_string())?; + let end = loop { + if let Some(i) = self.buffer.windows(4).position(|w| w == b"\r\n\r\n") { + break i + 4; + } + if self.buffer.len() > MAX_HEADERS { + return Err("an answer that isn't a WebSocket's".into()); + } + self.read_more(deadline)?; + }; + let head = String::from_utf8_lossy(&self.buffer[..end]).into_owned(); + self.buffer.drain(..end); + let status = head.split_whitespace().nth(1).unwrap_or(""); + if status != "101" { + return Err(match status { + "" => "an answer that isn't HTTP".to_string(), + status => format!("not a relay (HTTP {status})"), + }); + } + Ok(()) + } + + /// Sends a text message. + pub fn send(&mut self, text: &str) -> Result<(), String> { + self.send_frame(0x1, text.as_bytes()) + } + + fn send_frame(&mut self, opcode: u8, payload: &[u8]) -> Result<(), String> { + let mut frame = Vec::with_capacity(payload.len() + 14); + frame.push(0x80 | opcode); + // Every frame a client sends is masked. + match payload.len() { + n if n < 126 => frame.push(0x80 | n as u8), + n if n <= 0xffff => { + frame.push(0x80 | 126); + frame.extend_from_slice(&(n as u16).to_be_bytes()); + } + n => { + frame.push(0x80 | 127); + frame.extend_from_slice(&(n as u64).to_be_bytes()); + } + } + let mut mask = [0u8; 4]; + getrandom::fill(&mut mask).map_err(|e| e.to_string())?; + frame.extend_from_slice(&mask); + frame.extend(payload.iter().enumerate().map(|(i, b)| b ^ mask[i % 4])); + self.stream + .write_all(&frame) + .and_then(|()| self.stream.flush()) + .map_err(|e| e.to_string()) + } + + /// The next text message, by `deadline`. Pings are answered on the way; + /// the relay closing the connection is an error. + pub fn receive(&mut self, deadline: Instant) -> Result { + let mut message: Vec = Vec::new(); + let mut in_message = false; + loop { + let (fin, opcode, payload) = self.frame(deadline)?; + match opcode { + 0x0 if in_message => message.extend_from_slice(&payload), + 0x1 | 0x2 if !in_message => { + message = payload; + in_message = true; + } + 0x8 => return Err("closed by the relay".into()), + 0x9 => { + self.send_frame(0xA, &payload)?; + continue; + } + 0xA => continue, + _ => return Err("a WebSocket frame that makes no sense".into()), + } + if message.len() > MAX_MESSAGE { + return Err("a message too long".into()); + } + if fin { + return String::from_utf8(message).map_err(|_| "a message that isn't text".into()); + } + } + } + + /// One frame: whether it's the message's last, its opcode, its payload. + fn frame(&mut self, deadline: Instant) -> Result<(bool, u8, Vec), String> { + self.fill(2, deadline)?; + let (first, second) = (self.buffer[0], self.buffer[1]); + let masked = second & 0x80 != 0; + let (length, mut at) = match second & 0x7f { + 126 => { + self.fill(4, deadline)?; + ( + u64::from(u16::from_be_bytes([self.buffer[2], self.buffer[3]])), + 4, + ) + } + 127 => { + self.fill(10, deadline)?; + let mut bytes = [0u8; 8]; + bytes.copy_from_slice(&self.buffer[2..10]); + (u64::from_be_bytes(bytes), 10) + } + n => (u64::from(n), 2), + }; + if length > MAX_MESSAGE as u64 { + return Err("a message too long".into()); + } + let length = length as usize; + let mut mask = None; + if masked { + self.fill(at + 4, deadline)?; + mask = Some([ + self.buffer[at], + self.buffer[at + 1], + self.buffer[at + 2], + self.buffer[at + 3], + ]); + at += 4; + } + self.fill(at + length, deadline)?; + let mut payload: Vec = self.buffer.drain(..at + length).skip(at).collect(); + if let Some(mask) = mask { + for (i, b) in payload.iter_mut().enumerate() { + *b ^= mask[i % 4]; + } + } + Ok((first & 0x80 != 0, first & 0x0f, payload)) + } + + /// Reads until at least `n` bytes are waiting. + fn fill(&mut self, n: usize, deadline: Instant) -> Result<(), String> { + while self.buffer.len() < n { + self.read_more(deadline)?; + } + Ok(()) + } + + fn read_more(&mut self, deadline: Instant) -> Result<(), String> { + if Instant::now() >= deadline { + return Err("timed out".into()); + } + let mut chunk = [0u8; 16 * 1024]; + loop { + match self.stream.read(&mut chunk) { + Ok(0) => return Err("the relay hung up".into()), + Ok(n) => { + self.buffer.extend_from_slice(&chunk[..n]); + return Ok(()); + } + Err(e) if e.kind() == ErrorKind::Interrupted => continue, + Err(e) if matches!(e.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut) => { + return Err("timed out".into()); + } + Err(e) => return Err(e.to_string()), + } + } + } +} + +/// A connection let go says goodbye (whatever happens then doesn't matter). +impl Drop for WebSocket { + fn drop(&mut self) { + let _ = self.send_frame(0x8, &1000u16.to_be_bytes()); + } +} + +/// Base64, standard alphabet, padded (for Sec-WebSocket-Key). +fn base64(bytes: &[u8]) -> String { + const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let mut out = String::with_capacity(bytes.len().div_ceil(3) * 4); + for chunk in bytes.chunks(3) { + let n = (u32::from(chunk[0]) << 16) + | (u32::from(*chunk.get(1).unwrap_or(&0)) << 8) + | u32::from(*chunk.get(2).unwrap_or(&0)); + for i in 0..4 { + if i <= chunk.len() { + out.push(ALPHABET[(n >> (18 - 6 * i) & 63) as usize] as char); + } else { + out.push('='); + } + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use std::net::{TcpListener, TcpStream}; + + #[test] + fn base64_of_keys() { + assert_eq!(base64(b""), ""); + assert_eq!(base64(b"f"), "Zg=="); + assert_eq!(base64(b"fo"), "Zm8="); + assert_eq!(base64(b"foo"), "Zm9v"); + assert_eq!(base64(b"foobar"), "Zm9vYmFy"); + // RFC 6455's example key. + assert_eq!(base64(b"the sample nonce"), "dGhlIHNhbXBsZSBub25jZQ=="); + } + + /// A server's frame: unmasked, or masked (which servers mustn't do, + /// but it's read anyway). + fn frame(fin: bool, opcode: u8, payload: &[u8], mask: Option<[u8; 4]>) -> Vec { + let mut out = vec![if fin { 0x80 } else { 0 } | opcode]; + let bit = if mask.is_some() { 0x80 } else { 0 }; + match payload.len() { + n if n < 126 => out.push(bit | n as u8), + n if n <= 0xffff => { + out.push(bit | 126); + out.extend_from_slice(&(n as u16).to_be_bytes()); + } + n => { + out.push(bit | 127); + out.extend_from_slice(&(n as u64).to_be_bytes()); + } + } + match mask { + Some(mask) => { + out.extend_from_slice(&mask); + out.extend(payload.iter().enumerate().map(|(i, b)| b ^ mask[i % 4])); + } + None => out.extend_from_slice(payload), + } + out + } + + /// Reads one masked client frame: its opcode and payload. + fn client_frame(stream: &mut TcpStream) -> (u8, Vec) { + let mut head = [0u8; 2]; + stream.read_exact(&mut head).unwrap(); + assert!(head[1] & 0x80 != 0, "client frames are masked"); + let length = match head[1] & 0x7f { + 126 => { + let mut b = [0u8; 2]; + stream.read_exact(&mut b).unwrap(); + usize::from(u16::from_be_bytes(b)) + } + 127 => { + let mut b = [0u8; 8]; + stream.read_exact(&mut b).unwrap(); + u64::from_be_bytes(b) as usize + } + n => usize::from(n), + }; + let mut mask = [0u8; 4]; + stream.read_exact(&mut mask).unwrap(); + let mut payload = vec![0u8; length]; + stream.read_exact(&mut payload).unwrap(); + for (i, b) in payload.iter_mut().enumerate() { + *b ^= mask[i % 4]; + } + (head[0] & 0x0f, payload) + } + + #[test] + fn talking_to_a_relay() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let long = "x".repeat(70_000); + let long_copy = long.clone(); + let server = std::thread::spawn(move || { + let (mut s, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + s.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + let request = String::from_utf8(request).unwrap(); + // The answer comes with the first frame right behind it. + let mut answer = b"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n\ + Connection: Upgrade\r\n\r\n" + .to_vec(); + answer.extend(frame(true, 0x1, b"[\"NOTICE\",\"hi\"]", None)); + s.write_all(&answer).unwrap(); + let (opcode, text) = client_frame(&mut s); + assert_eq!( + (opcode, text.as_slice()), + (0x1, b"[\"REQ\",\"a\",{}]".as_slice()) + ); + // A ping in the middle of a message in two parts, then a long + // one, masked. + s.write_all(&frame(false, 0x1, b"[\"EOSE\",", None)) + .unwrap(); + s.write_all(&frame(true, 0x9, b"ping!", None)).unwrap(); + s.write_all(&frame(true, 0x0, b"\"a\"]", None)).unwrap(); + let (opcode, payload) = client_frame(&mut s); + assert_eq!((opcode, payload.as_slice()), (0xA, b"ping!".as_slice())); + s.write_all(&frame(true, 0x1, long_copy.as_bytes(), Some([1, 2, 3, 4]))) + .unwrap(); + s.write_all(&frame(true, 0x8, &1000u16.to_be_bytes(), None)) + .unwrap(); + request + }); + let url = Url::parse(&format!("ws://127.0.0.1:{port}/relay")).unwrap(); + let mut socket = WebSocket::connect(&url, None, Duration::from_secs(5), None).unwrap(); + let deadline = Instant::now() + Duration::from_secs(5); + assert_eq!(socket.receive(deadline).unwrap(), "[\"NOTICE\",\"hi\"]"); + socket.send("[\"REQ\",\"a\",{}]").unwrap(); + assert_eq!(socket.receive(deadline).unwrap(), "[\"EOSE\",\"a\"]"); + assert_eq!(socket.receive(deadline).unwrap(), long); + assert_eq!( + socket.receive(deadline), + Err("closed by the relay".to_string()) + ); + let request = server.join().unwrap(); + assert!(request.starts_with("GET /relay HTTP/1.1\r\n"), "{request}"); + assert!(request.contains(&format!("\r\nHost: 127.0.0.1:{port}\r\n"))); + assert!(request.contains("\r\nUpgrade: websocket\r\n")); + assert!(request.contains("\r\nSec-WebSocket-Version: 13\r\n")); + let key = request + .lines() + .find_map(|l| l.strip_prefix("Sec-WebSocket-Key: ")) + .unwrap(); + assert_eq!(key.len(), 24); + } + + #[test] + fn not_a_relay() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = std::thread::spawn(move || { + let (mut s, _) = listener.accept().unwrap(); + let mut buf = [0u8; 1024]; + let _ = s.read(&mut buf).unwrap(); + s.write_all(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n") + .unwrap(); + }); + let url = Url::parse(&format!("ws://127.0.0.1:{port}/")).unwrap(); + let error = WebSocket::connect(&url, None, Duration::from_secs(5), None) + .err() + .unwrap(); + assert_eq!(error, "not a relay (HTTP 404)"); + server.join().unwrap(); + // And the other end of a frame stream: a frame too long is refused + // before it's read. + let mut huge = vec![0x81, 127]; + huge.extend_from_slice(&(u64::MAX).to_be_bytes()); + let mut socket = WebSocket::over(std::io::Cursor::new(huge)); + assert_eq!( + socket.receive(Instant::now() + Duration::from_secs(1)), + Err("a message too long".to_string()) + ); + } +} diff --git a/src/urls.rs b/src/urls.rs index 4ef7674..1dc7260 100644 --- a/src/urls.rs +++ b/src/urls.rs @@ -14,7 +14,11 @@ const TRACKING_PARAMS: &[&str] = &[ pub struct Url { /// Lowercase scheme, e.g. `https`. pub scheme: String, - /// Lowercase host, without port or credentials. + /// Lowercase host, without port or credentials. Except for `ipfs://`, + /// `ipns://` and `ar://`, whose "hosts" are content IDs, keys and + /// Arweave transaction IDs, which mean something else in another case + /// (`Qm…`, `12D3Koo…` in base58; Arweave's in base64). For a Nostr URI + /// (`nostr:npub1…`, NIP-21), what follows `nostr:`. pub host: String, /// Port, if one was given and it isn't the scheme's default. pub port: Option, @@ -28,9 +32,23 @@ pub struct Url { impl Url { /// Parses an absolute URL. Returns `None` for anything that isn't one. + /// + /// Nostr URIs have no `//` (`nostr:npub1…`): what follows `nostr:` is + /// kept as the host, lowercase (it's bech32, which has one case), with + /// `/` as the path. `nostr://npub1…` is taken as the same. pub fn parse(input: &str) -> Option { let input = input.trim(); - let (scheme, rest) = input.split_once("://")?; + let Some((scheme, rest)) = input.split_once("://") else { + let (scheme, rest) = input.split_once(':')?; + return if scheme.eq_ignore_ascii_case("nostr") { + nostr(rest) + } else { + None + }; + }; + if scheme.eq_ignore_ascii_case("nostr") { + return nostr(rest); + } if scheme.is_empty() || !scheme .chars() @@ -55,6 +73,14 @@ impl Url { // Drop credentials. let authority = authority.rsplit_once('@').map_or(authority, |(_, a)| a); + let keep_case = keeps_case(&scheme); + let fold = |host: &str| { + if keep_case { + host.to_string() + } else { + host.to_ascii_lowercase() + } + }; let (host, port) = if let Some(stripped) = authority.strip_prefix('[') { // IPv6 literal: [::1]:8080 let (h, after) = stripped.split_once(']')?; @@ -63,10 +89,10 @@ impl Url { } else { match authority.rsplit_once(':') { Some((h, p)) if !p.is_empty() && p.chars().all(|c| c.is_ascii_digit()) => { - (h.to_ascii_lowercase(), p.parse().ok()) + (fold(h), p.parse().ok()) } - Some((h, "")) => (h.to_ascii_lowercase(), None), - _ => (authority.to_ascii_lowercase(), None), + Some((h, "")) => (fold(h), None), + _ => (fold(authority), None), } }; let host = host.trim_end_matches('.').to_string(); @@ -92,6 +118,9 @@ impl Url { /// - A trailing slash is removed from the path (except for the root `/`). /// - Route-style fragments of JavaScript apps (`#!/…`, `#/…`) are kept. pub fn normalized(&self) -> String { + if self.scheme == "nostr" { + return format!("nostr:{}", self.host); + } let scheme = match self.scheme.as_str() { "http" | "https" => "web", other => other, @@ -155,6 +184,9 @@ impl Url { impl std::fmt::Display for Url { /// The URL written out again (without credentials). fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + if self.scheme == "nostr" { + return write!(f, "nostr:{}", self.host); + } write!(f, "{}://{}", self.scheme, self.host)?; if let Some(port) = self.port { write!(f, ":{port}")?; @@ -170,6 +202,23 @@ impl std::fmt::Display for Url { } } +/// A Nostr URI's entity (`npub1…`, `note1…`, `naddr1…`…): letters and +/// digits only, up to a `/`, `?` or `#`. +fn nostr(rest: &str) -> Option { + let entity = rest.split(['/', '?', '#']).next().unwrap_or(""); + if entity.is_empty() || !entity.chars().all(|c| c.is_ascii_alphanumeric()) { + return None; + } + Some(Url { + scheme: "nostr".into(), + host: entity.to_ascii_lowercase(), + port: None, + path: "/".into(), + query: None, + fragment: None, + }) +} + /// Resolves a link found on the page at `base` into an absolute `http(s)` /// URL, without its `#fragment`. Returns `None` for links that don't lead to /// another page (`javascript:`, `mailto:`, `#anchor`, `data:`, …). @@ -195,7 +244,7 @@ pub fn resolve_any(base: &Url, href: &str) -> Option { .chars() .all(|c| c.is_ascii_alphanumeric() || c == '+' || c == '-' || c == '.') }); - let absolute = if has_scheme { + let absolute = if has_scheme || lower.starts_with("nostr:") { Url::parse(href)? } else if lower.contains(':') && !lower.starts_with('/') && !lower.starts_with('?') { // Some other scheme (javascript:, mailto:, tel:, data:, …) — or a @@ -296,6 +345,21 @@ fn uppercase_escapes(s: &str) -> String { out } +/// Whether a scheme's hosts are case-sensitive: content IDs and keys +/// (`ipfs://`, `ipns://`, `ar://`), where `Qm…` and `qm…` are two things. +pub fn keeps_case(scheme: &str) -> bool { + matches!(scheme, "ipfs" | "ipns" | "ar") +} + +/// The host of `url` as it's looked up and sent: [`ascii_host`]'s, but +/// untouched for a scheme whose hosts keep their case. +pub fn wire_host(url: &Url) -> Option { + if keeps_case(&url.scheme) { + return (!url.host.is_empty()).then(|| url.host.clone()); + } + ascii_host(&url.host) +} + /// A host name as it's looked up and sent: internationalized labels in /// punycode (`café.example` → `xn--caf-dma.example`), lowercase. ASCII /// names (and `[IPv6]` literals) come back lowercased. `None` for a name @@ -444,6 +508,11 @@ fn default_port(scheme: &str) -> Option { "ftp" => Some(21), "gopher" => Some(70), "gemini" => Some(1965), + "spartan" => Some(300), + "nex" => Some(1900), + "finger" => Some(79), + "ws" => Some(80), + "wss" => Some(443), _ => None, } } @@ -456,6 +525,30 @@ mod tests { Url::parse(s).unwrap().normalized() } + #[test] + fn content_ids_keep_their_case() { + let cid = "QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR"; + assert_eq!(Url::parse(&format!("ipfs://{cid}/a")).unwrap().host, cid); + assert_eq!( + Url::parse("IPNS://12D3KooWExample/").unwrap().host, + "12D3KooWExample" + ); + assert_eq!( + Url::parse("https://Example.COM/").unwrap().host, + "example.com" + ); + let tx = "bNbA3TEQVL60xlgCcqdz4ZPHFZ711cZ3hmkpGttDt_U"; + assert_eq!(Url::parse(&format!("ar://{tx}/")).unwrap().host, tx); + // And as they're sent. + let url = Url::parse(&format!("ar://{tx}/")).unwrap(); + assert_eq!(wire_host(&url).as_deref(), Some(tx)); + let url = Url::parse(&format!("ipfs://{cid}/")).unwrap(); + assert_eq!(wire_host(&url).as_deref(), Some(cid)); + let mut url = Url::parse("gemini://example.org/").unwrap(); + url.host = "Example.ORG".into(); + assert_eq!(wire_host(&url).as_deref(), Some("example.org")); + } + #[test] fn same_page_variants_normalize_identically() { let a = norm("https://www.Example.com/recipes/catfish/"); @@ -621,6 +714,37 @@ mod tests { assert_eq!(site_host("example.com/foo").as_deref(), Some("example.com")); } + #[test] + fn nostr_uris() { + let npub = "npub10elfcs4fr0l0r8af98jlmgdh9c8tcxjvz9qkw038js35mp4dma8qzvjptg"; + for written in [ + format!("nostr:{npub}"), + format!("NOSTR:{}", npub.to_uppercase()), + format!("nostr://{npub}/"), + format!(" nostr:{npub}#x "), + ] { + let url = Url::parse(&written).unwrap(); + assert_eq!(url.scheme, "nostr", "{written}"); + assert_eq!(url.host, npub, "{written}"); + assert_eq!(url.path, "/"); + assert_eq!(url.to_string(), format!("nostr:{npub}")); + assert_eq!(url.normalized(), format!("nostr:{npub}")); + } + assert_eq!(Url::parse("nostr:"), None); + assert_eq!(Url::parse("nostr:not a thing"), None); + assert_eq!(Url::parse("mailto:someone@example.org"), None); + // As a link on a page. + let base = Url::parse("https://example.org/a/").unwrap(); + assert_eq!( + resolve_any(&base, &format!("nostr:{npub}")).map(|u| u.to_string()), + Some(format!("nostr:{npub}")) + ); + assert_eq!(resolve(&base, &format!("nostr:{npub}")), None); + // Relays' default ports. + assert_eq!(Url::parse("wss://relay.example:443/").unwrap().port, None); + assert_eq!(Url::parse("ws://relay.example:80").unwrap().port, None); + } + #[test] fn extensions() { let ext = |s: &str| Url::parse(s).unwrap().extension();