diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 85efdcd..1a9ec0d 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -255,7 +255,7 @@ Sources that aren't web pages are turned into pages at import time. A Hacker New |---|---|---|---| | `key` | string (indexed) | yes | `blake3(normalized URL)` (§6.5); one per document | | `url` | string | yes | The original URL, for display | -| `url_text` | text (tokenized URL) | no | `inurl:`, `allinurl:` | +| `url_text` | text (the URL's words, lowercase, and each punctuation mark as a token of its own: `gemini` `:` `/` `/` `capsule` `.` `example` …) | no | `inurl:`, `allinurl:` — so `inurl:gemini://` or `inurl:.onion/` match those characters, not just the words | | `host_rev` | string, multi-valued | no | `site:` — every level of the host, reversed (`com`, `com.example`, `com.example.www`), so `site:` is a single term lookup that includes subdomains | | `ext` | string | no | `ext:`, `filetype:` (from the URL path) | | `sources` | string, multi-valued | yes | `common_crawl`, `hacker_news`, … | @@ -263,6 +263,8 @@ Sources that aren't web pages are turned into pages at import time. A Hacker New | `body` | text | yes | Plain text; `intext:`, snippets | | `links` | string, multi-valued | yes | Outgoing links; `link:` with a full URL | | `link_hosts` | string, multi-valued | no | Every level of every linked host, reversed (like `host_rev`); `link:` with a site | +| `link_ext` | string, multi-valued | no | The file extensions of the links (`pdf`, `zip`, …); `contains:` | +| `lang` | string | no | The page's language, an ISO 639-3 code (`deu`), detected from its title and the first 1,000 characters of its text when it's indexed (`whatlang`, 70 languages); none when the text is too short or too mixed to tell. Not stored with the page: it's worked out again whenever the index is built. `lang:` | | `crawled` | date | yes | When the page was captured | | `import_id` | u64, indexed | yes | The import this version of the page came from | @@ -321,7 +323,7 @@ Merging only works if every source spells a URL the same way. The **key** is com Data written by one release of LOST must be readable by the next, or upgraded by it — never silently misread, and never thrown away without saying so. -- **Search index.** `index/lost-index.json` records the index's **layout** (`INDEX_LAYOUT` in `engine.rs`: its fields and what goes in them, §6.2) and the LOST that last opened it: `{"layout": 1, "written_by": "LOST 0.2.0"}`. Every release that changes the layout raises the number and adds an upgrade step from the layout before (`engine::upgrade`), done in place where possible — adding a field, say — or by rebuilding the index: from the capture archive once there is one (§6.7), otherwise by importing again (the message says so). An index from a **newer** LOST isn't touched; LOST says to update. An index made before stamps existed counts as layout 1. Tantivy's own file format is Tantivy's to keep compatible: upgrading Tantivy across a format break counts as a layout change. +- **Search index.** `index/lost-index.json` records the index's **layout** (`INDEX_LAYOUT` in `engine.rs`: its fields and what goes in them, §6.2) and the LOST that last opened it: `{"layout": 2, "written_by": "LOST 0.3.0"}`. Every release that changes the layout raises the number and adds an upgrade step from the layout before (`engine::upgrade`), done in place where possible, or by **rebuilding** the index from its stored fields, which hold all of every page (`url`, `title`, `body`, `links`, `sources`, `crawled`, `import_id`): everything else is worked out from them. A rebuild writes a new index next to the old one (`index.upgrading`, so it needs room for a second copy), holding the busy lock (§6.3), with progress on stderr, then swaps the folders (`index` → `index.old`, `index.upgrading` → `index`) and removes the old one; interrupted, it starts over next time, and an interrupted swap is undone when LOST opens. Layouts: **1** (LOST 0.1–0.2); **2** (0.3: URLs tokenized with their punctuation, `link_ext`, `lang`), rebuilt from 1. An index from a **newer** LOST isn't touched; LOST says to update. An index made before stamps existed counts as layout 1. Tantivy's own file format is Tantivy's to keep compatible: upgrading Tantivy across a format break counts as a layout change. - **App database.** redb tables are added, never repurposed: a new feature brings new tables (`captures`, say) and older ones stay as they are. A table whose rows have to change gets a new name, with an upgrade step that copies the rows over. - **`.lost` files** carry their format version in the first bytes and a schema version for the page records (§9.1). New page fields are optional, and readers skip fields they don't know, so a file from a newer LOST still imports into an older one, minus what's new. @@ -380,7 +382,11 @@ Two modules, kept independent so either can be inspected, tested or replaced on | `filetype:` / `ext:` | `filetype:pdf` | Term lookup on `ext` (case and a leading dot don't matter) | ✅ | | `link:` | `link:example.com`, `link:https://example.com/page` | A site (no path): pages linking anywhere on it, via `link_hosts`. A full URL: pages linking to exactly it, via `links`. Data from Common Crawl WAT files | ✅ | | `cache:` | `cache:https://example.com/page` | Not a search: the stored text of that one page (§7.6) | ✅ | -| `related:` | `related:example.com` | More-like-this, seeded from that site's documents | later | +| `related:` | `related:example.com`, `related:https://example.com/page` | Pages like that site's, on other sites: the 30 words that most set its pages apart (up to 20 of them, the first 16 KB of each; by how often they come up there and how rare they are elsewhere, as BM25 weighs them) are looked for, and a page needs a fifth of them (1–6). Words only those pages have, or that more than a tenth of all pages have, are left out. With a path: pages like that one page, which is left out. Matches nothing if the site or page isn't stored. Can be combined with the rest (`related:example.com lang:de`) | ✅ | +| `contains:` | `contains:pdf` | Pages linking to files of that type (Bing's operator): a term lookup on `link_ext` | ✅ | +| `lang:` / `language:` | `lang:de`, `language:german`, `-lang:en` | The page's detected language (`lang`). Takes ISO 639-1, 639-2 and 639-3 codes and English or native names (`de`, `deu`, `ger`, `German`, `Deutsch`); a language LOST can't tell matches nothing. Pages too short to tell have none, so they're left out of `lang:` and kept by `-lang:` | ✅ | +| `near:` | `catfish near:3 cornmeal`, `"fried catfish" near:2 cornmeal` | Bing's operator: words or phrases at most N words apart (5 without a number, 100 at most), in either order, in the title or the text (or where `allintitle:` and the like say). Chained, `a near:2 b near:4 c`, all of them, as far apart as the widest says (in that order or reversed). A sloppy phrase query; phrases must also be found as written. Binds tighter than `OR` | ✅ | +| `word*` | `catf*`, `intitle:catf*`, `"fried catf*"` | Words starting with what's before the `*` (at least 3 letters or digits): in each field, the 64 most common of them in the dictionary. In a phrase, only on the last word (Tantivy's phrase-prefix query, which takes the first 64 alphabetically). Shorter, it's just the word. Matches are highlighted | ✅ | An operator can be written apart from its value (`intitle: catfish`); one with no value at all is ignored. @@ -401,7 +407,7 @@ Offline means no "what other people searched". Suggestions come from the data it - **Completing the word being typed:** a prefix scan of the title and text term dictionaries (FSTs, so it's fast), most common words first. When other words come before it, the 20 most common completions are re-ranked by how many pages contain them *together with* those words, and completions that never appear with them are dropped. So `rust comp` offers `rust compiler` before `rust company`. - **`_` as a wildcard**, like Google: `how to _ catfish` finds pages with `how to` and `catfish` one word apart (titles and text), and offers the words that actually fill the gap there, most frequent first. -- Nothing is suggested once the last word is finished (followed by a space), or when it's an operator, an exclusion or a phrase. +- Nothing is suggested once the last word is finished (followed by a space), or when it's an operator, an exclusion, a phrase or a `*` prefix. Available as `lost --suggest TEXT` (one suggestion per line), and in step 4 as `/api/suggest` for the web UI and the GUI. @@ -769,6 +775,7 @@ lost/ ├── record.rs records, documents, merge rules (§6.4) ├── query.rs search syntax parser ├── engine.rs search engine (Tantivy) + ├── lang.rs pages' languages (whatlang), lang: names and codes ├── store.rs app database (redb) ├── lostfile.rs .lost format reader / writer ├── datasources.rs data source detection and shared helpers diff --git a/Cargo.lock b/Cargo.lock index f3bcfa2..a4f2486 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1558,6 +1558,12 @@ version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + [[package]] name = "foldhash" version = "0.2.0" @@ -1894,7 +1900,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed4a1bb24121291d27230c1b1b44e07d6a9b28cefdb32fe1581dfb84e14f940a" dependencies = [ "bytemuck", - "foldhash", + "foldhash 0.2.0", "hashbrown 0.17.1", "log", "peniko", @@ -2077,6 +2083,17 @@ dependencies = [ "smallvec", ] +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.1.5", +] + [[package]] name = "hashbrown" version = "0.16.1" @@ -2085,7 +2102,7 @@ checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" dependencies = [ "allocator-api2", "equivalent", - "foldhash", + "foldhash 0.2.0", ] [[package]] @@ -2094,7 +2111,7 @@ version = "0.17.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" dependencies = [ - "foldhash", + "foldhash 0.2.0", ] [[package]] @@ -2715,6 +2732,7 @@ dependencies = [ "tokio", "toml 1.1.8+spec-1.1.0", "tray-icon", + "whatlang", "windows-sys 0.61.2", "winit", "winresource", @@ -5446,6 +5464,15 @@ dependencies = [ "web-sys", ] +[[package]] +name = "whatlang" +version = "0.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5e8f38b596e2a359b755342473520a99421e43658548c79489ee221b728c107" +dependencies = [ + "hashbrown 0.15.5", +] + [[package]] name = "widestring" version = "1.2.1" diff --git a/Cargo.toml b/Cargo.toml index b446965..efffad7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -48,6 +48,8 @@ serde = { version = "1", features = ["derive"] } serde_json = "1" tantivy = "0.26" toml = "1" +# Telling pages' languages, for lang: (ARCHITECTURE.md §7.1). +whatlang = "0.18" terminal_size = { version = "0.4", optional = true } axum = { version = "0.8", optional = true } tokio = { version = "1", features = ["rt-multi-thread", "net", "signal", "macros"], optional = true } diff --git a/README.md b/README.md index 3aaa69b..7af33c5 100644 --- a/README.md +++ b/README.md @@ -54,12 +54,17 @@ lost -e rust.lost --sign rust # the same, signed with your ID (exports are u - `site:` - `intitle:` - `allintitle:` -- `inurl:` +- `inurl:` (`inurl:gemini://`, `inurl:.onion/`) - `allinurl:` - `intext:` - `allintext:` - `filetype:` - `link:` +- `contains:` (`contains:pdf`: pages linking to PDFs) +- `related:` (`related:example.com`) +- `lang:` / `language:` (`lang:de`, `language:german`) +- `near:` (`catfish near:3 cornmeal`) +- `word*` (`catf*`) ## Data diff --git a/src/engine.rs b/src/engine.rs index 6d3a57d..538b15c 100644 --- a/src/engine.rs +++ b/src/engine.rs @@ -11,13 +11,19 @@ use std::time::Instant; use serde::Serialize; use tantivy::collector::{Count, DocSetCollector, TopDocs}; use tantivy::directory::MmapDirectory; -use tantivy::query::{BooleanQuery, BoostQuery, Occur, PhraseQuery, Query, TermQuery}; -use tantivy::schema::{Field, INDEXED, IndexRecordOption, STORED, STRING, Schema, TEXT, Value}; +use tantivy::query::{ + BooleanQuery, BoostQuery, EmptyQuery, EnableScoring, Explanation, Occur, PhrasePrefixQuery, + PhraseQuery, Query, QueryClone, Scorer, TermQuery, Weight, +}; +use tantivy::schema::{ + Field, INDEXED, IndexRecordOption, STORED, STRING, Schema, TEXT, TextFieldIndexing, + TextOptions, Value, +}; use tantivy::snippet::SnippetGenerator; -use tantivy::tokenizer::{TextAnalyzer, TokenStream}; +use tantivy::tokenizer::{TextAnalyzer, Token, TokenStream, Tokenizer}; use tantivy::{ - DateTime, DocAddress, Index, IndexReader, IndexWriter, ReloadPolicy, Searcher, TantivyDocument, - Term, + DateTime, DocAddress, DocId, DocSet, Index, IndexReader, IndexWriter, ReloadPolicy, Score, + Searcher, SegmentReader, TantivyDocument, Term, }; use crate::Result; @@ -32,7 +38,11 @@ pub const INDEX_DIR: &str = "index"; /// The layout of the index: its fields and what goes in them (ARCHITECTURE.md /// §6.2). Raised whenever that changes, with a step in [`upgrade`] that /// brings an index of the layout before up to date. -pub const INDEX_LAYOUT: u32 = 1; +/// +/// - 1: LOST 0.1–0.2. +/// - 2: URLs tokenized with their punctuation (`inurl:gemini://`), links' +/// file extensions (`contains:`), pages' languages (`lang:`). +pub const INDEX_LAYOUT: u32 = 2; /// In the index folder: the layout it has, and the LOST that last opened it. const STAMP_FILE: &str = "lost-index.json"; @@ -68,6 +78,23 @@ const MAX_SUGGEST_TEXT: usize = 64 * 1024; /// Most combinations of `*` gap sizes tried for one phrase. const MAX_WILDCARD_LAYOUTS: usize = 9; +/// `catf*`: the most words it stands for (the most common ones), and the +/// most dictionary words looked at per field and segment to find them. +const MAX_PREFIX_TERMS: usize = 64; +const MAX_PREFIX_SCAN: usize = 200_000; + +/// `related:`: pages of a site read to learn what it's about, how much of +/// each page's text, and how many of its most telling words are looked for. +const RELATED_SEEDS: usize = 20; +const RELATED_TEXT: usize = 16 * 1024; +const RELATED_TERMS: usize = 30; + +/// The tokenizer of `url_text`: words and every punctuation mark on its own +/// (ARCHITECTURE.md §6.2), so `inurl:gemini://` means just that. +const URL_TOKENIZER: &str = "url"; +/// URL words longer than this (hashes, base64…) aren't indexed. +const MAX_URL_WORD: usize = 64; + /// Snippet length, in characters. const SNIPPET_CHARS: usize = 240; @@ -88,16 +115,104 @@ struct Fields { body: Field, links: Field, link_hosts: Field, + link_ext: Field, + lang: Field, crawled: Field, import_id: Field, } +impl Fields { + fn stored(&self) -> Stored { + Stored { + key: self.key, + url: self.url, + sources: self.sources, + title: self.title, + body: self.body, + links: self.links, + crawled: self.crawled, + import_id: self.import_id, + } + } +} + +/// The stored fields, which hold everything a page is: what an index of +/// any layout is rebuilt from. +#[derive(Clone, Copy)] +struct Stored { + key: Field, + url: Field, + sources: Field, + title: Field, + body: Field, + links: Field, + crawled: Field, + import_id: Field, +} + +impl Stored { + /// The stored fields of an index, found by name (so in an index of an + /// older layout too). + fn of(schema: &Schema) -> Result { + let field = |name: &str| { + schema + .get_field(name) + .map_err(|_| format!("the search index has no {name} field")) + }; + Ok(Stored { + key: field("key")?, + url: field("url")?, + sources: field("sources")?, + title: field("title")?, + body: field("body")?, + links: field("links")?, + crawled: field("crawled")?, + import_id: field("import_id")?, + }) + } + + fn decode(&self, d: &TantivyDocument) -> Doc { + let text = |field: Field| -> Option { + d.get_first(field) + .and_then(|v| v.as_str().map(str::to_string)) + }; + let number = |field: Field| -> Option { d.get_first(field).and_then(|v| v.as_u64()) }; + let date = |field: Field| -> Option { + d.get_first(field) + .and_then(|v| v.as_datetime()) + .map(|t| t.into_timestamp_secs()) + }; + let all = |field: Field| -> Vec { + d.get_all(field) + .filter_map(|v| v.as_str().map(str::to_string)) + .collect() + }; + Doc { + key: text(self.key).unwrap_or_default(), + url: text(self.url).unwrap_or_default(), + sources: all(self.sources), + title: text(self.title), + body: text(self.body).unwrap_or_default(), + links: all(self.links), + crawled: date(self.crawled), + import_id: number(self.import_id).unwrap_or(0), + } + } +} + fn schema() -> (Schema, Fields) { let mut b = Schema::builder(); let fields = Fields { key: b.add_text_field("key", STRING | STORED), url: b.add_text_field("url", STORED), - url_text: b.add_text_field("url_text", TEXT), + url_text: b.add_text_field( + "url_text", + TextOptions::default().set_indexing_options( + TextFieldIndexing::default() + .set_tokenizer(URL_TOKENIZER) + .set_index_option(IndexRecordOption::WithFreqsAndPositions), + ), + ), host_rev: b.add_text_field("host_rev", STRING), ext: b.add_text_field("ext", STRING), sources: b.add_text_field("sources", STRING | STORED), @@ -105,6 +220,8 @@ fn schema() -> (Schema, Fields) { body: b.add_text_field("body", TEXT | STORED), links: b.add_text_field("links", STRING | STORED), link_hosts: b.add_text_field("link_hosts", STRING), + link_ext: b.add_text_field("link_ext", STRING), + lang: b.add_text_field("lang", STRING), crawled: b.add_date_field("crawled", STORED), import_id: b.add_u64_field("import_id", INDEXED | STORED), }; @@ -143,15 +260,19 @@ fn check_layout(dir: &Path) -> Result<()> { while now < INDEX_LAYOUT { now = upgrade(dir, now)?; } + write_stamp(dir, now) +} + +/// Stamps the index in `dir` with `layout`, and this LOST as the last to +/// open it (unless it says that already). +fn write_stamp(dir: &Path, layout: u32) -> Result<()> { + let path = dir.join(STAMP_FILE); let written_by = format!("LOST {}", env!("CARGO_PKG_VERSION")); let current = std::fs::read(&path) .ok() .and_then(|b| serde_json::from_slice::(&b).ok()); - if current.is_none_or(|s| s.layout != now || s.written_by != written_by) { - let stamp = serde_json::to_vec_pretty(&Stamp { - layout: now, - written_by, - })?; + if current.is_none_or(|s| s.layout != layout || s.written_by != written_by) { + let stamp = serde_json::to_vec_pretty(&Stamp { layout, written_by })?; let partial = dir.join(format!("{STAMP_FILE}.part")); std::fs::write(&partial, stamp) .and_then(|()| std::fs::rename(&partial, &path)) @@ -160,17 +281,270 @@ fn check_layout(dir: &Path) -> Result<()> { Ok(()) } -/// Brings an index of layout `from` to the next layout, returning it. Each -/// layout change adds its step here: done in place where it can be, or by -/// rebuilding the index — from the capture archive once there is one -/// (§6.7), otherwise by importing again. There are none yet. +/// Brings an index of layout `from` up to date, returning the layout it has +/// now. Each layout change adds its step here: done in place where it can +/// be, or by rebuilding the index from its stored fields. fn upgrade(dir: &Path, from: u32) -> Result { - Err(format!( - "the search index in {} was made by an older version of LOST (layout {from}) \ - and can't be upgraded; remove it and import your datasets again", - dir.display() - ) - .into()) + match from { + // Layout 2 changes only what's worked out from the stored fields. + 1 => { + rebuild(dir)?; + Ok(INDEX_LAYOUT) + } + _ => Err(format!( + "the search index in {} has a layout this LOST doesn't know ({from}); \ + remove it and import your datasets again", + dir.display() + ) + .into()), + } +} + +/// `index.old`, `index.upgrading`: folders next to the index's. +fn sibling(dir: &Path, what: &str) -> std::path::PathBuf { + let name = dir + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_else(|| INDEX_DIR.to_string()); + dir.with_file_name(format!("{name}.{what}")) +} + +/// After an upgrade was interrupted while swapping folders: the index is +/// put back (it'll be upgraded again), or what's left of the old one goes. +fn recover(dir: &Path) -> Result<()> { + let previous = sibling(dir, "old"); + if !previous.exists() { + return Ok(()); + } + if dir.exists() { + std::fs::remove_dir_all(&previous) + .map_err(|e| format!("can't remove {}: {e}", previous.display()))?; + } else { + std::fs::rename(&previous, dir) + .map_err(|e| format!("can't move {} back: {e}", previous.display()))?; + } + Ok(()) +} + +/// Builds the index in `dir` again, in the current layout, from what it +/// stores — every page is all there — next to it in `index.upgrading`, +/// then puts it in its place. Needs room for a second copy while it runs. +/// The busy lock keeps imports and other upgrades out meanwhile. +fn rebuild(dir: &Path) -> Result<()> { + let data_dir = dir.parent().unwrap_or(dir); + let _busy = crate::store::BusyLock::acquire(data_dir, "upgrade of the search index")?; + let fresh = sibling(dir, "upgrading"); + let previous = sibling(dir, "old"); + let _ = std::fs::remove_dir_all(&fresh); + std::fs::create_dir_all(&fresh) + .map_err(|e| format!("can't create {}: {e}", fresh.display()))?; + + { + let old = Index::open(MmapDirectory::open(dir)?) + .map_err(|e| format!("can't open the index in {}: {e}", dir.display()))?; + let stored = Stored::of(&old.schema())?; + // Read once, with nothing watching the folder: it's moved afterwards. + let reader: IndexReader = old + .reader_builder() + .reload_policy(ReloadPolicy::Manual) + .try_into()?; + let searcher = reader.searcher(); + let total = searcher.num_docs(); + + let (schema, f) = schema(); + let index = Index::create_in_dir(&fresh, schema) + .map_err(|e| format!("can't create an index in {}: {e}", fresh.display()))?; + register_tokenizers(&index); + let threads = writer_threads(); + let mut writer: IndexWriter = + index.writer_with_num_threads(threads, threads * 50_000_000)?; + + let interactive = std::io::IsTerminal::is_terminal(&std::io::stderr()); + let report = |done: u64| { + let line = format!( + "Upgrading the search index for LOST {}: {} of {} pages…", + env!("CARGO_PKG_VERSION"), + crate::text::thousands(done), + crate::text::thousands(total) + ); + if interactive { + eprint!("\r{line}"); + } else if done == 0 || done == total { + eprintln!("{line}"); + } + }; + report(0); + let mut batch: Vec = Vec::with_capacity(BATCH_SIZE); + let mut done = 0; + for (ord, segment) in searcher.segment_readers().iter().enumerate() { + for id in segment.doc_ids_alive() { + let td: TantivyDocument = searcher.doc(DocAddress::new(ord as u32, id))?; + batch.push(stored.decode(&td)); + if batch.len() >= BATCH_SIZE { + done += write_docs(&writer, &f, &mut batch)?; + report(done); + } + } + } + done += write_docs(&writer, &f, &mut batch)?; + writer.commit()?; + writer.wait_merging_threads()?; + report(done); + if interactive { + eprintln!(); + } + write_stamp(&fresh, INDEX_LAYOUT)?; + } + + std::fs::rename(dir, &previous) + .map_err(|e| format!("can't move {} aside: {e}", dir.display()))?; + std::fs::rename(&fresh, dir).map_err(|e| { + let _ = std::fs::rename(&previous, dir); + format!("can't put the upgraded index in {}: {e}", dir.display()) + })?; + std::fs::remove_dir_all(&previous) + .map_err(|e| format!("can't remove {}: {e}", previous.display()))?; + Ok(()) +} + +/// Indexing threads: as many as there are cores, up to 4. +fn writer_threads() -> usize { + std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(1) + .clamp(1, 4) +} + +/// Adds `docs` (emptying it) to the index, and returns how many. +fn write_docs(writer: &IndexWriter, f: &Fields, docs: &mut Vec) -> Result { + let languages = detect_languages(docs); + let count = docs.len() as u64; + for (doc, lang) in docs.drain(..).zip(languages) { + writer.add_document(encode(f, &doc, lang))?; + } + Ok(count) +} + +/// The language of each page (ARCHITECTURE.md §7.1), worked out on every +/// core: it's most of the work of indexing a page. +fn detect_languages(docs: &[Doc]) -> Vec> { + let threads = std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(1) + .clamp(1, 8); + if docs.len() < 64 || threads == 1 { + return docs + .iter() + .map(|d| crate::lang::detect(d.title.as_deref(), &d.body)) + .collect(); + } + let size = docs.len().div_ceil(threads); + std::thread::scope(|scope| { + let handles: Vec<_> = docs + .chunks(size) + .map(|chunk| { + scope.spawn(move || { + chunk + .iter() + .map(|d| crate::lang::detect(d.title.as_deref(), &d.body)) + .collect::>() + }) + }) + .collect(); + handles + .into_iter() + .flat_map(|h| h.join().unwrap_or_default()) + .collect() + }) +} + +/// Registers the tokenizers the schema names, besides Tantivy's own. +fn register_tokenizers(index: &Index) { + index.tokenizers().register(URL_TOKENIZER, UrlTokenizer); +} + +/// A URL's words (letters and digits, lowercase) and each punctuation mark +/// on its own, with their byte offsets and positions: `gemini://a.b/c` is +/// `gemini` `:` `/` `/` `a` `.` `b` `/` `c`. A word longer than +/// [`MAX_URL_WORD`] bytes is left out, leaving a gap. +fn url_tokens(text: &str) -> Vec { + let mut tokens = Vec::new(); + let mut position = 0; + let mut chars = text.char_indices().peekable(); + while let Some((from, c)) = chars.next() { + let to = if c.is_alphanumeric() { + let mut to = from + c.len_utf8(); + while let Some(&(at, next)) = chars.peek() { + if !next.is_alphanumeric() { + break; + } + to = at + next.len_utf8(); + chars.next(); + } + to + } else if c.is_whitespace() { + continue; + } else { + from + c.len_utf8() + }; + if to - from <= MAX_URL_WORD { + tokens.push(Token { + offset_from: from, + offset_to: to, + position, + text: text[from..to].to_lowercase(), + position_length: 1, + }); + } + position += 1; + } + tokens +} + +#[derive(Clone)] +struct UrlTokenizer; + +struct UrlTokenStream { + tokens: Vec, + /// The token `token()` gives, plus one: none before the first `advance`. + next: usize, + empty: Token, +} + +impl Tokenizer for UrlTokenizer { + type TokenStream<'a> = UrlTokenStream; + fn token_stream<'a>(&'a mut self, text: &'a str) -> UrlTokenStream { + UrlTokenStream { + tokens: url_tokens(text), + next: 0, + empty: Token::default(), + } + } +} + +impl TokenStream for UrlTokenStream { + fn advance(&mut self) -> bool { + if self.next < self.tokens.len() { + self.next += 1; + true + } else { + false + } + } + + fn token(&self) -> &Token { + match self.next.checked_sub(1) { + Some(at) => &self.tokens[at], + None => &self.empty, + } + } + + fn token_mut(&mut self) -> &mut Token { + match self.next.checked_sub(1) { + Some(at) => &mut self.tokens[at], + None => &mut self.empty, + } + } } /// One search result. @@ -231,12 +605,14 @@ impl Engine { /// Opens the index in `data_dir`, creating it if it doesn't exist. pub fn open(data_dir: &Path) -> Result { let dir = data_dir.join(INDEX_DIR); + recover(&dir)?; std::fs::create_dir_all(&dir) .map_err(|e| format!("can't create {}: {e}", dir.display()))?; check_layout(&dir)?; let (schema, f) = schema(); let index = Index::open_or_create(MmapDirectory::open(&dir)?, schema) .map_err(|e| format!("can't open the index in {}: {e}", dir.display()))?; + register_tokenizers(&index); let reader = index .reader_builder() .reload_policy(ReloadPolicy::Manual) @@ -296,25 +672,15 @@ impl Engine { snippets.set_max_num_chars(SNIPPET_CHARS); // The index terms of every word and phrase that isn't excluded. + let marks = self.marks(node.as_ref())?; let mut analyzer = self.index.tokenizer_for_field(self.f.body)?; - let terms: HashSet = node - .as_ref() - .map(query::positive_terms) - .unwrap_or_default() - .iter() - .flat_map(|t| { - tokenize(&mut analyzer, t) - .into_iter() - .map(|(text, _, _)| text) - }) - .collect(); let mut word_ranges = |text: &str| -> Vec<(usize, usize)> { - if terms.is_empty() { + if marks.is_empty() { return Vec::new(); } tokenize(&mut analyzer, text) .into_iter() - .filter(|(token, _, _)| terms.contains(token)) + .filter(|(token, _, _)| marks.matches(token)) .map(|(_, from, to)| (from, to)) .collect() }; @@ -379,6 +745,7 @@ impl Engine { || last.contains(':') || last.starts_with('-') || last.contains('"') + || last.ends_with('*') { return Ok(Vec::new()); } @@ -536,24 +903,15 @@ impl Engine { return Ok(None); }; - let terms: HashSet = highlight - .and_then(query::parse) - .map(|node| query::positive_terms(&node)) - .unwrap_or_default() - .iter() - .map(|t| self.tokens(t)) - .collect::>>()? - .into_iter() - .flatten() - .collect(); + let marks = self.marks(highlight.and_then(query::parse).as_ref())?; let mut analyzer = self.index.tokenizer_for_field(self.f.body)?; let mut matches = |text: &str| -> Vec<[usize; 2]> { - if terms.is_empty() { + if marks.is_empty() { return Vec::new(); } let bytes: Vec<(usize, usize)> = tokenize(&mut analyzer, text) .into_iter() - .filter(|(token, _, _)| terms.contains(token)) + .filter(|(token, _, _)| marks.matches(token)) .map(|(_, from, to)| (from, to)) .collect(); char_ranges(text, &bytes) @@ -661,10 +1019,7 @@ impl Engine { /// Starts adding documents. Only one indexer may exist at a time, across /// all processes; the busy lock (store.rs) makes sure of that. pub fn indexer(&self) -> Result> { - let threads = std::thread::available_parallelism() - .map(|n| n.get()) - .unwrap_or(1) - .clamp(1, 4); + let threads = writer_threads(); let writer = self .index .writer_with_num_threads(threads, threads * 50_000_000)?; @@ -710,6 +1065,18 @@ impl Engine { Some(LinkTarget::Host) => site_term(target).map(|t| term(self.f.link_hosts, &t)), None => None, }, + Node::Near { + parts, + distance, + scope, + } => self.near_query(parts, *distance, *scope)?, + Node::Contains(ext) => (!ext.is_empty()).then(|| term(self.f.link_ext, ext)), + Node::Related(target) => self.related_query(target)?, + // A language LOST can't tell matches nothing. + Node::Lang(name) => Some(match crate::lang::code(name) { + Some(code) => term(self.f.lang, code), + None => Box::new(EmptyQuery) as Box, + }), // An exclusion on its own (e.g. in an OR) asks for nothing. Node::Not(_) => None, Node::And(children) => { @@ -718,7 +1085,7 @@ impl Engine { match child { Node::Not(inner) => { if let Some(q) = self.build(inner, fields)? { - clauses.push((Occur::MustNot, q)); + clauses.push((Occur::MustNot, Box::new(Excluded(q)))); } } other => { @@ -750,12 +1117,42 @@ impl Engine { /// A word or phrase searched in the given fields. Text that tokenizes into /// several words ("e-mail", "fried catfish") is matched as a phrase; a `*` - /// in a phrase stands for one to three missing words. + /// in a phrase stands for one to three missing words, and one at the end + /// of the last word (`catf*`) for the rest of it. fn text_query(&self, text: &str, fields: &[(Field, f32)]) -> Result>> { + let mut clauses: Vec<(Occur, Box)> = Vec::new(); + for &(field, boost) in fields { + if let Some(q) = self.field_text_query(text, field)? { + clauses.push((Occur::Should, Box::new(BoostQuery::new(q, boost)))); + } + } + Ok((!clauses.is_empty()).then(|| Box::new(BooleanQuery::new(clauses)) as Box)) + } + + /// [`Engine::text_query`] in one field, tokenized as that field is + /// (URLs keep their punctuation). `None` if nothing in `text` is a word + /// there. + fn field_text_query(&self, text: &str, field: Field) -> Result>> { + let mut analyzer = self.index.tokenizer_for_field(field)?; + let pieces = query::wildcard_chunks(text); + let last_piece = pieces.len().saturating_sub(1); // Each chunk's terms, and the number of `*` before it. let mut chunks: Vec<(usize, Vec)> = Vec::new(); - for (stars, chunk) in query::wildcard_chunks(text) { - let tokens = self.tokens(&chunk)?; + // Whether the last term is a prefix (`catf*`). + let mut prefix = false; + for (i, (stars, chunk)) in pieces.into_iter().enumerate() { + let words: Vec<&str> = chunk.split_whitespace().collect(); + let mut tokens = Vec::new(); + for (j, word) in words.iter().enumerate() { + let last = i == last_piece && j + 1 == words.len(); + let (piece, is_prefix) = match query::prefix(word) { + Some(stem) => (stem, last), + None => (word.trim_end_matches('*'), false), + }; + let found = tokens_of(&mut analyzer, piece); + prefix = is_prefix && !found.is_empty(); + tokens.extend(found); + } if tokens.is_empty() { continue; } @@ -768,37 +1165,279 @@ impl Engine { let gaps: Vec = chunks.iter().skip(1).map(|(stars, _)| *stars).collect(); let layouts = query::gap_layouts(&gaps, MAX_WILDCARD_LAYOUTS); + let mut alternatives: Vec<(Occur, Box)> = Vec::new(); + for layout in &layouts { + // Term positions: consecutive within a chunk, with a gap before each later chunk. + let mut terms = Vec::new(); + let mut position = 0; + for (i, (_, tokens)) in chunks.iter().enumerate() { + if i > 0 { + position += layout[i - 1]; + } + for token in tokens { + terms.push((position, Term::from_field_text(field, token))); + position += 1; + } + } + let q: Box = if prefix && terms.len() == 1 { + let last = chunks.last().and_then(|(_, t)| t.last()).cloned(); + self.prefix_query(field, &last.unwrap_or_default())? + } else if prefix { + let mut q = PhrasePrefixQuery::new_with_offset(terms); + q.set_max_expansions(MAX_PREFIX_TERMS as u32); + Box::new(q) + } else if terms.len() == 1 { + Box::new(TermQuery::new( + terms.remove(0).1, + IndexRecordOption::WithFreqs, + )) + } else if layout.iter().all(|&g| g == 0) { + Box::new(PhraseQuery::new( + terms.into_iter().map(|(_, t)| t).collect(), + )) + } else { + Box::new(PhraseQuery::new_with_offset(terms)) + }; + alternatives.push((Occur::Should, q)); + } + Ok(Some(if alternatives.len() == 1 { + alternatives.remove(0).1 + } else { + Box::new(BooleanQuery::new(alternatives)) + })) + } + + /// `catf*` in one field: its [`MAX_PREFIX_TERMS`] most common words + /// starting with `prefix`. Matches nothing if there are none. + fn prefix_query(&self, field: Field, prefix: &str) -> Result> { + let searcher = self.reader.searcher(); + let mut counts: HashMap = HashMap::new(); + for segment in searcher.segment_readers() { + let index = segment.inverted_index(field)?; + let mut stream = index.terms().range().ge(prefix.as_bytes()).into_stream()?; + let mut seen = 0; + while stream.advance() && seen < MAX_PREFIX_SCAN { + let key = stream.key(); + if !key.starts_with(prefix.as_bytes()) { + break; + } + seen += 1; + if let Ok(word) = std::str::from_utf8(key) { + *counts.entry(word.to_string()).or_default() += + u64::from(stream.value().doc_freq); + } + } + } + let mut words: Vec<(String, u64)> = counts.into_iter().collect(); + words.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0))); + words.truncate(MAX_PREFIX_TERMS); + if words.is_empty() { + return Ok(Box::new(EmptyQuery)); + } + let clauses = words + .into_iter() + .map(|(word, _)| { + let q: Box = Box::new(TermQuery::new( + Term::from_field_text(field, &word), + IndexRecordOption::WithFreqs, + )); + (Occur::Should, q) + }) + .collect(); + Ok(Box::new(BooleanQuery::new(clauses))) + } + + /// `catfish near:3 cornmeal`: the words (or phrases) at most `distance` + /// words apart, in either order, in the title or the text (or where + /// `scope` says). Phrases must also be found as they're written. + fn near_query( + &self, + parts: &[String], + distance: u32, + scope: Option, + ) -> Result>> { + let fields = match scope { + Some(scope) => self.text_fields(Some(scope)), + None => vec![(self.f.title, BOOST_TITLE), (self.f.body, BOOST_BODY)], + }; let mut clauses: Vec<(Occur, Box)> = Vec::new(); - for &(field, boost) in fields { - for layout in &layouts { - // Term positions: consecutive within a chunk, with a gap before each later chunk. - let mut terms = Vec::new(); - let mut position = 0; - for (i, (_, tokens)) in chunks.iter().enumerate() { - if i > 0 { - position += layout[i - 1]; - } - for token in tokens { - terms.push((position, Term::from_field_text(field, token))); - position += 1; + for (field, boost) in fields { + let mut analyzer = self.index.tokenizer_for_field(field)?; + // Each part's terms (stars mean nothing here). + let tokenized: Vec<(&String, Vec)> = parts + .iter() + .map(|part| (part, tokens_of(&mut analyzer, &part.replace('*', " ")))) + .filter(|(_, tokens)| !tokens.is_empty()) + .collect(); + let q: Option> = match tokenized.as_slice() { + [] => None, + // Nothing to be near: just the one part. + [(part, _)] => self.field_text_query(part, field)?, + _ => { + let phrase = |order: Vec<&Vec>| -> Box { + let terms: Vec<(usize, Term)> = order + .into_iter() + .flatten() + .enumerate() + .map(|(i, word)| (i, Term::from_field_text(field, word))) + .collect(); + Box::new(PhraseQuery::new_with_offset_and_slop(terms, distance)) + }; + let forward = phrase(tokenized.iter().map(|(_, t)| t).collect()); + let backward = phrase(tokenized.iter().rev().map(|(_, t)| t).collect()); + let mut must: Vec<(Occur, Box)> = vec![( + Occur::Must, + Box::new(BooleanQuery::new(vec![ + (Occur::Should, forward), + (Occur::Should, backward), + ])), + )]; + for (_, tokens) in &tokenized { + if tokens.len() > 1 { + let terms = tokens + .iter() + .map(|word| Term::from_field_text(field, word)) + .collect(); + must.push((Occur::Must, Box::new(PhraseQuery::new(terms)))); + } } + Some(Box::new(BooleanQuery::new(must))) } - let q: Box = if terms.len() == 1 { - Box::new(TermQuery::new( - terms.remove(0).1, - IndexRecordOption::WithFreqs, - )) - } else if layout.iter().all(|&g| g == 0) { - Box::new(PhraseQuery::new( - terms.into_iter().map(|(_, t)| t).collect(), - )) - } else { - Box::new(PhraseQuery::new_with_offset(terms)) - }; + }; + if let Some(q) = q { clauses.push((Occur::Should, Box::new(BoostQuery::new(q, boost)))); } } - Ok(Some(Box::new(BooleanQuery::new(clauses)))) + Ok((!clauses.is_empty()).then(|| Box::new(BooleanQuery::new(clauses)) as Box)) + } + + /// `related:example.com`: pages that use the words that most set the + /// site's pages apart, on other sites. `related:URL` (with a path): pages + /// like that one page. Matches nothing if the site or page isn't stored. + fn related_query(&self, target: &str) -> Result>> { + let searcher = self.reader.searcher(); + let nothing = || Ok(Some(Box::new(EmptyQuery) as Box)); + let target = target.trim(); + let as_url = if target.contains("://") { + target.to_string() + } else { + format!("https://{target}") + }; + let page = Url::parse(&as_url).filter(|u| u.path != "/" || u.query.is_some()); + let (seeds, exclude): (Vec, Box) = match page { + Some(_) => { + let Some(key) = crate::record::url_key(&as_url) else { + return nothing(); + }; + let Some(doc) = self.load(&searcher, &key)? else { + return nothing(); + }; + let this = TermQuery::new( + Term::from_field_text(self.f.key, &key), + IndexRecordOption::Basic, + ); + (vec![doc], Box::new(this)) + } + None => { + let Some(site) = site_term(target) else { + return Ok(None); + }; + let on_site = TermQuery::new( + Term::from_field_text(self.f.host_rev, &site), + IndexRecordOption::Basic, + ); + let top = searcher.search( + &on_site, + &TopDocs::with_limit(RELATED_SEEDS).order_by_score(), + )?; + let mut docs = Vec::new(); + for (_, address) in top { + docs.push(self.decode(&searcher.doc::(address)?)); + } + (docs, Box::new(on_site)) + } + }; + if seeds.is_empty() { + return nothing(); + } + + // How often each word comes up in the seeds' titles and texts. + let mut analyzer = self.index.tokenizer_for_field(self.f.body)?; + let mut counts: HashMap<(Field, String), u32> = HashMap::new(); + for doc in &seeds { + let texts = [ + (self.f.title, doc.title.as_deref().unwrap_or("")), + (self.f.body, truncate_at_word(&doc.body, RELATED_TEXT)), + ]; + for (field, text) in texts { + for word in tokens_of(&mut analyzer, text) { + let telling = + word.chars().count() >= 3 && !word.chars().all(|c| c.is_numeric()); + if telling { + *counts.entry((field, word)).or_default() += 1; + } + } + } + } + // Weighed by how rare they are elsewhere (as BM25's IDF does). Words + // only the seeds have can't find anything else; words on more than a + // tenth of all pages (in a database big enough to say) tell nothing. + let total = searcher.num_docs() as f32; + let mut scored: Vec<(Field, String, f32)> = Vec::new(); + for ((field, word), count) in counts { + let doc_freq = searcher.doc_freq(&Term::from_field_text(field, &word))? as f32; + if doc_freq <= seeds.len() as f32 || (total >= 1000.0 && doc_freq * 10.0 > total) { + continue; + } + let idf = ((total - doc_freq + 0.5) / (doc_freq + 0.5) + 1.0).ln(); + scored.push((field, word, count as f32 * idf)); + } + scored.sort_by(|a, b| b.2.total_cmp(&a.2).then_with(|| a.1.cmp(&b.1))); + scored.truncate(RELATED_TERMS); + let Some(best) = scored.first().map(|(_, _, score)| *score) else { + return nothing(); + }; + let clauses: Vec<(Occur, Box)> = scored + .iter() + .map(|(field, word, score)| { + let term = TermQuery::new( + Term::from_field_text(*field, word), + IndexRecordOption::WithFreqs, + ); + let q: Box = Box::new(BoostQuery::new(Box::new(term), score / best)); + (Occur::Should, q) + }) + .collect(); + // A page has to share a few of those words, not just one. + let required = (clauses.len() / 5).clamp(1, 6); + let similar = BooleanQuery::with_minimum_required_clauses(clauses, required); + Ok(Some(Box::new(BooleanQuery::new(vec![ + (Occur::Must, Box::new(similar)), + (Occur::MustNot, exclude), + ])))) + } + + /// What to mark in titles and snippets: the index terms of every word and + /// phrase of `node` that isn't excluded, and the beginnings of words + /// that `catf*` stands for. + fn marks(&self, node: Option<&Node>) -> Result { + let mut marks = Marks::default(); + let mut analyzer = self.index.tokenizer_for_field(self.f.body)?; + for text in node.map(query::positive_terms).unwrap_or_default() { + for word in text.split_whitespace() { + match query::prefix(word) { + Some(stem) => { + let mut tokens = tokens_of(&mut analyzer, stem); + if let Some(last) = tokens.pop() { + marks.prefixes.push(last); + } + marks.words.extend(tokens); + } + None => marks.words.extend(tokens_of(&mut analyzer, word)), + } + } + } + Ok(marks) } /// Splits text into index terms, the same way the text fields were indexed. @@ -827,72 +1466,145 @@ impl Engine { } } - fn encode(&self, doc: &Doc) -> TantivyDocument { - let f = &self.f; - let mut d = TantivyDocument::default(); - d.add_text(f.key, &doc.key); - d.add_text(f.url, &doc.url); - d.add_text(f.url_text, &doc.url); - if let Some(url) = Url::parse(&doc.url) { - for suffix in url.host_suffixes() { - d.add_text(f.host_rev, &suffix); - } - if let Some(ext) = url.extension() { - d.add_text(f.ext, &ext); - } - } - for source in &doc.sources { - d.add_text(f.sources, source); - } - if let Some(title) = &doc.title { - d.add_text(f.title, title); - } - d.add_text(f.body, &doc.body); - let mut hosts = HashSet::new(); - for link in &doc.links { - d.add_text(f.links, link); - if let Some(url) = Url::parse(link) { - hosts.extend(url.host_suffixes()); - } + fn decode(&self, d: &TantivyDocument) -> Doc { + self.f.stored().decode(d) + } +} + +/// What to mark in a result: [`Engine::marks`]. +#[derive(Default)] +struct Marks { + words: HashSet, + prefixes: Vec, +} + +impl Marks { + fn is_empty(&self) -> bool { + self.words.is_empty() && self.prefixes.is_empty() + } + + fn matches(&self, token: &str) -> bool { + self.words.contains(token) || self.prefixes.iter().any(|p| token.starts_with(p.as_str())) + } +} + +/// A page as the index holds it; `lang` is its language (§7.1). +fn encode(f: &Fields, doc: &Doc, lang: Option<&str>) -> TantivyDocument { + let mut d = TantivyDocument::default(); + d.add_text(f.key, &doc.key); + d.add_text(f.url, &doc.url); + d.add_text(f.url_text, &doc.url); + if let Some(url) = Url::parse(&doc.url) { + for suffix in url.host_suffixes() { + d.add_text(f.host_rev, &suffix); } - for host in hosts { - d.add_text(f.link_hosts, &host); + if let Some(ext) = url.extension() { + d.add_text(f.ext, &ext); } - if let Some(crawled) = doc.crawled { - d.add_date(f.crawled, DateTime::from_timestamp_secs(crawled)); + } + for source in &doc.sources { + d.add_text(f.sources, source); + } + if let Some(title) = &doc.title { + d.add_text(f.title, title); + } + d.add_text(f.body, &doc.body); + let mut hosts = HashSet::new(); + let mut extensions = HashSet::new(); + for link in &doc.links { + d.add_text(f.links, link); + if let Some(url) = Url::parse(link) { + hosts.extend(url.host_suffixes()); + extensions.extend(url.extension()); } - d.add_u64(f.import_id, doc.import_id); - d } + for host in hosts { + d.add_text(f.link_hosts, &host); + } + for extension in extensions { + d.add_text(f.link_ext, &extension); + } + if let Some(lang) = lang { + d.add_text(f.lang, lang); + } + if let Some(crawled) = doc.crawled { + d.add_date(f.crawled, DateTime::from_timestamp_secs(crawled)); + } + d.add_u64(f.import_id, doc.import_id); + d +} - fn decode(&self, d: &TantivyDocument) -> Doc { - let f = &self.f; - let text = |field: Field| -> Option { - d.get_first(field) - .and_then(|v| v.as_str().map(str::to_string)) - }; - let number = |field: Field| -> Option { d.get_first(field).and_then(|v| v.as_u64()) }; - let date = |field: Field| -> Option { - d.get_first(field) - .and_then(|v| v.as_datetime()) - .map(|t| t.into_timestamp_secs()) - }; - let all = |field: Field| -> Vec { - d.get_all(field) - .filter_map(|v| v.as_str().map(str::to_string)) - .collect() - }; +/// A query whose matches are excluded (`-"exact phrase"`, `-inurl:gemini://`). +/// +/// Tantivy checks each candidate against an exclusion with `seek_danger`, +/// which for a phrase asserts the candidate isn't before the phrase's +/// current match — but a phrase scorer moves to its first match as soon as +/// it's made, so the assertion fails in debug builds whenever that match +/// comes after the first candidate (release builds answer correctly). This +/// wrapper leaves `seek_danger` to `DocSet`'s default, which allows it. +struct Excluded(Box); + +impl Clone for Excluded { + fn clone(&self) -> Self { + Excluded(self.0.box_clone()) + } +} - Doc { - key: text(f.key).unwrap_or_default(), - url: text(f.url).unwrap_or_default(), - sources: all(f.sources), - title: text(f.title), - body: text(f.body).unwrap_or_default(), - links: all(f.links), - crawled: date(f.crawled), - import_id: number(f.import_id).unwrap_or(0), - } +impl std::fmt::Debug for Excluded { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_tuple("Excluded").field(&self.0).finish() + } +} + +impl Query for Excluded { + fn weight(&self, enable_scoring: EnableScoring<'_>) -> tantivy::Result> { + Ok(Box::new(ExcludedWeight(self.0.weight(enable_scoring)?))) + } + + fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) { + self.0.query_terms(visitor); + } +} + +struct ExcludedWeight(Box); + +impl Weight for ExcludedWeight { + fn scorer(&self, reader: &SegmentReader, boost: Score) -> tantivy::Result> { + Ok(Box::new(ExcludedScorer(self.0.scorer(reader, boost)?))) + } + + fn explain(&self, reader: &SegmentReader, doc: DocId) -> tantivy::Result { + self.0.explain(reader, doc) + } +} + +struct ExcludedScorer(Box); + +impl DocSet for ExcludedScorer { + fn advance(&mut self) -> DocId { + self.0.advance() + } + + fn seek(&mut self, target: DocId) -> DocId { + self.0.seek(target) + } + + fn doc(&self) -> DocId { + self.0.doc() + } + + fn size_hint(&self) -> u32 { + self.0.size_hint() + } + + fn cost(&self) -> u64 { + self.0.cost() + } +} + +impl Scorer for ExcludedScorer { + fn score(&mut self) -> Score { + self.0.score() } } @@ -927,6 +1639,14 @@ fn tokenize(analyzer: &mut TextAnalyzer, text: &str) -> Vec<(String, usize, usiz out } +/// Index terms of `text`. +fn tokens_of(analyzer: &mut TextAnalyzer, text: &str) -> Vec { + tokenize(analyzer, text) + .into_iter() + .map(|(term, _, _)| term) + .collect() +} + /// Counts from an indexing run. #[derive(Debug, Clone, Copy, Default)] pub struct IndexStats { @@ -975,6 +1695,7 @@ impl Indexer<'_> { engine.reader.reload()?; let searcher = engine.reader.searcher(); + let mut docs = Vec::with_capacity(self.batch.len()); for (key, doc) in self.batch.drain() { let doc = match engine.load(&searcher, &key)? { Some(existing) => { @@ -993,8 +1714,9 @@ impl Indexer<'_> { }; self.writer .delete_term(Term::from_field_text(engine.f.key, &key)); - self.writer.add_document(engine.encode(&doc))?; + docs.push(doc); } + write_docs(&self.writer, &engine.f, &mut docs)?; self.writer.commit()?; engine.reader.reload()?; Ok(()) @@ -1026,11 +1748,15 @@ mod layout_tests { let stamp = std::fs::read_to_string(new.join(STAMP_FILE)).unwrap(); assert!(stamp.contains(&format!("\"layout\": {INDEX_LAYOUT}"))); - // One from before stamps is layout 1, which is current. + // One stamped by an older LOST, of the current layout, gets this + // LOST's name. (One from before stamps is layout 1: upgraded, see + // `search_tests::upgrading_from_layout_1`.) let old = temp("old"); - std::fs::write(old.join("meta.json"), "{}").unwrap(); + let stamp = format!("{{\"layout\": {INDEX_LAYOUT}, \"written_by\": \"LOST 0.0.1\"}}"); + std::fs::write(old.join(STAMP_FILE), stamp).unwrap(); check_layout(&old).unwrap(); - assert!(old.join(STAMP_FILE).exists()); + let stamp = std::fs::read_to_string(old.join(STAMP_FILE)).unwrap(); + assert!(stamp.contains(env!("CARGO_PKG_VERSION")), "{stamp}"); // One from a newer LOST is left alone. let newer = temp("newer"); @@ -1051,3 +1777,250 @@ mod layout_tests { } } } + +#[cfg(test)] +mod search_tests { + use super::*; + + fn temp(name: &str) -> std::path::PathBuf { + let dir = std::env::temp_dir().join(format!("lost-search-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + dir + } + + fn doc(url: &str, title: &str, body: &str, links: &[&str]) -> Doc { + Doc { + key: crate::record::url_key(url).unwrap(), + url: url.into(), + sources: vec!["test".into()], + title: Some(title.into()), + body: body.into(), + links: links.iter().map(|l| l.to_string()).collect(), + crawled: None, + import_id: 1, + } + } + + fn found(engine: &Engine, query: &str) -> Vec { + let mut urls: Vec = engine + .search(query, Limit::new(0, 100)) + .unwrap() + .results + .into_iter() + .map(|hit| hit.url) + .collect(); + urls.sort(); + urls + } + + const GERMAN: &str = "Das ist ein ganz normaler deutscher Satz über das Wetter in Berlin, \ + und noch einer über die Stadt und ihre vielen schönen Häuser."; + + fn pages() -> Vec { + vec![ + doc( + "gemini://capsule.example/notes/catfish.gmi", + "Fried catfish", + "Roll the catfish in cornmeal and fry it.", + &["gemini://capsule.example/recipes.pdf"], + ), + doc( + "https://web.example/gemini/protocol", + "About the Gemini protocol", + "Cornmeal is not mentioned here, but catfish are, far away from that \ + other word: one two three four five six seven eight nine ten cornmeal.", + &["https://web.example/spec.html"], + ), + doc( + "http://hidden.onion/index.html", + "Hidden service", + GERMAN, + &[], + ), + ] + } + + fn engine_with(name: &str, docs: Vec) -> (Engine, std::path::PathBuf) { + let dir = temp(name); + let engine = Engine::open(&dir).unwrap(); + let mut indexer = engine.indexer().unwrap(); + for d in docs { + indexer.add(d).unwrap(); + } + indexer.finish().unwrap(); + (engine, dir) + } + + #[test] + fn url_words_and_punctuation() { + let tokens: Vec<(String, usize)> = url_tokens("Gemini://A.b/c-d?é=1") + .into_iter() + .map(|t| (t.text, t.position)) + .collect(); + let expected: Vec<(String, usize)> = [ + "gemini", ":", "/", "/", "a", ".", "b", "/", "c", "-", "d", "?", "é", "=", "1", + ] + .iter() + .enumerate() + .map(|(i, t)| (t.to_string(), i)) + .collect(); + assert_eq!(tokens, expected); + // A long word is left out, but keeps its place. + let long = format!("a/{}/b", "x".repeat(MAX_URL_WORD + 1)); + let tokens: Vec<(String, usize)> = url_tokens(&long) + .into_iter() + .map(|t| (t.text, t.position)) + .collect(); + assert_eq!( + tokens, + [ + ("a".into(), 0), + ("/".into(), 1), + ("/".into(), 3), + ("b".into(), 4) + ] + ); + } + + #[test] + fn operators() { + let (engine, dir) = engine_with("operators", pages()); + let gemini = "gemini://capsule.example/notes/catfish.gmi"; + let web = "https://web.example/gemini/protocol"; + let onion = "http://hidden.onion/index.html"; + + // inurl: with punctuation. + assert_eq!(found(&engine, "inurl:gemini"), [gemini, web]); + assert_eq!(found(&engine, "inurl:gemini://"), [gemini]); + assert_eq!(found(&engine, "inurl:.onion/"), [onion]); + assert_eq!(found(&engine, "allinurl:https:// gemini"), [web]); + assert_eq!(found(&engine, "-inurl:gemini:// catfish"), [web]); + + // near: + assert_eq!(found(&engine, "catfish near:3 cornmeal"), [gemini]); + assert_eq!(found(&engine, "cornmeal near:3 catfish"), [gemini]); + assert_eq!(found(&engine, "catfish near:30 cornmeal").len(), 2); + assert_eq!(found(&engine, "catfish cornmeal").len(), 2); + + // Prefixes. + assert_eq!(found(&engine, "catf*").len(), 2); + assert_eq!(found(&engine, "intitle:catf*"), [gemini]); + assert_eq!(found(&engine, "\"fried catf*\""), [gemini]); + assert_eq!(found(&engine, "catfz*"), Vec::::new()); + assert_eq!(found(&engine, "inurl:capsu*"), [gemini]); + let hits = engine.search("corn*", Limit::new(0, 10)).unwrap(); + assert!(hits.results.iter().all(|h| !h.snippet_matches.is_empty())); + + // contains: + assert_eq!(found(&engine, "contains:pdf"), [gemini]); + assert_eq!(found(&engine, "contains:.HTML"), [web]); + + // lang: + assert_eq!(found(&engine, "lang:de"), [onion]); + assert_eq!(found(&engine, "language:German"), [onion]); + assert_eq!(found(&engine, "lang:deutsch Berlin"), [onion]); + assert!(found(&engine, "lang:klingon").is_empty()); + assert!(found(&engine, "lang:de -lang:deu").is_empty()); + + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn related_pages() { + let words = "catfish cornmeal buttermilk skillet hushpuppies remoulade"; + let mut docs = vec![ + doc("https://fish.example/", "Fish fry", words, &[]), + doc("https://fish.example/more", "More fish fry", words, &[]), + doc("https://other.example/fry", "Southern fry", words, &[]), + doc( + "https://unrelated.example/", + "Gardening", + "tomatoes compost trellis mulch seedlings", + &[], + ), + ]; + // Enough other pages for the words to be rare. + for i in 0..20 { + docs.push(doc( + &format!("https://filler.example/{i}"), + "Filler", + "nothing to see here at all", + &[], + )); + } + let (engine, dir) = engine_with("related", docs); + assert_eq!( + found(&engine, "related:fish.example"), + ["https://other.example/fry"] + ); + assert_eq!( + found(&engine, "related:https://other.example/fry"), + ["https://fish.example/", "https://fish.example/more"] + ); + assert!(found(&engine, "related:nowhere.example").is_empty()); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn upgrading_from_layout_1() { + // An index as LOST 0.2 made it: URLs tokenized like text, no link + // extensions or languages, no stamp. + let data = temp("upgrade"); + let dir = data.join(INDEX_DIR); + std::fs::create_dir_all(&dir).unwrap(); + { + let mut b = Schema::builder(); + let key = b.add_text_field("key", STRING | STORED); + let url = b.add_text_field("url", STORED); + let url_text = b.add_text_field("url_text", TEXT); + b.add_text_field("host_rev", STRING); + b.add_text_field("ext", STRING); + let sources = b.add_text_field("sources", STRING | STORED); + let title = b.add_text_field("title", TEXT | STORED); + let body = b.add_text_field("body", TEXT | STORED); + let links = b.add_text_field("links", STRING | STORED); + b.add_text_field("link_hosts", STRING); + b.add_date_field("crawled", STORED); + let import_id = b.add_u64_field("import_id", INDEXED | STORED); + let index = Index::create_in_dir(&dir, b.build()).unwrap(); + let mut writer: IndexWriter = index.writer_with_num_threads(1, 15_000_000).unwrap(); + for page in pages() { + let mut d = TantivyDocument::default(); + d.add_text(key, &page.key); + d.add_text(url, &page.url); + d.add_text(url_text, &page.url); + d.add_text(sources, "test"); + d.add_text(title, page.title.as_deref().unwrap()); + d.add_text(body, &page.body); + for link in &page.links { + d.add_text(links, link); + } + d.add_u64(import_id, 7); + writer.add_document(d).unwrap(); + } + writer.commit().unwrap(); + } + let engine = Engine::open(&data).unwrap(); + assert_eq!(engine.num_docs(), 3); + assert_eq!( + found(&engine, "inurl:gemini://"), + ["gemini://capsule.example/notes/catfish.gmi"] + ); + assert_eq!( + found(&engine, "lang:de"), + ["http://hidden.onion/index.html"] + ); + assert_eq!(engine.count_import(7).unwrap(), 3); + let stamp = std::fs::read_to_string(dir.join(STAMP_FILE)).unwrap(); + assert!(stamp.contains(&format!("\"layout\": {INDEX_LAYOUT}"))); + assert!(!sibling(&dir, "old").exists()); + assert!(!sibling(&dir, "upgrading").exists()); + drop(engine); + // Opened again: nothing more to do. + let engine = Engine::open(&data).unwrap(); + assert_eq!(engine.num_docs(), 3); + drop(engine); + let _ = std::fs::remove_dir_all(data); + } +} diff --git a/src/lang.rs b/src/lang.rs new file mode 100644 index 0000000..7576be6 --- /dev/null +++ b/src/lang.rs @@ -0,0 +1,197 @@ +//! Page languages, for `lang:` / `language:` (ARCHITECTURE.md §7.1). +//! +//! A page's language is guessed from its text when it's indexed, with +//! `whatlang` (70 languages, trigram statistics, no data files). It isn't +//! stored with the page or in `.lost` files: it's worked out again from the +//! text whenever the index is built, so better detection later needs no +//! change anywhere else. Languages are kept as ISO 639-3 codes (`deu`), and +//! `lang:` takes any of the usual ways to write one: `de`, `deu`, `ger`, +//! `german`, `deutsch`. + +/// How much of a page is read to tell its language: enough to be sure, +/// little enough not to slow imports down. +const SAMPLE_CHARS: usize = 1_000; + +/// The language of a page, as an ISO 639-3 code, or `None` when its text is +/// too short or too mixed to tell. +pub fn detect(title: Option<&str>, body: &str) -> Option<&'static str> { + let mut sample = String::with_capacity(SAMPLE_CHARS * 2); + if let Some(title) = title { + sample.push_str(title); + sample.push('\n'); + } + sample.extend(body.chars().take(SAMPLE_CHARS)); + let info = whatlang::detect(&sample)?; + info.is_reliable().then(|| info.lang().code()) +} + +/// Other ways to write the languages `whatlang` knows: ISO 639-1 codes, +/// ISO 639-2/B codes, and common names that aren't its own. Its codes +/// (ISO 639-3) and its English and native names work too, without being +/// listed here. +const ALIASES: &[(&str, &[&str])] = &[ + ("afr", &["af"]), + ("aka", &["ak"]), + ("amh", &["am"]), + ("ara", &["ar"]), + ("aze", &["az"]), + ("bel", &["be"]), + ("ben", &["bn", "bangla"]), + ("bul", &["bg"]), + ("cat", &["ca"]), + ("ces", &["cs", "cze"]), + ("cmn", &["zh", "zho", "chi", "chinese"]), + ("cym", &["cy", "wel"]), + ("dan", &["da"]), + ("deu", &["de", "ger"]), + ("ell", &["el", "gre"]), + ("eng", &["en"]), + ("epo", &["eo"]), + ("est", &["et"]), + ("fin", &["fi"]), + ("fra", &["fr", "fre"]), + ("guj", &["gu"]), + ("heb", &["he", "iw"]), + ("hin", &["hi"]), + ("hrv", &["hr"]), + ("hun", &["hu"]), + ("hye", &["hy", "arm"]), + ("ind", &["id", "in"]), + ("ita", &["it"]), + ("jav", &["jv"]), + ("jpn", &["ja"]), + ("kan", &["kn"]), + ("kat", &["ka", "geo"]), + ("khm", &["km"]), + ("kor", &["ko"]), + ("lat", &["la"]), + ("lav", &["lv"]), + ("lit", &["lt"]), + ("mal", &["ml"]), + ("mar", &["mr"]), + ("mkd", &["mk", "mac"]), + ("mya", &["my", "bur", "myanmar"]), + ("nep", &["ne"]), + ("nld", &["nl", "dut", "flemish"]), + ("nob", &["nb", "no", "nor", "norwegian"]), + ("ori", &["or", "odia"]), + ("pan", &["pa", "panjabi"]), + ("pes", &["fa", "fas", "per", "farsi"]), + ("pol", &["pl"]), + ("por", &["pt"]), + ("ron", &["ro", "rum", "moldovan"]), + ("rus", &["ru"]), + ("sin", &["si", "sinhala"]), + ("slk", &["sk", "slo"]), + ("slv", &["sl", "slovenian"]), + ("sna", &["sn"]), + ("spa", &["es"]), + ("srp", &["sr"]), + ("swe", &["sv"]), + ("tam", &["ta"]), + ("tel", &["te"]), + ("tgl", &["tl", "fil", "filipino"]), + ("tha", &["th"]), + ("tuk", &["tk"]), + ("tur", &["tr"]), + ("ukr", &["uk"]), + ("urd", &["ur"]), + ("uzb", &["uz"]), + ("vie", &["vi"]), + ("yid", &["yi", "ji"]), + ("zul", &["zu"]), +]; + +/// The ISO 639-3 code of a language written as a code or a name, in any +/// case: `de`, `DEU`, `ger`, `German`, `Deutsch` all give `deu`. `None` for +/// a language LOST can't tell apart (it would match nothing). +pub fn code(name: &str) -> Option<&'static str> { + let name = name.trim().to_lowercase(); + if name.is_empty() { + return None; + } + for lang in whatlang::Lang::all() { + if lang.code() == name + || lang.eng_name().to_lowercase() == name + || lang.name().to_lowercase() == name + { + return Some(lang.code()); + } + } + ALIASES + .iter() + .find(|(_, aliases)| aliases.contains(&name.as_str())) + .map(|(code, _)| *code) +} + +/// A language's English name, from its ISO 639-3 code. +pub fn english_name(code: &str) -> Option<&'static str> { + whatlang::Lang::from_code(code).map(|lang| lang.eng_name()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn names_and_codes() { + for name in ["de", "DEU", "ger", "German", "deutsch", " Deutsch "] { + assert_eq!(code(name), Some("deu"), "{name}"); + } + assert_eq!(code("русский"), Some("rus")); + assert_eq!(code("zh"), Some("cmn")); + assert_eq!(code("Chinese"), Some("cmn")); + assert_eq!(code("Mandarin"), Some("cmn")); + assert_eq!(code("no"), Some("nob")); + assert_eq!(code("farsi"), Some("pes")); + assert_eq!(code("klingon"), None); + assert_eq!(code(""), None); + assert_eq!(english_name("deu"), Some("German")); + assert_eq!(english_name("xyz"), None); + } + + #[test] + fn every_language_has_its_short_code() { + // Each of whatlang's languages can be written as a two-letter code, + // and no alias points at a language it doesn't know. + for lang in whatlang::Lang::all() { + let listed = ALIASES.iter().find(|(c, _)| *c == lang.code()); + let short = listed.and_then(|(_, a)| a.iter().find(|a| a.len() == 2)); + assert!(short.is_some(), "{} has no two-letter code", lang.code()); + } + for (c, _) in ALIASES { + assert!(whatlang::Lang::from_code(*c).is_some(), "{c}"); + } + // And no alias is taken twice. + let mut all: Vec<&str> = ALIASES + .iter() + .flat_map(|(_, a)| a.iter().copied()) + .collect(); + let count = all.len(); + all.sort_unstable(); + all.dedup(); + assert_eq!(all.len(), count); + } + + #[test] + fn detection() { + assert_eq!( + detect( + Some("Das Wetter"), + "Das ist ein ganz normaler deutscher Satz über das Wetter in Berlin und die Stadt." + ), + Some("deu") + ); + assert_eq!( + detect( + None, + "I2P is an anonymous network layer that allows for censorship-resistant, \ + peer-to-peer communication between its users." + ), + Some("eng") + ); + // Too little to tell. + assert_eq!(detect(Some("Welcome"), ""), None); + assert_eq!(detect(None, ""), None); + } +} diff --git a/src/lib.rs b/src/lib.rs index 9b345ee..e521ed5 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -18,6 +18,7 @@ pub mod config; pub mod datasources; pub mod engine; +pub mod lang; pub mod lostfile; pub mod query; pub mod record; diff --git a/src/query.rs b/src/query.rs index 05fd106..7f0d1dd 100644 --- a/src/query.rs +++ b/src/query.rs @@ -3,19 +3,27 @@ //! Turns what was typed into the search box into an engine-agnostic syntax //! tree. No Tantivy types here; `engine.rs` translates the tree. //! -//! Words, `"phrases"` (with `*` for missing words), `-exclusions`, `OR` / `|`, -//! `AND`, `( )`, and the operators `site:`, `intitle:`, `allintitle:`, -//! `inurl:`, `allinurl:`, `intext:`, `allintext:`, `filetype:` / `ext:` and -//! `link:`. Implicit AND binds loosest, as on Google: -//! `catfish OR tilapia recipe` means `(catfish OR tilapia) AND recipe`. +//! Words (`catf*` for any word starting with `catf`), `"phrases"` (with `*` +//! for missing words), `-exclusions`, `OR` / `|`, `AND`, `( )`, `near:N`, +//! and the operators `site:`, `intitle:`, `allintitle:`, `inurl:`, +//! `allinurl:`, `intext:`, `allintext:`, `filetype:` / `ext:`, `link:`, +//! `contains:`, `related:` and `lang:` / `language:`. Implicit AND binds +//! loosest, as on Google: `catfish OR tilapia recipe` means +//! `(catfish OR tilapia) AND recipe`; `near:` binds tightest. use std::fmt; use std::str::FromStr; +/// `near:` without a number: at most this many words apart. +pub const DEFAULT_NEAR: u32 = 5; +/// The most `near:` takes. +pub const MAX_NEAR: u32 = 100; + /// A parsed query. #[derive(Debug, Clone, PartialEq, Eq)] pub enum Node { - /// A single word as typed (the engine tokenizes it). + /// A single word as typed (the engine tokenizes it). One ending in `*` + /// (`catf*`) stands for every word starting with what's before it. Word(String), /// An exact phrase. Phrase(String), @@ -28,6 +36,22 @@ pub enum Node { Ext(String), /// `link:` — pages linking to this URL, or to anywhere on this site. Link(String), + /// `catfish near:3 cornmeal` — words or phrases at most `distance` words + /// apart, in either order; in one part of the page if `scope` says so + /// (`allintitle:`, or `intitle:` on each of them). + Near { + parts: Vec, + distance: u32, + scope: Option, + }, + /// `contains:` — pages linking to files with this extension (lowercase, + /// without the dot). + Contains(String), + /// `related:` — pages like the ones on this site, or like this one page. + Related(String), + /// `lang:` / `language:` — pages in this language, as typed (a code or + /// a name; the engine works out which language it is). + Lang(String), /// Results must not match this. Not(Box), /// All of these must match. @@ -78,6 +102,10 @@ enum Op { AllIn(Scope), Ext, Link, + Near, + Contains, + Related, + Lang, } #[derive(Debug, Clone, PartialEq, Eq)] @@ -173,6 +201,10 @@ fn classify(word: &str) -> Token { "allintext" => Some(Op::AllIn(Scope::Text)), "filetype" | "ext" => Some(Op::Ext), "link" => Some(Op::Link), + "near" => Some(Op::Near), + "contains" => Some(Op::Contains), + "related" => Some(Op::Related), + "lang" | "language" => Some(Op::Lang), _ => None, }; if let Some(op) = op { @@ -229,11 +261,11 @@ impl Parser { } fn parse_or(&mut self) -> Option { - let first = self.parse_unary(); + let first = self.parse_near(); let mut alternatives: Vec = first.into_iter().collect(); while self.peek() == Some(&Token::Or) { self.pos += 1; - if let Some(node) = self.parse_unary() { + if let Some(node) = self.parse_near() { alternatives.push(node); } } @@ -244,6 +276,24 @@ impl Parser { } } + /// `a near:3 b near:2 c`: what's on either side of each `near:`. + fn parse_near(&mut self) -> Option { + let mut left = self.parse_unary(); + while let Some(Token::Op(Op::Near, value)) = self.peek().cloned() { + self.pos += 1; + let distance = value + .and_then(|v| v.parse::().ok()) + .unwrap_or(DEFAULT_NEAR) + .min(MAX_NEAR); + let right = self.parse_unary(); + left = match (left, right) { + (Some(l), Some(r)) => Some(near(l, r, distance)), + (l, r) => l.or(r), + }; + } + left + } + fn parse_unary(&mut self) -> Option { match self.peek()? { Token::Neg => { @@ -310,21 +360,81 @@ impl Parser { .operand(value) .map(|n| Node::Ext(text(n).trim_start_matches('.').to_ascii_lowercase())), Op::Link => self.operand(value).map(|n| Node::Link(text(n))), + Op::Contains => self + .operand(value) + .map(|n| Node::Contains(text(n).trim_start_matches('.').to_ascii_lowercase())), + Op::Related => self.operand(value).map(|n| Node::Related(text(n))), + Op::Lang => self.operand(value).map(|n| Node::Lang(text(n))), + // Only between two words or phrases (`parse_near`). + Op::Near => None, } } } +/// `left near:distance right`. Only words and phrases can be near each other +/// (both in the same part of the page, if one is named); anything else is +/// simply required as well, as if `near:` weren't there. +fn near(left: Node, right: Node, distance: u32) -> Node { + fn part(node: &Node) -> Option<(String, Option)> { + match node { + Node::Word(w) | Node::Phrase(w) => Some((w.clone(), None)), + Node::In(scope, inner) => match inner.as_ref() { + Node::Word(w) | Node::Phrase(w) => Some((w.clone(), Some(*scope))), + _ => None, + }, + _ => None, + } + } + let joined = match (&left, part(&right)) { + ( + Node::Near { + parts, + distance: d, + scope, + }, + Some((text, s)), + ) if *scope == s => Some(Node::Near { + parts: parts.iter().cloned().chain([text]).collect(), + distance: distance.max(*d), + scope: s, + }), + (_, Some((r, rs))) => match part(&left) { + Some((l, ls)) if ls == rs => Some(Node::Near { + parts: vec![l, r], + distance, + scope: ls, + }), + _ => None, + }, + _ => None, + }; + joined.unwrap_or_else(|| match left { + Node::And(mut children) => { + children.push(right); + Node::And(children) + } + left => Node::And(vec![left, right]), + }) +} + /// The words a result is expected to contain: every word and phrase that /// isn't excluded. Used to highlight matches. pub fn positive_terms(node: &Node) -> Vec { fn walk(node: &Node, out: &mut Vec) { match node { Node::Word(w) | Node::Phrase(w) => out.push(w.clone()), + Node::Near { parts, .. } => out.extend(parts.iter().cloned()), Node::And(children) | Node::Or(children) => { children.iter().for_each(|c| walk(c, out)); } Node::In(_, inner) => walk(inner, out), - Node::Not(_) | Node::Site(_) | Node::Ext(_) | Node::Link(_) => {} + Node::Not(_) + | Node::Site(_) + | Node::Ext(_) + | Node::Link(_) + | Node::Contains(_) + | Node::Related(_) + | Node::Lang(_) => {} } } let mut out = Vec::new(); @@ -343,6 +453,18 @@ pub fn has_positive(node: &Node) -> bool { } } +/// `catf*`: the word before the `*`, if `word` is a prefix to look up. A +/// lone `*` isn't one (in a phrase it stands for missing words), nor is one +/// with fewer than [`MIN_PREFIX`] characters, which would stand for too many +/// words to be of use. +pub fn prefix(word: &str) -> Option<&str> { + let stem = word.strip_suffix('*')?.trim_end_matches('*'); + (stem.chars().filter(|c| c.is_alphanumeric()).count() >= MIN_PREFIX).then_some(stem) +} + +/// The fewest letters or digits a `*` prefix can have. +pub const MIN_PREFIX: usize = 3; + /// Splits a phrase at standalone `*` words: the text chunks in order, each /// with the number of `*` right before it. Leading and trailing `*` say /// nothing and are dropped: `"how to * catfish"` gives `[(0, "how to"), (1, "catfish")]`. @@ -605,6 +727,110 @@ mod tests { assert_eq!(parse("foo:bar"), Some(w("foo:bar"))); } + fn near_of(parts: &[&str], distance: u32, scope: Option) -> Node { + Node::Near { + parts: parts.iter().map(|p| p.to_string()).collect(), + distance, + scope, + } + } + + #[test] + fn near() { + assert_eq!( + parse("catfish near:3 cornmeal"), + Some(near_of(&["catfish", "cornmeal"], 3, None)) + ); + assert_eq!( + parse("\"fried catfish\" NEAR:2 cornmeal recipe"), + Some(Node::And(vec![ + near_of(&["fried catfish", "cornmeal"], 2, None), + w("recipe") + ])) + ); + // No number, or one that isn't: the default. Too big: the most. + assert_eq!( + parse("a near: b"), + Some(near_of(&["a", "b"], DEFAULT_NEAR, None)) + ); + assert_eq!( + parse("a near:x b"), + Some(near_of(&["a", "b"], DEFAULT_NEAR, None)) + ); + assert_eq!( + parse("a near:5000 b"), + Some(near_of(&["a", "b"], MAX_NEAR, None)) + ); + // Chained: all of them, as far apart as the widest says. + assert_eq!( + parse("a near:2 b near:4 c"), + Some(near_of(&["a", "b", "c"], 4, None)) + ); + // Binds tighter than OR. + assert_eq!( + parse("a near:2 b OR c"), + Some(Node::Or(vec![near_of(&["a", "b"], 2, None), w("c")])) + ); + // In one part of the page. + assert_eq!( + parse("allintitle: fried near:1 catfish"), + Some(near_of(&["fried", "catfish"], 1, Some(Scope::Title))) + ); + assert_eq!( + parse("intitle:fried near:1 intitle:catfish"), + Some(near_of(&["fried", "catfish"], 1, Some(Scope::Title))) + ); + // What can't be near something is just required. + assert_eq!( + parse("site:x.com near:3 b"), + Some(Node::And(vec![Node::Site("x.com".into()), w("b")])) + ); + assert_eq!( + parse("intitle:a near:3 b"), + Some(Node::And(vec![within(Scope::Title, w("a")), w("b")])) + ); + assert_eq!(parse("near:3 b"), Some(w("b"))); + assert_eq!(parse("a near:3"), Some(w("a"))); + assert_eq!( + positive_terms(&parse("a near:3 \"b c\"").unwrap()), + vec!["a", "b c"] + ); + } + + #[test] + fn prefixes() { + assert_eq!(parse("catf*"), Some(w("catf*"))); + assert_eq!(prefix("catf*"), Some("catf")); + assert_eq!(prefix("catf**"), Some("catf")); + assert_eq!(prefix("ca*"), None); + assert_eq!(prefix("*"), None); + assert_eq!(prefix("catfish"), None); + assert_eq!(prefix("e-ma*"), Some("e-ma")); + } + + #[test] + fn links_languages_and_relatives() { + assert_eq!(parse("contains:.PDF"), Some(Node::Contains("pdf".into()))); + assert_eq!( + parse("related:example.com"), + Some(Node::Related("example.com".into())) + ); + assert_eq!(parse("lang:de"), Some(Node::Lang("de".into()))); + assert_eq!( + parse("Language: German catfish"), + Some(Node::And(vec![Node::Lang("German".into()), w("catfish")])) + ); + assert_eq!( + parse("catfish -lang:en"), + Some(Node::And(vec![ + w("catfish"), + Node::Not(Box::new(Node::Lang("en".into()))) + ])) + ); + assert!(has_positive(&parse("lang:de").unwrap())); + assert!(positive_terms(&parse("lang:de related:x.org contains:pdf a").unwrap()) == ["a"]); + } + #[test] fn exclusions_alone_ask_for_nothing() { assert!(!has_positive(&parse("-site:news.ycombinator.com").unwrap()));