diff --git a/README.md b/README.md index 023ca5b..d7aa60e 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ print!("{}", out.to_table()); ## Documentation -- [Install](https://db.mqlang.org/book/start/install) and [CLI](https://db.mqlang.org/book/start/cli) (`index`, `find`, `lint`, `stats`, `vacuum`, …) +- [Install](https://db.mqlang.org/book/start/install) and [CLI](https://db.mqlang.org/book/start/cli) (`index`, `find`, `stats`, `vacuum`, …) - [Library API](https://db.mqlang.org/book/start/library), [HTTP server](https://db.mqlang.org/book/start/http-server) - [SQL schema](https://db.mqlang.org/book/reference/sql-schema), [functions](https://db.mqlang.org/book/reference/sql-functions) and [examples](https://db.mqlang.org/book/reference/sql-examples) - [Block model](https://db.mqlang.org/book/reference/block-model), [index layers](https://db.mqlang.org/book/reference/index-layers) and [storage format](https://db.mqlang.org/book/reference/storage-format) diff --git a/docs/books/src/reference/index-layers.md b/docs/books/src/reference/index-layers.md index 1f1be43..ed57cae 100644 --- a/docs/books/src/reference/index-layers.md +++ b/docs/books/src/reference/index-layers.md @@ -31,8 +31,6 @@ Built once per document and stored in the `.mq-db` file. Checked before any bloc | `depth = N` (`N > 0`) | `N` exceeds `max_heading_depth` | | `block_type = 'heading' AND content = 'X'` | `heading_contents` has no case-insensitive match for `X` | -**Via the Rust API**: `store.query().documents(|doc| ...)` lets you filter on *any* zone-map field yourself (`heading_slugs`, `frontmatter_keys`, `title`, `tags`, …), not just the patterns `SqlEngine` recognizes automatically. - ## Layer 2: Interval Index (section hierarchy) Heading hierarchy is encoded as `(pre, post)` pairs via Pre-Post Order (Nested Set) traversal: diff --git a/docs/books/src/reference/sql-ddl.md b/docs/books/src/reference/sql-ddl.md index 1689812..18ebbd0 100644 --- a/docs/books/src/reference/sql-ddl.md +++ b/docs/books/src/reference/sql-ddl.md @@ -1,12 +1,14 @@ # DDL Statements -`mq-db` supports a small set of DDL statements for defining **custom in-memory tables** alongside the built-in `documents`/`blocks` virtual tables. Custom tables live only for the process lifetime; they are not persisted to the `.mq-db` store file. +`mq-db` supports a small set of DDL statements for defining **custom in-memory tables** alongside the built-in `documents`/`blocks` virtual tables. Custom tables are saved in the `.mq-db` store file, so they are there the next time it is opened. | Statement | Description | | --- | --- | | `CREATE TABLE name AS SELECT …` | Create a custom table from a query result | | `CREATE TABLE name (col TYPE, …)` | Create an empty custom table with explicit schema | | `INSERT INTO name VALUES (…)` | Insert a row into a custom table | +| `UPDATE name SET col = expr, … [WHERE …]` | Change rows of a custom table. Every expression sees the row as it was before the statement, so `SET a = b, b = a` swaps | +| `DELETE FROM name [WHERE …]` | Delete rows of a custom table | | `DROP TABLE name` | Drop a custom table | | `SHOW TABLES` | List all custom tables and views | | `DESC name` | Show the schema of a custom table or view | @@ -16,6 +18,10 @@ | `ATTACH DATABASE 'path' AS alias` | Mount another `.mq-db` store as `alias.` | | `DETACH alias` | Unmount a previously attached store | +`UPDATE` and `DELETE` accept an alias and subqueries in `WHERE`, enforce `NOT NULL` and `UNIQUE`, and take effect atomically: a failing statement leaves the table unchanged. They roll back with `ROLLBACK`. `UPDATE ... FROM`, `JOIN`, `RETURNING`, `ORDER BY` and `LIMIT` are not supported. The same statements on `blocks` are [write-back](write-back.md) and need `--write-back`. + +Cells are stored as text. In arithmetic a cell holding a number counts as that number (`SET age = age + 1`), and the result is stored back as text. + ## Examples ```bash diff --git a/docs/books/src/reference/vector-search.md b/docs/books/src/reference/vector-search.md index 3082bfb..ff80f3e 100644 --- a/docs/books/src/reference/vector-search.md +++ b/docs/books/src/reference/vector-search.md @@ -34,8 +34,7 @@ INSERT INTO emb SELECT * FROM read_json('new-embeddings.jsonl'); Drop embeddings of blocks that no longer exist: ```sql -CREATE OR REPLACE TABLE emb AS -SELECT * FROM emb WHERE content_hash IN (SELECT content_hash FROM blocks); +DELETE FROM emb WHERE content_hash NOT IN (SELECT content_hash FROM blocks); ``` ## Searching diff --git a/docs/books/src/reference/write-back.md b/docs/books/src/reference/write-back.md index 1b6e289..35c88a5 100644 --- a/docs/books/src/reference/write-back.md +++ b/docs/books/src/reference/write-back.md @@ -1,5 +1,7 @@ # INSERT / UPDATE / DELETE with Write-Back +Statements on [custom tables](sql-ddl.md) change only the `.mq-db` file and never need the flag. The rest of this page is about `blocks`. + `INSERT`/`UPDATE`/`DELETE` on `blocks` write the change back to the document's *source Markdown file* (re-parsed in place, same `DocumentId`), pass `--write-back` to allow it; without the flag the statement is rejected: ```bash diff --git a/docs/books/src/start/cli.md b/docs/books/src/start/cli.md index cfa25fc..c18edd7 100644 --- a/docs/books/src/start/cli.md +++ b/docs/books/src/start/cli.md @@ -96,22 +96,6 @@ mq-db repl --db store.mq-db --mode sql See [REPL](repl.md) for the full command list. -## `lint` - -Run structural lint checks (currently: a heading at the given depth immediately followed by a list). - -```bash -mq-db lint --db store.mq-db --depth 2 -``` - -```text -✗ 1 violation (H2 immediately followed by list) - - file heading - ──────────────────────────────────────── ────────────────────────────── - docs/DESIGN.md "Quick Start" -``` - ## `stats` Show store-wide statistics: document/block counts, block-type distribution, code-language distribution. diff --git a/docs/books/src/start/library.md b/docs/books/src/start/library.md index 6a5afbc..349a611 100644 --- a/docs/books/src/start/library.md +++ b/docs/books/src/start/library.md @@ -8,20 +8,13 @@ mq-db = "0.1" ``` ```rust -use mq_db::{DocumentStore, SqlEngine, MqEngine, block::BlockType}; +use mq_db::{DocumentStore, SqlEngine, MqEngine}; // Build in memory let mut store = DocumentStore::new(); store.add_file("docs/DESIGN.md")?; store.add_str("# Hello\n\n## Architecture\n\nDetails\n")?; -// Chainable query API: zone-map skip + interval scope + block predicates -let chunks = store.query() - .documents(|doc| doc.zone_maps.heading_contents.contains("Architecture")) - .under_heading("Architecture", Some(2)) - .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code)) - .blocks(); - // SQL engine (custom sqlparser-based evaluator, no SQLite dependency) let engine = SqlEngine::new(&store)?; let out = engine.execute( @@ -32,9 +25,6 @@ print!("{}", out.to_table()); // mq engine let results = MqEngine::eval_store(".h1", &store)?; -// Structural lint -let violations = store.query().lint_heading_followed_by(2, &[BlockType::List]); - // UPDATE/DELETE with write-back: rewrites the affected block's source file // (heading/paragraph content only), then re-parses it in place store.execute_sql_mut( @@ -66,14 +56,3 @@ let store = DocumentStore::load_catalog_only("store.mq-db")?; | `DocumentStore::load_catalog_only()` | Catalog only | `list` / `stats`-style metadata commands | When using `open()`, call `load_all_blocks()` and `load_all_indexes()` before running any `SqlEngine` query. - -## Query builder - -`store.query()` returns a chainable builder that applies the same three index layers used by the SQL engine, in order: - -1. `.documents(|doc| ...)`: zone-map predicate, skips whole documents -2. `.under_heading(title, depth)` / interval-scope helpers: narrows to a `(pre, post)` range -3. `.filter(|block| ...)`: per-block predicate over the remaining candidates -4. `.blocks()`: materializes the final `Vec<&Block>` - -See [Index Layers](../reference/index-layers.md) for how each layer works. diff --git a/src/bin/mq-db.rs b/src/bin/mq-db.rs index 2750ad5..55982cf 100644 --- a/src/bin/mq-db.rs +++ b/src/bin/mq-db.rs @@ -187,17 +187,6 @@ enum Commands { attach: Vec, }, - /// Run structural lint checks - Lint { - /// Path to .mq-db store file - #[arg(short, long, default_value = "store.mq-db")] - db: PathBuf, - - /// Heading depth to check (H1=1 .. H6=6) - #[arg(long, default_value_t = 2)] - depth: u8, - }, - /// Show store statistics Stats { /// Path to .mq-db store file @@ -894,43 +883,6 @@ async fn main() -> anyhow::Result<()> { run_repl(store, mode, write_back)?; } - // lint - Commands::Lint { db, depth } => { - let store = load_store(&db)?; - let q = store.query(); - let violations = q.lint_heading_followed_by(depth, &[BlockType::List]); - if violations.is_empty() { - println!( - "✓ No violations (H{} must not be immediately followed by a list)", - depth - ); - } else { - let n = violations.len(); - println!( - "✗ {} violation{} (H{} immediately followed by list)\n", - n, - if n == 1 { "" } else { "s" }, - depth - ); - println!(" {:<40} heading", "file"); - println!(" {} {}", "─".repeat(40), "─".repeat(30)); - for v in &violations { - let path = v - .document - .path - .as_ref() - .map(|p| p.to_string_lossy().to_string()) - .unwrap_or_else(|| format!("", v.document.id)); - let path_display = if path.len() > 40 { - format!("…{}", &path[path.len() - 39..]) - } else { - path - }; - println!(" {:<40} \"{}\"", path_display, v.heading.content); - } - } - } - // stats Commands::Stats { db } => { let store = load_store(&db)?; @@ -1473,27 +1425,9 @@ fn md_block_to_html(s: &str) -> String { // REPL -/// Crude prefix check used to gate write-back before even attempting a -/// parse. Only `INSERT INTO blocks` requires `--write-back`; custom tables -/// don't touch Markdown files. +/// Whether `sql` writes to Markdown files and so needs `--write-back`. fn is_write_statement(sql: &str) -> bool { - let trimmed = sql.trim().trim_end_matches(';'); - let upper = trimmed.to_ascii_uppercase(); - upper.starts_with("UPDATE ") || upper.starts_with("DELETE ") || is_insert_into_blocks(&upper) -} - -fn is_insert_into_blocks(upper: &str) -> bool { - let Some(after_into) = upper.strip_prefix("INSERT INTO") else { - return false; - }; - let table = after_into - .trim_start() - .trim_start_matches(['"', '`']) - .trim_start(); - table == "BLOCKS" - || ["BLOCKS ", "BLOCKS(", "BLOCKS\"", "BLOCKS`"] - .iter() - .any(|p| table.starts_with(p)) + mq_db::sql::statement_needs_write_back(sql) } fn run_repl( diff --git a/src/lib.rs b/src/lib.rs index c486492..6587b81 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -16,38 +16,20 @@ //! slugs, code languages, front-matter keys/tags) that let the query //! engine skip irrelevant documents before scanning their blocks. //! -//! * **Chainable query API** – document-level and block-level predicates, -//! `UNDER heading` section scoping, built-in linter helpers. -//! //! ## Quick Start //! //! ```rust -//! use mq_db::{DocumentStore, block::BlockType}; -//! -//! let mut store = DocumentStore::new(); -//! store.add_str("# Hello\n\n## Architecture\n\nDetails\n\n```rust\ncode\n```\n").unwrap(); -//! -//! // Extract all content under the "Architecture" H2 -//! let chunks = store.query() -//! .under_heading("Architecture", Some(2)) -//! .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code)) -//! .blocks(); -//! -//! assert_eq!(chunks.len(), 2); -//! ``` -//! -//! ## Structural Linting -//! -//! ```rust -//! use mq_db::{DocumentStore, block::BlockType}; +//! use mq_db::{DocumentStore, SqlEngine}; //! //! let mut store = DocumentStore::new(); -//! store.add_str("## Section\n\n- item without intro paragraph\n").unwrap(); +//! store.add_str("# Hello\n\n## Architecture\n\nDetails\n").unwrap(); //! -//! let q = store.query(); -//! let violations = q.lint_heading_followed_by(2, &[BlockType::List]); +//! let engine = SqlEngine::new(&store).unwrap(); +//! let out = engine +//! .execute("SELECT content FROM blocks WHERE block_type = 'heading' ORDER BY pre") +//! .unwrap(); //! -//! assert_eq!(violations.len(), 1); +//! assert_eq!(out.rows.len(), 2); //! ``` pub mod block; @@ -57,7 +39,6 @@ pub mod error; pub mod index; pub mod indexes; pub mod mq_engine; -pub mod query; pub mod search; pub mod sql; pub mod storage; @@ -67,7 +48,6 @@ pub mod tui; pub use document::Document; pub use error::MqdbError; pub use mq_engine::MqEngine; -pub use query::{LintViolation, Query, QueryResult}; pub use sql::{QueryOutput, SqlEngine}; pub use storage::Storage; pub use storage::catalog::CatalogEntry; diff --git a/src/query.rs b/src/query.rs deleted file mode 100644 index 8f66a9d..0000000 --- a/src/query.rs +++ /dev/null @@ -1,468 +0,0 @@ -use crate::{ - block::{Block, BlockType}, - document::Document, - store::DocumentStore, -}; - -enum SectionAnchor { - /// Directly supply a known (pre, post) interval. - Interval { pre: u32, post: u32 }, - /// Find the first heading matching the given content and optional depth. - Heading { content: String, depth: Option }, -} - -/// A query result pairing a matched [`Block`] with its parent [`Document`]. -pub struct QueryResult<'a> { - pub block: &'a Block, - pub document: &'a Document, -} - -/// Chainable, lazy query builder over a [`DocumentStore`]. -/// -/// Filters are applied in evaluation order (cheapest first): -/// 1. Document-level zone-map skip (applied per document before scanning blocks) -/// 2. Section `UNDER` constraint (interval check) -/// 3. Block-level predicates -/// -/// # Example – RAG chunk extraction -/// -/// ```rust -/// use mq_db::{DocumentStore, block::BlockType}; -/// -/// let mut store = DocumentStore::new(); -/// store.add_str("# Doc\n\n## Architecture\n\nExplanation\n\n```rust\ncode\n```\n").unwrap(); -/// -/// let results = store.query() -/// .under_heading("Architecture", Some(2)) -/// .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code)) -/// .blocks(); -/// -/// assert_eq!(results.len(), 2); -/// ``` -type DocPredicate<'store> = Box bool + 'store>; -type BlockPredicate<'store> = Box bool + 'store>; - -pub struct Query<'store> { - store: &'store DocumentStore, - doc_predicate: Option>, - block_predicates: Vec>, - anchor: Option, - limit: Option, -} - -impl<'store> Query<'store> { - pub(crate) fn new(store: &'store DocumentStore) -> Self { - Self { - store, - doc_predicate: None, - block_predicates: Vec::new(), - anchor: None, - limit: None, - } - } - - /// Skip documents for which `predicate` returns `false`. - /// - /// Use this to leverage zone-map statistics before scanning blocks: - /// ```rust - /// # use mq_db::DocumentStore; - /// # let store = DocumentStore::new(); - /// store.query() - /// .documents(|doc| doc.zone_maps.code_languages.contains("python")); - /// ``` - pub fn documents(mut self, f: F) -> Self - where - F: Fn(&Document) -> bool + 'store, - { - self.doc_predicate = Some(Box::new(f)); - self - } - - /// Restrict results to blocks that fall within the heading section - /// identified by `content` and optional `depth`. - /// - /// Equivalent to the SQL `WHERE b UNDER (SELECT id FROM blocks WHERE ...)`. - /// - /// For best performance, chain a `.documents(|d| d.zone_maps.heading_contents.contains("..."))` - /// filter before this to skip irrelevant documents via zone maps. - pub fn under_heading(mut self, content: impl Into, depth: Option) -> Self { - self.anchor = Some(SectionAnchor::Heading { - content: content.into(), - depth, - }); - self - } - - /// Restrict results to blocks that fall within the interval `(pre, post)`. - /// - /// Use this when you already know the ancestor block's interval values. - pub fn under_interval(mut self, pre: u32, post: u32) -> Self { - self.anchor = Some(SectionAnchor::Interval { pre, post }); - self - } - - /// Keep only blocks for which `predicate` returns `true`. - pub fn filter(mut self, f: F) -> Self - where - F: Fn(&Block) -> bool + 'store, - { - self.block_predicates.push(Box::new(f)); - self - } - - /// Keep only blocks with the given [`BlockType`]. - pub fn block_type(self, ty: BlockType) -> Self { - self.filter(move |b| b.block_type == ty) - } - - /// Keep only heading blocks at the given depth. - pub fn heading_depth(self, depth: u8) -> Self { - self.filter(move |b| b.block_type == BlockType::Heading && b.heading_depth() == Some(depth)) - } - - /// Keep only code blocks with the given language tag. - pub fn code_lang(self, lang: impl Into) -> Self { - let lang = lang.into(); - self.filter(move |b| b.code_lang() == Some(lang.as_str())) - } - - /// Keep only blocks whose content contains `substring` (case-sensitive). - pub fn content_contains(self, substring: impl Into) -> Self { - let s = substring.into(); - self.filter(move |b| b.content.contains(s.as_str())) - } - - /// Keep only blocks whose content matches `pattern` (case-insensitive). - pub fn content_contains_ci(self, pattern: impl Into) -> Self { - let pat = pattern.into().to_lowercase(); - self.filter(move |b| b.content.to_lowercase().contains(pat.as_str())) - } - - /// Stop collecting after `n` results. - pub fn limit(mut self, n: usize) -> Self { - self.limit = Some(n); - self - } - - /// Execute the query, returning matched (block, document) pairs in - /// document order. - pub fn collect(&self) -> Vec> { - let mut results: Vec> = Vec::new(); - - 'doc: for doc in self.store.documents() { - // Zone-map skip - if let Some(dp) = &self.doc_predicate - && !dp(doc) - { - continue 'doc; - } - - // Resolve the section anchor for this document - let interval: Option<(u32, u32)> = match &self.anchor { - None => None, - Some(SectionAnchor::Interval { pre, post }) => Some((*pre, *post)), - Some(SectionAnchor::Heading { content, depth }) => doc - .blocks - .iter() - .find(|b| { - b.block_type == BlockType::Heading - && b.content == *content - && depth.is_none_or(|d| b.heading_depth() == Some(d)) - }) - .map(|h| (h.pre, h.post)), - }; - - for block in &doc.blocks { - // UNDER interval check - if let Some((anc_pre, anc_post)) = interval - && !block.is_under_interval(anc_pre, anc_post) - { - continue; - } - - // Block predicates - if self.block_predicates.iter().all(|f| f(block)) { - results.push(QueryResult { - block, - document: doc, - }); - - if let Some(limit) = self.limit - && results.len() >= limit - { - return results; - } - } - } - } - - results - } - - /// Like [`collect`], but returns cloned blocks (discarding document context). - /// - /// Returns owned `Vec` so it can be used on temporaries: - /// ```rust - /// # use mq_db::{DocumentStore, block::BlockType}; - /// # let mut store = DocumentStore::new(); - /// # store.add_str("# Hello\n").unwrap(); - /// let blocks = store.query().heading_depth(1).blocks(); - /// assert_eq!(blocks.len(), 1); - /// ``` - pub fn blocks(&self) -> Vec { - self.collect() - .into_iter() - .map(|r| r.block.clone()) - .collect() - } - - /// Returns the number of matching blocks without materialising them. - pub fn count(&self) -> usize { - self.collect().len() - } - - /// Find all (heading, next_sibling) pairs where the heading matches the - /// given depth and the immediately following sibling has one of the - /// `forbidden_types`. - /// - /// This is the foundation for structural lint rules such as: - /// > "An H2 heading must not be immediately followed by a list." - /// - /// # Example - /// - /// ```rust - /// use mq_db::{DocumentStore, block::BlockType}; - /// - /// let mut store = DocumentStore::new(); - /// store.add_str("## Section\n\n- item\n").unwrap(); - /// - /// let q = store.query(); - /// let violations = q.lint_heading_followed_by(2, &[BlockType::List]); - /// - /// assert_eq!(violations.len(), 1); - /// ``` - pub fn lint_heading_followed_by( - &self, - heading_depth: u8, - forbidden_types: &[BlockType], - ) -> Vec> { - let mut violations = Vec::new(); - - for doc in self.store.documents() { - if let Some(dp) = &self.doc_predicate - && !dp(doc) - { - continue; - } - - for block in &doc.blocks { - if block.block_type != BlockType::Heading { - continue; - } - if block.heading_depth() != Some(heading_depth) { - continue; - } - - if let Some(next) = doc.first_child(block) - && forbidden_types.contains(&next.block_type) - { - violations.push(LintViolation { - heading: block, - offending: next, - document: doc, - }); - } - } - } - - violations - } -} - -/// A structural lint violation: a heading followed by a forbidden block type. -pub struct LintViolation<'a> { - pub heading: &'a Block, - pub offending: &'a Block, - pub document: &'a Document, -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::DocumentStore; - use rstest::rstest; - - fn store_with(sources: &[&str]) -> DocumentStore { - let mut s = DocumentStore::new(); - for src in sources { - s.add_str(src).unwrap(); - } - s - } - - #[test] - fn test_block_type_filter() { - let store = store_with(&["# H1\n\n## H2\n\nParagraph\n\n```rust\ncode\n```\n"]); - let q = store.query().block_type(BlockType::Heading); - let headings = q.blocks(); - assert_eq!(headings.len(), 2); - } - - #[test] - fn test_heading_depth_filter() { - let store = store_with(&["# H1\n\n## H2\n\n### H3\n"]); - let q = store.query().heading_depth(2); - let h2s = q.blocks(); - assert_eq!(h2s.len(), 1); - assert_eq!(h2s[0].content, "H2"); - } - - #[test] - fn test_under_heading_filter() { - let store = store_with(&[ - "# Doc\n\n## Architecture\n\nExplanation\n\n```rust\ncode\n```\n\n## Other\n\nOther para\n", - ]); - - let q = store - .query() - .under_heading("Architecture", Some(2)) - .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code)); - let results = q.blocks(); - - // Should contain "Explanation" and the code block, but NOT "Other para" - assert_eq!(results.len(), 2); - assert!(results.iter().any(|b| b.content.contains("Explanation"))); - assert!(results.iter().any(|b| b.block_type == BlockType::Code)); - assert!(!results.iter().any(|b| b.content.contains("Other para"))); - } - - #[test] - fn test_limit() { - let store = store_with(&["# A\n\n# B\n\n# C\n\n# D\n"]); - let q = store.query().heading_depth(1).limit(2); - let results = q.blocks(); - assert_eq!(results.len(), 2); - } - - #[test] - fn test_content_contains() { - let store = store_with(&["# Hello World\n\n## Goodbye\n"]); - let q = store.query().content_contains("World"); - let results = q.blocks(); - assert_eq!(results.len(), 1); - assert_eq!(results[0].content, "Hello World"); - } - - #[test] - fn test_code_lang_filter() { - let store = store_with(&["```rust\nfn x(){}\n```\n\n```python\nx=1\n```\n"]); - let q = store.query().code_lang("rust"); - let results = q.blocks(); - assert_eq!(results.len(), 1); - assert_eq!(results[0].code_lang(), Some("rust")); - } - - #[test] - fn test_zone_map_document_skip() { - let store = store_with(&["```python\nx=1\n```\n", "```rust\nfn x(){}\n```\n"]); - - let q = store - .query() - .documents(|doc| doc.zone_maps.code_languages.contains("rust")) - .block_type(BlockType::Code); - let results = q.blocks(); - - assert_eq!(results.len(), 1); - assert_eq!(results[0].code_lang(), Some("rust")); - } - - #[test] - fn test_lint_heading_followed_by_list() { - let store = store_with(&[ - "## Good\n\nParagraph intro\n\n- item\n", - "## Bad\n\n- item without intro\n", - ]); - - let q = store.query(); - let violations = q.lint_heading_followed_by(2, &[BlockType::List]); - // Only the second doc has a violation - assert_eq!(violations.len(), 1); - assert_eq!(violations[0].heading.content, "Bad"); - assert_eq!(violations[0].offending.block_type, BlockType::List); - } - - #[test] - fn test_multi_document_query() { - let store = store_with(&[ - "# Doc1\n\n```rust\nfn a(){}\n```\n", - "# Doc2\n\n```python\nx=1\n```\n", - "# Doc3\n\n```rust\nfn b(){}\n```\n", - ]); - - let q = store.query().code_lang("rust"); - let results = q.blocks(); - assert_eq!(results.len(), 2); - } - - #[rstest] - #[case(BlockType::Heading, 3)] - #[case(BlockType::Paragraph, 1)] - #[case(BlockType::Code, 1)] - #[case(BlockType::List, 1)] - fn test_block_type_filter_count_param(#[case] block_type: BlockType, #[case] expected: usize) { - let store = - store_with(&["# H1\n\n## H2\n\n### H3\n\nParagraph\n\n```rust\ncode\n```\n\n- item\n"]); - assert_eq!( - store.query().block_type(block_type).blocks().len(), - expected - ); - } - - #[rstest] - #[case(1, 1, "H1")] - #[case(2, 1, "H2")] - #[case(3, 1, "H3")] - fn test_heading_depth_count_param( - #[case] depth: u8, - #[case] expected: usize, - #[case] content: &str, - ) { - let store = store_with(&["# H1\n\n## H2\n\n### H3\n"]); - let results = store.query().heading_depth(depth).blocks(); - assert_eq!(results.len(), expected); - assert_eq!(results[0].content, content); - } - - #[rstest] - #[case("Hello", 1)] - #[case("World", 1)] - #[case("Goodbye", 1)] - #[case("nonexistent_xyz", 0)] - fn test_content_contains_count_param(#[case] needle: &str, #[case] expected: usize) { - let store = store_with(&["# Hello World\n\n## Goodbye\n"]); - assert_eq!( - store.query().content_contains(needle).blocks().len(), - expected - ); - } - - #[rstest] - #[case("rust", 1)] - #[case("python", 1)] - #[case("go", 0)] - fn test_code_lang_count_param(#[case] lang: &str, #[case] expected: usize) { - let store = store_with(&["```rust\nfn x(){}\n```\n\n```python\nx=1\n```\n"]); - assert_eq!(store.query().code_lang(lang).blocks().len(), expected); - } - - #[rstest] - #[case(1, 1)] - #[case(2, 2)] - #[case(3, 3)] - #[case(100, 4)] - fn test_limit_count_param(#[case] limit: usize, #[case] expected: usize) { - let store = store_with(&["# A\n\n# B\n\n# C\n\n# D\n"]); - let results = store.query().heading_depth(1).limit(limit).blocks(); - assert_eq!(results.len(), expected); - } -} diff --git a/src/sql.rs b/src/sql.rs index bd9ba31..ee93d98 100644 --- a/src/sql.rs +++ b/src/sql.rs @@ -59,6 +59,7 @@ mod aggregate; mod analysis; mod datetime; mod ddl; +mod dml; mod engine; mod explain; mod expr; @@ -82,3 +83,4 @@ mod write_back; pub use engine::SqlEngine; pub use output::{QueryOutput, html_escape}; pub use value::Value; +pub use write_back::statement_needs_write_back; diff --git a/src/sql/dml.rs b/src/sql/dml.rs new file mode 100644 index 0000000..b936848 --- /dev/null +++ b/src/sql/dml.rs @@ -0,0 +1,242 @@ +//! `UPDATE` and `DELETE` on custom tables. + +use super::engine::SqlEngine; +use super::expr::eval_expr; +use super::output::QueryOutput; +use super::rows::{ident_value, qualify_row, require_unqualified}; +use super::value::{Row, Value}; +use crate::MqdbError; +use crate::store::CustomTableState; +use sqlparser::ast::{ + Assignment, AssignmentTarget, Delete, Expr, FromTable, TableFactor, TableWithJoins, Update, +}; + +const BUILT_IN: [&str; 3] = ["blocks", "documents", "links"]; + +/// The table an `UPDATE`/`DELETE` names, and the prefix its columns carry in +/// expressions (the alias if there is one). +fn target(twj: &TableWithJoins) -> Result<(String, String), MqdbError> { + if !twj.joins.is_empty() { + return Err(MqdbError::SqlExec( + "UPDATE/DELETE does not support JOIN".into(), + )); + } + let TableFactor::Table { name, alias, .. } = &twj.relation else { + return Err(MqdbError::SqlExec( + "unsupported UPDATE/DELETE target".into(), + )); + }; + let table = require_unqualified(name)?; + if BUILT_IN.contains(&table.as_str()) { + return Err(MqdbError::SqlExec(format!( + "cannot modify built-in table '{table}' with plain SQL; \ + UPDATE/DELETE on blocks needs write-back mode" + ))); + } + let prefix = alias + .as_ref() + .map_or_else(|| table.clone(), |a| a.name.value.clone()); + Ok((table, prefix)) +} + +fn row_of(columns: &[String], cells: &[String], prefix: &str) -> Row { + qualify_row( + Row { + columns: columns.to_vec(), + values: cells.iter().map(|c| Value::Str(c.clone())).collect(), + }, + prefix, + ) +} + +impl SqlEngine<'_> { + pub(super) fn exec_update(&self, update: &Update) -> Result { + if update.from.is_some() + || update.returning.is_some() + || !update.order_by.is_empty() + || update.limit.is_some() + { + return Err(MqdbError::SqlExec( + "UPDATE supports only SET and WHERE".into(), + )); + } + let (table, prefix) = target(&update.table)?; + let columns = self.table_columns(&table)?; + let assignments = resolve_assignments(&table, &columns, &update.assignments)?; + let n = self.rewrite_rows(&table, &prefix, update.selection.as_ref(), |row, cells| { + for (idx, expr) in &assignments { + cells[*idx] = eval_expr(expr, row).display(); + } + true + })?; + Ok(count_result("updated", n)) + } + + pub(super) fn exec_delete(&self, delete: &Delete) -> Result { + if !delete.tables.is_empty() + || delete.using.is_some() + || delete.returning.is_some() + || !delete.order_by.is_empty() + || delete.limit.is_some() + { + return Err(MqdbError::SqlExec( + "DELETE supports only FROM and WHERE".into(), + )); + } + let (FromTable::WithFromKeyword(tables) | FromTable::WithoutKeyword(tables)) = &delete.from; + let [twj] = tables.as_slice() else { + return Err(MqdbError::SqlExec( + "DELETE supports exactly one target table".into(), + )); + }; + let (table, prefix) = target(twj)?; + self.table_columns(&table)?; + let n = self.rewrite_rows(&table, &prefix, delete.selection.as_ref(), |_, _| false)?; + Ok(count_result("deleted", n)) + } + + fn table_columns(&self, table: &str) -> Result, MqdbError> { + self.store + .custom_tables + .read() + .unwrap() + .get(table) + .map(|s| s.columns.clone()) + .ok_or_else(|| MqdbError::SqlExec(format!("unknown table: {table}"))) + } + + /// Applies `edit` to every row matching `selection` and persists the + /// result; returns how many rows matched. `edit` mutates the row's cells + /// and returns whether to keep it. Predicates and new values are all + /// evaluated against the table as it was before the statement. + fn rewrite_rows( + &self, + table: &str, + prefix: &str, + selection: Option<&Expr>, + edit: impl Fn(&Row, &mut Vec) -> bool, + ) -> Result { + // Serialize mutate+flush+rollback against other catalog writers. + let _catalog_guard = self.store.catalog_commit.lock().unwrap(); + + let (columns, old_rows) = { + let guard = self.store.custom_tables.read().unwrap(); + let state = guard + .get(table) + .ok_or_else(|| MqdbError::SqlExec(format!("unknown table: {table}")))?; + (state.columns.clone(), state.rows.clone()) + }; + + let mut matched = 0; + let mut new_rows = Vec::with_capacity(old_rows.len()); + for cells in &old_rows { + let row = row_of(&columns, cells, prefix); + let hit = match selection { + None => true, + Some(expr) => { + let resolved = self.resolve_subqueries(expr, Some(&row))?; + eval_expr(&resolved, &row).is_truthy() + } + }; + if !hit { + new_rows.push(cells.clone()); + continue; + } + matched += 1; + let mut updated = cells.clone(); + if edit(&row, &mut updated) { + new_rows.push(updated); + } + } + if matched == 0 { + return Ok(0); + } + + let previous = { + let mut guard = self.store.custom_tables.write().unwrap(); + let state = guard + .get_mut(table) + .ok_or_else(|| MqdbError::SqlExec(format!("unknown table: {table}")))?; + check_constraints(table, state, &new_rows)?; + let previous = ( + state.rows.clone(), + state.first_row_page, + state.last_row_page, + ); + state.rows = new_rows; + previous + }; // write lock released before flush + if let Err(err) = self.store.append_table_rows_to_storage(table, &[]) { + let mut guard = self.store.custom_tables.write().unwrap(); + if let Some(state) = guard.get_mut(table) { + (state.rows, state.first_row_page, state.last_row_page) = previous; + } + return Err(err); + } + self.store.note_catalog_mutation(self.session_id); + Ok(matched) + } +} + +fn resolve_assignments( + table: &str, + columns: &[String], + assignments: &[Assignment], +) -> Result, MqdbError> { + assignments + .iter() + .map(|a| { + let AssignmentTarget::ColumnName(name) = &a.target else { + return Err(MqdbError::SqlExec( + "UPDATE does not support tuple assignment targets".into(), + )); + }; + let col = name.0.last().map(ident_value).unwrap_or(""); + let idx = columns + .iter() + .position(|c| c.eq_ignore_ascii_case(col)) + .ok_or_else(|| MqdbError::SqlExec(format!("unknown column '{col}' in {table}")))?; + Ok((idx, a.value.clone())) + }) + .collect() +} + +/// Mirrors the checks `INSERT` applies, over the whole resulting table. +fn check_constraints( + table: &str, + state: &CustomTableState, + rows: &[Vec], +) -> Result<(), MqdbError> { + for row in rows { + for &idx in &state.not_null { + if row.get(idx).is_none_or(|v| v.is_empty() || v == "NULL") { + return Err(MqdbError::SqlExec(format!( + "NOT NULL constraint failed: {}.{}", + table, state.columns[idx] + ))); + } + } + } + for group in &state.unique { + let mut seen = std::collections::HashSet::new(); + for row in rows { + let key: Vec<&String> = group.iter().map(|&i| &row[i]).collect(); + if !seen.insert(key) { + let cols: Vec<&str> = group.iter().map(|&i| state.columns[i].as_str()).collect(); + return Err(MqdbError::SqlExec(format!( + "UNIQUE constraint failed: {}.{}", + table, + cols.join(", ") + ))); + } + } + } + Ok(()) +} + +fn count_result(label: &str, n: usize) -> QueryOutput { + QueryOutput { + columns: vec![label.to_string()], + rows: vec![vec![n.to_string()]], + } +} diff --git a/src/sql/engine.rs b/src/sql/engine.rs index 9268166..9ed3e02 100644 --- a/src/sql/engine.rs +++ b/src/sql/engine.rs @@ -132,6 +132,8 @@ impl<'a> SqlEngine<'a> { Statement::Query(q) => self.exec_query(&q), Statement::CreateTable(ct) => self.exec_create_table(&ct), Statement::Insert(ins) => self.exec_insert(&ins), + Statement::Update(update) => self.exec_update(&update), + Statement::Delete(delete) => self.exec_delete(&delete), Statement::Drop { object_type: ObjectType::Table, names, @@ -178,7 +180,7 @@ impl<'a> SqlEngine<'a> { Ok(ok_result()) } _ => Err(MqdbError::SqlExec( - "unsupported statement; supported: SELECT, CREATE TABLE, INSERT INTO, DROP TABLE, CREATE VIEW, DROP VIEW, ATTACH DATABASE, DETACH, DESC, SHOW TABLES, EXPLAIN, BEGIN, COMMIT, ROLLBACK".into(), + "unsupported statement; supported: SELECT, CREATE TABLE, INSERT INTO, UPDATE, DELETE, DROP TABLE, CREATE VIEW, DROP VIEW, ATTACH DATABASE, DETACH, DESC, SHOW TABLES, EXPLAIN, BEGIN, COMMIT, ROLLBACK".into(), )), } } diff --git a/src/sql/expr.rs b/src/sql/expr.rs index 18a2bcf..6c56972 100644 --- a/src/sql/expr.rs +++ b/src/sql/expr.rs @@ -288,7 +288,10 @@ fn eval_binary(left: &Expr, op: &BinaryOperator, right: &Expr, row: &Row) -> Val BinaryOperator::Minus => arith_op(left, right, row, |a, b| a - b, |a, b| a - b), BinaryOperator::Multiply => arith_op(left, right, row, |a, b| a * b, |a, b| a * b), BinaryOperator::Divide => { - let (l, r) = (eval_expr(left, row), eval_expr(right, row)); + let (l, r) = ( + numeric(eval_expr(left, row)), + numeric(eval_expr(right, row)), + ); match (&l, &r) { (Value::Int(a), Value::Int(b)) if *b != 0 => Value::Int(a / b), _ => match (l.as_f64(), r.as_f64()) { @@ -315,6 +318,23 @@ fn cmp_op(l: &Expr, r: &Expr, row: &Row, f: impl Fn(std::cmp::Ordering) -> bool) ) } +/// A string holding a number, such as a custom table cell, as that number. +/// Anything else is returned unchanged. +fn numeric(v: Value) -> Value { + match &v { + Value::Str(s) => { + if let Ok(n) = s.trim().parse::() { + Value::Int(n) + } else if let Some(f) = s.trim().parse::().ok().filter(|f| f.is_finite()) { + Value::Float(f) + } else { + v + } + } + _ => v, + } +} + fn arith_op( l: &Expr, r: &Expr, @@ -322,7 +342,7 @@ fn arith_op( int_f: impl Fn(i64, i64) -> i64, flt_f: impl Fn(f64, f64) -> f64, ) -> Value { - let (lv, rv) = (eval_expr(l, row), eval_expr(r, row)); + let (lv, rv) = (numeric(eval_expr(l, row)), numeric(eval_expr(r, row))); match (&lv, &rv) { (Value::Int(a), Value::Int(b)) => Value::Int(int_f(*a, *b)), _ => match (lv.as_f64(), rv.as_f64()) { diff --git a/src/sql/tests.rs b/src/sql/tests.rs index 99a3a35..620f53a 100644 --- a/src/sql/tests.rs +++ b/src/sql/tests.rs @@ -3509,3 +3509,158 @@ fn test_sql_order_by_mixed_integer_and_numeric_string_keys() { let names: Vec<&str> = out.rows.iter().map(|r| r[0].as_str()).collect(); assert_eq!(names, vec!["second", "third", "first"]); } + +fn people_engine_store() -> DocumentStore { + let store = DocumentStore::new(); + let engine = SqlEngine::new(&store).unwrap(); + engine + .execute("CREATE TABLE p (id INTEGER PRIMARY KEY, name TEXT NOT NULL, age INTEGER)") + .unwrap(); + engine + .execute("INSERT INTO p VALUES (1, 'ann', 30), (2, 'bob', 25), (3, 'cy', 41)") + .unwrap(); + store +} + +fn names(store: &DocumentStore) -> Vec { + SqlEngine::new(store) + .unwrap() + .execute("SELECT name FROM p ORDER BY id") + .unwrap() + .rows + .into_iter() + .map(|r| r[0].clone()) + .collect() +} + +#[test] +fn test_sql_update_custom_table() { + let store = people_engine_store(); + let engine = SqlEngine::new(&store).unwrap(); + let out = engine + .execute("UPDATE p SET name = 'ANN', age = age + 1 WHERE id = 1") + .unwrap(); + assert_eq!(out.rows, vec![vec!["1".to_string()]]); + let out = engine + .execute("SELECT name, age FROM p WHERE id = 1") + .unwrap(); + assert_eq!(out.rows, vec![vec!["ANN".to_string(), "31".to_string()]]); + assert_eq!(names(&store), vec!["ANN", "bob", "cy"]); +} + +#[test] +fn test_sql_update_all_rows_and_swap_uses_old_values() { + let store = people_engine_store(); + let engine = SqlEngine::new(&store).unwrap(); + engine.execute("CREATE TABLE s (a TEXT, b TEXT)").unwrap(); + engine.execute("INSERT INTO s VALUES ('x', 'y')").unwrap(); + engine.execute("UPDATE s SET a = b, b = a").unwrap(); + let out = engine.execute("SELECT a, b FROM s").unwrap(); + assert_eq!(out.rows, vec![vec!["y".to_string(), "x".to_string()]]); +} + +#[test] +fn test_sql_delete_custom_table() { + let store = people_engine_store(); + let engine = SqlEngine::new(&store).unwrap(); + let out = engine.execute("DELETE FROM p WHERE age > 28").unwrap(); + assert_eq!(out.rows, vec![vec!["2".to_string()]]); + assert_eq!(names(&store), vec!["bob"]); + let out = engine.execute("DELETE FROM p").unwrap(); + assert_eq!(out.rows, vec![vec!["1".to_string()]]); + assert!(names(&store).is_empty()); +} + +#[test] +fn test_sql_delete_with_subquery_and_alias() { + let mut store = people_engine_store(); + store.add_str("# ann\n\nx\n").unwrap(); + let engine = SqlEngine::new(&store).unwrap(); + let out = engine + .execute( + "DELETE FROM p AS q WHERE q.name IN (SELECT content FROM blocks WHERE block_type = 'heading')", + ) + .unwrap(); + assert_eq!(out.rows, vec![vec!["1".to_string()]]); + assert_eq!(names(&store), vec!["bob", "cy"]); +} + +#[test] +fn test_sql_update_enforces_constraints_and_leaves_table_unchanged() { + let store = people_engine_store(); + let engine = SqlEngine::new(&store).unwrap(); + let err = engine + .execute("UPDATE p SET name = '' WHERE id = 2") + .unwrap_err() + .to_string(); + assert!(err.contains("NOT NULL"), "{err}"); + let err = engine + .execute("UPDATE p SET id = 1 WHERE id = 3") + .unwrap_err() + .to_string(); + assert!(err.contains("UNIQUE"), "{err}"); + assert_eq!(names(&store), vec!["ann", "bob", "cy"]); +} + +#[test] +fn test_sql_update_unknown_column_and_table() { + let store = people_engine_store(); + let engine = SqlEngine::new(&store).unwrap(); + assert!(engine.execute("UPDATE p SET nope = 1").is_err()); + assert!(engine.execute("UPDATE nope SET a = 1").is_err()); + assert!(engine.execute("DELETE FROM nope").is_err()); +} + +#[test] +fn test_sql_update_delete_reject_built_in_tables_without_write_back() { + let mut store = DocumentStore::new(); + store.add_str("# A\n\nx\n").unwrap(); + let engine = SqlEngine::new(&store).unwrap(); + for sql in [ + "UPDATE blocks SET content = 'y'", + "DELETE FROM blocks", + "DELETE FROM documents", + ] { + let err = engine.execute(sql).unwrap_err().to_string(); + assert!(err.contains("built-in"), "{sql}: {err}"); + } +} + +#[test] +fn test_sql_update_delete_roll_back_with_transaction() { + let store = people_engine_store(); + let engine = SqlEngine::new(&store).unwrap(); + engine.execute("BEGIN").unwrap(); + engine.execute("DELETE FROM p WHERE id = 1").unwrap(); + engine + .execute("UPDATE p SET name = 'zz' WHERE id = 2") + .unwrap(); + assert_eq!(names(&store), vec!["zz", "cy"]); + engine.execute("ROLLBACK").unwrap(); + assert_eq!(names(&store), vec!["ann", "bob", "cy"]); +} + +#[rstest::rstest] +#[case("UPDATE blocks SET content = 'x'", true)] +#[case("DELETE FROM blocks WHERE id = 1", true)] +#[case("INSERT INTO blocks VALUES (1)", true)] +#[case("UPDATE p SET name = 'x'", false)] +#[case("DELETE FROM p", false)] +#[case("INSERT INTO p VALUES (1)", false)] +#[case("SELECT * FROM blocks", false)] +#[case("UPDATE", false)] +fn test_statement_needs_write_back(#[case] sql: &str, #[case] expected: bool) { + assert_eq!(super::statement_needs_write_back(sql), expected, "{sql}"); +} + +#[rstest::rstest] +#[case("SELECT '30' + 1", "31")] +#[case("SELECT '1.5' * 2", "3")] +#[case("SELECT 7 / '2'", "3")] +#[case("SELECT 'abc' + 1", "NULL")] +#[case("SELECT '' + 1", "NULL")] +fn test_sql_arithmetic_on_numeric_text(#[case] sql: &str, #[case] expected: &str) { + let store = DocumentStore::new(); + let engine = SqlEngine::new(&store).unwrap(); + assert_eq!(engine.execute(sql).unwrap().rows[0][0], expected, "{sql}"); +} diff --git a/src/sql/write_back.rs b/src/sql/write_back.rs index adc42b7..57c973a 100644 --- a/src/sql/write_back.rs +++ b/src/sql/write_back.rs @@ -42,6 +42,35 @@ fn single_table_name(twj: &TableWithJoins) -> Result { } } +/// Whether `sql` would modify Markdown files, i.e. `UPDATE`/`DELETE`/`INSERT` +/// targeting `blocks`. Statements on custom tables do not. A target that is +/// not a plain table name counts as needing write-back, the safe default; SQL +/// that does not parse needs nothing, since executing it only reports the +/// parse error. +pub fn statement_needs_write_back(sql: &str) -> bool { + let Ok(stmts) = Parser::parse_sql(&GenericDialect {}, sql) else { + return false; + }; + let is_blocks = |twj: &TableWithJoins| single_table_name(twj).is_ok_and(|n| n == "blocks"); + match stmts.first() { + Some(Statement::Update(update)) => { + single_table_name(&update.table).map_or(true, |n| n == "blocks") + } + Some(Statement::Delete(delete)) => { + let (FromTable::WithFromKeyword(tables) | FromTable::WithoutKeyword(tables)) = + &delete.from; + tables.len() != 1 || tables.iter().any(is_blocks) || !delete.tables.is_empty() + } + Some(Statement::Insert(ins)) => match &ins.table { + TableObject::TableName(name) => { + require_unqualified(name).map_or(true, |n| n == "blocks") + } + _ => true, + }, + _ => false, + } +} + /// Materialises the rows matched by `target`/`selection`, optionally /// evaluating `set_value` (the `UPDATE ... SET content = ` value, /// per matched row) into `MatchedBlockEdit`s. `set_value` is `None` for @@ -522,6 +551,12 @@ impl DocumentStore { .ok_or_else(|| MqdbError::SqlParse("empty query".into()))?; match stmt { + Statement::Update(_) if !statement_needs_write_back(sql) => { + SqlEngine::with_session(self, self.write_session)?.execute(sql) + } + Statement::Delete(_) if !statement_needs_write_back(sql) => { + SqlEngine::with_session(self, self.write_session)?.execute(sql) + } Statement::Update(update) => { if update.from.is_some() { return Err(MqdbError::SqlExec( diff --git a/src/store.rs b/src/store.rs index 11eb019..c4c47c4 100644 --- a/src/store.rs +++ b/src/store.rs @@ -93,7 +93,6 @@ use crate::{ error::MqdbError, index, indexes::{DocumentIndex, TokenizerKind}, - query::Query, storage::{ Storage, catalog::{CatalogEntry, CustomTableEntry, ViewEntry}, @@ -199,9 +198,7 @@ impl VacuumReport { /// let mut store = DocumentStore::new(); /// store.add_str("# Hello\n\nWorld").unwrap(); /// -/// let results = store.query().heading_depth(1).blocks(); -/// assert_eq!(results.len(), 1); -/// assert_eq!(results[0].content, "Hello"); +/// assert_eq!(store.len(), 1); /// ``` pub struct DocumentStore { documents: Vec, @@ -740,11 +737,6 @@ impl DocumentStore { self.documents.is_empty() } - /// Creates a new query builder backed by this store. - pub fn query(&self) -> Query<'_> { - Query::new(self) - } - /// Aggregate block-type / code-language statistics across every /// document currently loaded in memory. pub fn stats(&self) -> StoreStats {