Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ print!("{}", out.to_table());

## Documentation

- [Install](https://db.mqlang.org/book/start/install) and [CLI](https://db.mqlang.org/book/start/cli) (`index`, `find`, `lint`, `stats`, `vacuum`, …)
- [Install](https://db.mqlang.org/book/start/install) and [CLI](https://db.mqlang.org/book/start/cli) (`index`, `find`, `stats`, `vacuum`, …)
- [Library API](https://db.mqlang.org/book/start/library), [HTTP server](https://db.mqlang.org/book/start/http-server)
- [SQL schema](https://db.mqlang.org/book/reference/sql-schema), [functions](https://db.mqlang.org/book/reference/sql-functions) and [examples](https://db.mqlang.org/book/reference/sql-examples)
- [Block model](https://db.mqlang.org/book/reference/block-model), [index layers](https://db.mqlang.org/book/reference/index-layers) and [storage format](https://db.mqlang.org/book/reference/storage-format)
Expand Down
2 changes: 0 additions & 2 deletions docs/books/src/reference/index-layers.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ Built once per document and stored in the `.mq-db` file. Checked before any bloc
| `depth = N` (`N > 0`) | `N` exceeds `max_heading_depth` |
| `block_type = 'heading' AND content = 'X'` | `heading_contents` has no case-insensitive match for `X` |

**Via the Rust API**: `store.query().documents(|doc| ...)` lets you filter on *any* zone-map field yourself (`heading_slugs`, `frontmatter_keys`, `title`, `tags`, …), not just the patterns `SqlEngine` recognizes automatically.

## Layer 2: Interval Index (section hierarchy)

Heading hierarchy is encoded as `(pre, post)` pairs via Pre-Post Order (Nested Set) traversal:
Expand Down
8 changes: 7 additions & 1 deletion docs/books/src/reference/sql-ddl.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# DDL Statements

`mq-db` supports a small set of DDL statements for defining **custom in-memory tables** alongside the built-in `documents`/`blocks` virtual tables. Custom tables live only for the process lifetime; they are not persisted to the `.mq-db` store file.
`mq-db` supports a small set of DDL statements for defining **custom in-memory tables** alongside the built-in `documents`/`blocks` virtual tables. Custom tables are saved in the `.mq-db` store file, so they are there the next time it is opened.

| Statement | Description |
| --- | --- |
| `CREATE TABLE name AS SELECT …` | Create a custom table from a query result |
| `CREATE TABLE name (col TYPE, …)` | Create an empty custom table with explicit schema |
| `INSERT INTO name VALUES (…)` | Insert a row into a custom table |
| `UPDATE name SET col = expr, … [WHERE …]` | Change rows of a custom table. Every expression sees the row as it was before the statement, so `SET a = b, b = a` swaps |
| `DELETE FROM name [WHERE …]` | Delete rows of a custom table |
| `DROP TABLE name` | Drop a custom table |
| `SHOW TABLES` | List all custom tables and views |
| `DESC name` | Show the schema of a custom table or view |
Expand All @@ -16,6 +18,10 @@
| `ATTACH DATABASE 'path' AS alias` | Mount another `.mq-db` store as `alias.<table>` |
| `DETACH alias` | Unmount a previously attached store |

`UPDATE` and `DELETE` accept an alias and subqueries in `WHERE`, enforce `NOT NULL` and `UNIQUE`, and take effect atomically: a failing statement leaves the table unchanged. They roll back with `ROLLBACK`. `UPDATE ... FROM`, `JOIN`, `RETURNING`, `ORDER BY` and `LIMIT` are not supported. The same statements on `blocks` are [write-back](write-back.md) and need `--write-back`.

Cells are stored as text. In arithmetic a cell holding a number counts as that number (`SET age = age + 1`), and the result is stored back as text.

## Examples

```bash
Expand Down
3 changes: 1 addition & 2 deletions docs/books/src/reference/vector-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ INSERT INTO emb SELECT * FROM read_json('new-embeddings.jsonl');
Drop embeddings of blocks that no longer exist:

```sql
CREATE OR REPLACE TABLE emb AS
SELECT * FROM emb WHERE content_hash IN (SELECT content_hash FROM blocks);
DELETE FROM emb WHERE content_hash NOT IN (SELECT content_hash FROM blocks);
```

## Searching
Expand Down
2 changes: 2 additions & 0 deletions docs/books/src/reference/write-back.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# INSERT / UPDATE / DELETE with Write-Back

Statements on [custom tables](sql-ddl.md) change only the `.mq-db` file and never need the flag. The rest of this page is about `blocks`.

`INSERT`/`UPDATE`/`DELETE` on `blocks` write the change back to the document's *source Markdown file* (re-parsed in place, same `DocumentId`), pass `--write-back` to allow it; without the flag the statement is rejected:

```bash
Expand Down
16 changes: 0 additions & 16 deletions docs/books/src/start/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,22 +96,6 @@ mq-db repl --db store.mq-db --mode sql

See [REPL](repl.md) for the full command list.

## `lint`

Run structural lint checks (currently: a heading at the given depth immediately followed by a list).

```bash
mq-db lint --db store.mq-db --depth 2
```

```text
✗ 1 violation (H2 immediately followed by list)

file heading
──────────────────────────────────────── ──────────────────────────────
docs/DESIGN.md "Quick Start"
```

## `stats`

Show store-wide statistics: document/block counts, block-type distribution, code-language distribution.
Expand Down
23 changes: 1 addition & 22 deletions docs/books/src/start/library.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,13 @@ mq-db = "0.1"
```

```rust
use mq_db::{DocumentStore, SqlEngine, MqEngine, block::BlockType};
use mq_db::{DocumentStore, SqlEngine, MqEngine};

// Build in memory
let mut store = DocumentStore::new();
store.add_file("docs/DESIGN.md")?;
store.add_str("# Hello\n\n## Architecture\n\nDetails\n")?;

// Chainable query API: zone-map skip + interval scope + block predicates
let chunks = store.query()
.documents(|doc| doc.zone_maps.heading_contents.contains("Architecture"))
.under_heading("Architecture", Some(2))
.filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code))
.blocks();

// SQL engine (custom sqlparser-based evaluator, no SQLite dependency)
let engine = SqlEngine::new(&store)?;
let out = engine.execute(
Expand All @@ -32,9 +25,6 @@ print!("{}", out.to_table());
// mq engine
let results = MqEngine::eval_store(".h1", &store)?;

// Structural lint
let violations = store.query().lint_heading_followed_by(2, &[BlockType::List]);

// UPDATE/DELETE with write-back: rewrites the affected block's source file
// (heading/paragraph content only), then re-parses it in place
store.execute_sql_mut(
Expand Down Expand Up @@ -66,14 +56,3 @@ let store = DocumentStore::load_catalog_only("store.mq-db")?;
| `DocumentStore::load_catalog_only()` | Catalog only | `list` / `stats`-style metadata commands |

When using `open()`, call `load_all_blocks()` and `load_all_indexes()` before running any `SqlEngine` query.

## Query builder

`store.query()` returns a chainable builder that applies the same three index layers used by the SQL engine, in order:

1. `.documents(|doc| ...)`: zone-map predicate, skips whole documents
2. `.under_heading(title, depth)` / interval-scope helpers: narrows to a `(pre, post)` range
3. `.filter(|block| ...)`: per-block predicate over the remaining candidates
4. `.blocks()`: materializes the final `Vec<&Block>`

See [Index Layers](../reference/index-layers.md) for how each layer works.
70 changes: 2 additions & 68 deletions src/bin/mq-db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -187,17 +187,6 @@ enum Commands {
attach: Vec<String>,
},

/// Run structural lint checks
Lint {
/// Path to .mq-db store file
#[arg(short, long, default_value = "store.mq-db")]
db: PathBuf,

/// Heading depth to check (H1=1 .. H6=6)
#[arg(long, default_value_t = 2)]
depth: u8,
},

/// Show store statistics
Stats {
/// Path to .mq-db store file
Expand Down Expand Up @@ -894,43 +883,6 @@ async fn main() -> anyhow::Result<()> {
run_repl(store, mode, write_back)?;
}

// lint
Commands::Lint { db, depth } => {
let store = load_store(&db)?;
let q = store.query();
let violations = q.lint_heading_followed_by(depth, &[BlockType::List]);
if violations.is_empty() {
println!(
"✓ No violations (H{} must not be immediately followed by a list)",
depth
);
} else {
let n = violations.len();
println!(
"✗ {} violation{} (H{} immediately followed by list)\n",
n,
if n == 1 { "" } else { "s" },
depth
);
println!(" {:<40} heading", "file");
println!(" {} {}", "─".repeat(40), "─".repeat(30));
for v in &violations {
let path = v
.document
.path
.as_ref()
.map(|p| p.to_string_lossy().to_string())
.unwrap_or_else(|| format!("<doc {}>", v.document.id));
let path_display = if path.len() > 40 {
format!("…{}", &path[path.len() - 39..])
} else {
path
};
println!(" {:<40} \"{}\"", path_display, v.heading.content);
}
}
}

// stats
Commands::Stats { db } => {
let store = load_store(&db)?;
Expand Down Expand Up @@ -1473,27 +1425,9 @@ fn md_block_to_html(s: &str) -> String {

// REPL

/// Crude prefix check used to gate write-back before even attempting a
/// parse. Only `INSERT INTO blocks` requires `--write-back`; custom tables
/// don't touch Markdown files.
/// Whether `sql` writes to Markdown files and so needs `--write-back`.
fn is_write_statement(sql: &str) -> bool {
let trimmed = sql.trim().trim_end_matches(';');
let upper = trimmed.to_ascii_uppercase();
upper.starts_with("UPDATE ") || upper.starts_with("DELETE ") || is_insert_into_blocks(&upper)
}

fn is_insert_into_blocks(upper: &str) -> bool {
let Some(after_into) = upper.strip_prefix("INSERT INTO") else {
return false;
};
let table = after_into
.trim_start()
.trim_start_matches(['"', '`'])
.trim_start();
table == "BLOCKS"
|| ["BLOCKS ", "BLOCKS(", "BLOCKS\"", "BLOCKS`"]
.iter()
.any(|p| table.starts_with(p))
mq_db::sql::statement_needs_write_back(sql)
}

fn run_repl(
Expand Down
34 changes: 7 additions & 27 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,38 +16,20 @@
//! slugs, code languages, front-matter keys/tags) that let the query
//! engine skip irrelevant documents before scanning their blocks.
//!
//! * **Chainable query API** – document-level and block-level predicates,
//! `UNDER heading` section scoping, built-in linter helpers.
//!
//! ## Quick Start
//!
//! ```rust
//! use mq_db::{DocumentStore, block::BlockType};
//!
//! let mut store = DocumentStore::new();
//! store.add_str("# Hello\n\n## Architecture\n\nDetails\n\n```rust\ncode\n```\n").unwrap();
//!
//! // Extract all content under the "Architecture" H2
//! let chunks = store.query()
//! .under_heading("Architecture", Some(2))
//! .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code))
//! .blocks();
//!
//! assert_eq!(chunks.len(), 2);
//! ```
//!
//! ## Structural Linting
//!
//! ```rust
//! use mq_db::{DocumentStore, block::BlockType};
//! use mq_db::{DocumentStore, SqlEngine};
//!
//! let mut store = DocumentStore::new();
//! store.add_str("## Section\n\n- item without intro paragraph\n").unwrap();
//! store.add_str("# Hello\n\n## Architecture\n\nDetails\n").unwrap();
//!
//! let q = store.query();
//! let violations = q.lint_heading_followed_by(2, &[BlockType::List]);
//! let engine = SqlEngine::new(&store).unwrap();
//! let out = engine
//! .execute("SELECT content FROM blocks WHERE block_type = 'heading' ORDER BY pre")
//! .unwrap();
//!
//! assert_eq!(violations.len(), 1);
//! assert_eq!(out.rows.len(), 2);
//! ```

pub mod block;
Expand All @@ -57,7 +39,6 @@ pub mod error;
pub mod index;
pub mod indexes;
pub mod mq_engine;
pub mod query;
pub mod search;
pub mod sql;
pub mod storage;
Expand All @@ -67,7 +48,6 @@ pub mod tui;
pub use document::Document;
pub use error::MqdbError;
pub use mq_engine::MqEngine;
pub use query::{LintViolation, Query, QueryResult};
pub use sql::{QueryOutput, SqlEngine};
pub use storage::Storage;
pub use storage::catalog::CatalogEntry;
Expand Down
Loading
Loading