diff --git a/README.md b/README.md index 438659d..023ca5b 100644 --- a/README.md +++ b/README.md @@ -1,504 +1,91 @@
- + mq-db logo -

mq-db

+Visit the site 🌐 +β€” +Read the book πŸ“– -**Markdown-specialized embedded database with interval-indexed block storage and hierarchical query support.** +

mq-db

-[![ci](https://img.shields.io/github/actions/workflow/status/harehare/mq-db/ci.yml?logo=github-actions&label=ci)](https://github.com/harehare/mq-db/actions/workflows/ci.yml) -[![audit](https://img.shields.io/github/actions/workflow/status/harehare/mq-db/audit.yml?logo=shield&label=audit)](https://github.com/harehare/mq-db/actions/workflows/audit.yml) -[![crates.io](https://img.shields.io/crates/v/mq-db?logo=rust)](https://crates.io/crates/mq-db) -[![license](https://img.shields.io/badge/license-MIT-b3402c)](LICENSE) - -![demo](./assets/demo.gif) +[![ci](https://img.shields.io/github/actions/workflow/status/harehare/mq-db/ci.yml?style=flat-square&logo=github-actions&label=ci)](https://github.com/harehare/mq-db/actions/workflows/ci.yml) +[![crates.io](https://img.shields.io/crates/v/mq-db?style=flat-square&logo=rust)](https://crates.io/crates/mq-db) +[![license](https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square)](LICENSE)
-`mq-db` treats Markdown documents as **structured, hierarchical databases** rather than plain text. It parses Markdown into a flat block list with an **interval index** (Nested Set / Pre-Post Order), enabling O(1) section hierarchy queries. Documents can be queried with **SQL** or **[mq](https://github.com/harehare/mq)** and persisted to a compact custom page-file format. +`mq-db` is an embedded database that treats Markdown documents as structured, hierarchical data. +It parses Markdown into a flat block list with an interval index (pre/post order), so section hierarchy queries are a single integer comparison. Documents can be queried with **SQL** or **[mq](https://github.com/harehare/mq)** and persisted to a compact page-file format. -```mermaid -%%{init: {'theme':'base', 'themeVariables': {'primaryColor':'#f2ebdb','primaryTextColor':'#2a2420','primaryBorderColor':'#b3402c','lineColor':'#b3402c','secondaryColor':'#e3c3b7','tertiaryColor':'#faf6ef','background':'#faf6ef','fontFamily':'JetBrains Mono, monospace'}}}%% -flowchart TD - A["Markdown File(s)"] -->|"CST Parser (mq-markdown)"| B["Block Tree\n(heading Β· paragraph Β· code Β· list …)"] - B -->|"Interval Index + Secondary Indexes"| C["Flat Block Vector\n(pre/post integers)"] - C --> D["BitmapIndex\n(block_type)"] - C --> E["BTreeIndex\n(pre / post)"] - C --> F["HashIndex\n(content / lang / depth)"] - C --> G["Zone Maps\n(per-document stats)"] - C --> H["SQL Engine\n(sqlparser, custom native evaluator)"] - C --> I["mq Engine\n(mq-lang evaluator)"] -``` +![demo](./assets/demo.gif) -> [!IMPORTANT] -> This project is under active development and the API may change. +> [!WARNING] +> `mq-db` is currently under active development and the API may change. ## Features -- **Flat block storage**: every Markdown element becomes a typed `Block` with row-polymorphic properties -- **O(1) hierarchy queries**: interval index (`pre`/`post`) makes ancestor/descendant checks a single integer comparison -- **Four-layer secondary indexes**: `BitmapIndex` (block type), `BTreeIndex` (pre/post), `HashIndex` (content/lang/depth), `TermIndex` (tokenized content, full-text) for fast SQL predicate pushdown -- **Zone Maps**: per-document statistics skip irrelevant files before scanning any blocks -- **Dual query engines**: SQL via a custom `sqlparser`-based evaluator, and `mq` via `mq-lang` -- **`WITH` / `WITH RECURSIVE` support**: common table expressions, usable in `FROM`, `JOIN`, and subqueries; recursive CTEs via iterative fixed-point evaluation -- **Full-text search**: `match()`/`score()`/`bm25()` SQL functions backed by a persisted per-document inverted index -- **`EXPLAIN` / `EXPLAIN ANALYZE`**: see the zone-map/index/join plan a query resolves to, with actual row/timing stats under `ANALYZE` -- **Incremental re-indexing**: re-running `index` skips unchanged files (content-hash based), replaces changed ones in place (same `DocumentId`), and can `--prune` deleted ones -- **SQL `INSERT`/`UPDATE`/`DELETE` with write-back**: add, edit, or remove `blocks` and push the change back to the source Markdown file, opt-in via `--write-back` -- **DDL support**: `CREATE TABLE`, `INSERT INTO`, `DROP TABLE` for in-memory custom tables, with `NOT NULL`/`UNIQUE`/`PRIMARY KEY` constraints enforced on `INSERT` -- **`CREATE VIEW`**: persisted, live (non-materialized) named queries, re-run on every reference -- **`GROUP BY ... HAVING`, `LIMIT ... OFFSET`, `UNION`/`UNION ALL`/`INTERSECT`/`EXCEPT`**: standard aggregate filtering, pagination, and set operations across `SELECT`s -- **`BEGIN`/`COMMIT`/`ROLLBACK`**: transactions for custom-table and view DDL/DML -- **Comprehensive SQL function library**: string, numeric, null-handling, date/time (`date_trunc`/`date_diff`/`date_add`/`strftime`/`EXTRACT`), regex (`REGEXP`, `regexp_replace`, `regexp_extract`), `CASE`, and aggregate functions comparable to a general-purpose RDBMS -- **`mq()` scalar function**: run an mq program against Markdown content inline in SQL -- **`read_csv()` / `read_json()` table functions**: query external CSV/JSON Lines files directly in `FROM`, no import step -- **`ATTACH DATABASE` / `DETACH`**: query across multiple `.mq-db` stores as `.blocks`, session-scoped like SQLite -- **Custom page-file persistence**: 8 KB fixed pages, checksums, atomic writes -- **`vacuum`**: reclaim dead page chains left by write-back edits, `DROP TABLE`/`DROP VIEW`, and re-indexing changed files -- **CLI + interactive REPL + TUI**: full terminal experience - -Full documentation, including the SQL/mq reference and architecture deep-dive, lives at **[db.mqlang.org/book](https://db.mqlang.org/book/)**. +- Query Markdown with [SQL](https://db.mqlang.org/book/reference/sql-schema) or mq, and hierarchy queries with `under()` +- [Full-text search](https://db.mqlang.org/book/reference/full-text-search) (`match()` / `bm25()`) and [vector search](https://db.mqlang.org/book/reference/vector-search) +- [CTEs](https://db.mqlang.org/book/reference/cte), [`EXPLAIN`](https://db.mqlang.org/book/reference/explain), [custom tables, views and `ATTACH`](https://db.mqlang.org/book/reference/sql-ddl) +- [`read_csv()` / `read_json()`](https://db.mqlang.org/book/reference/external-files) to query external files in `FROM` +- [`INSERT` / `UPDATE` / `DELETE` with write-back](https://db.mqlang.org/book/reference/write-back) to the source Markdown +- Incremental re-indexing, zone maps and secondary indexes for fast scans +- [CLI](https://db.mqlang.org/book/start/cli), [REPL](https://db.mqlang.org/book/start/repl), [TUI](https://db.mqlang.org/book/start/tui), [HTTP server](https://db.mqlang.org/book/start/http-server) and [Rust library](https://db.mqlang.org/book/start/library) ## Installation -### Using the Installation Script (Recommended) - ```bash curl -fsSL https://raw.githubusercontent.com/harehare/mq-db/main/bin/install.sh | bash ``` -The installer will: -- Download the latest release for your platform -- Verify the binary with SHA256 checksum -- Install to `~/.local/bin/` -- Update your shell profile (bash, zsh, or fish) - -After installation, restart your terminal or run: -```bash -source ~/.bashrc # or ~/.zshrc, or ~/.config/fish/config.fish -``` - -### Using Cargo +or with Cargo: -```bash +```sh cargo install mq-db ``` -### From Source - -```bash -# Latest Development Version -cargo install --git https://github.com/harehare/mq-db.git -``` - -### Supported Platforms - -- **Linux**: x86_64, aarch64 -- **macOS**: x86_64 (Intel), aarch64 (Apple Silicon) -- **Windows**: x86_64 - -## CLI Usage - -### Index Markdown files - -```bash -mq-db index docs/ --recursive --output store.mq-db -mq-db index README.md DESIGN.md -mq-db index docs/ --no-spans # omit source spans (~21 bytes/block saved) -``` - -``` - + docs/DESIGN.md - + docs/API.md - -2 added, 0 updated, 0 unchanged, 0 removed β†’ store.mq-db -``` - -Re-running `index` against an existing `--output` is **incremental**: files -whose content hash hasn't changed are skipped, changed files are re-parsed -in place (keeping the same `DocumentId`), and new files are added. Pass -`--prune` to also drop catalogued documents whose file no longer exists: - -```bash -mq-db index docs/ --recursive --output store.mq-db --prune -``` - -### List indexed documents - -```bash -mq-db list --db store.mq-db -mq-db list --db store.mq-db --format json # also: csv, tsv, markdown, html -``` - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ ID β”‚ Path / Title β”‚ Blocks β”‚ Tags β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ 0 β”‚ docs/DESIGN.md β”‚ 142 β”‚ β”‚ -β”‚ 1 β”‚ docs/API.md β”‚ 87 β”‚ api, v2 β”‚ -β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -2 documents -``` - -### Quick full-text search - -`find` is a shortcut for `match()`/`bm25()` full-text search, no SQL needed. It falls back to a case-insensitive substring match too, so partial CJK queries still hit. Results show a snippet centred on the match, with matched terms highlighted when stdout is a terminal (respects `NO_COLOR`). See [Full-Text Search](https://db.mqlang.org/book/reference/full-text-search) for details and known limitations. - -```bash -mq-db find "error handling" --db store.mq-db -mq-db find "error handling" --db store.mq-db -n 5 -F json # top 5, JSON -``` - -``` -docs/API.md ΒΆ 0.67 Error handling follows RFC 7807 problem details... -docs/API.md # 0.25 Error Handling - -2 matches -``` - -### SQL queries +## Usage ```bash +mq-db index docs/ --recursive --output store.mq-db # incremental; --prune drops deleted files +mq-db list --db store.mq-db # list indexed documents +mq-db find "error handling" --db store.mq-db # full-text search mq-db sql "SELECT block_type, count(*) FROM blocks GROUP BY block_type" --db store.mq-db -mq-db sql --file query.sql --db store.mq-db # read SQL from a file -mq-db sql "SELECT ..." --db store.mq-db --format json # also: csv, tsv, markdown, html -``` - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ block_type β”‚ count(*) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ paragraph β”‚ 48 β”‚ -β”‚ heading β”‚ 21 β”‚ -β”‚ code β”‚ 15 β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -(3 rows) -``` - -**Hierarchy query with `under()`**: find all content inside a specific section: - -```bash -mq-db sql " - SELECT b.block_type, b.content - FROM blocks b - WHERE under(b.pre, b.post, - (SELECT pre FROM blocks WHERE block_type = 'heading' AND content = 'Architecture'), - (SELECT post FROM blocks WHERE block_type = 'heading' AND content = 'Architecture')) - ORDER BY b.pre -" --db store.mq-db -``` - -**`mq()` scalar function**: run an mq program against Markdown content inline: - -```bash -mq-db sql "SELECT mq('.h1 | to_text', content) AS title FROM blocks WHERE block_type = 'code'" --db store.mq-db -``` - -The SQL dialect also has CTEs, full-text search, `EXPLAIN`, external-file table functions, cross-store `ATTACH`, write-back DML, custom tables, and live views. Full reference and examples are in the book: - -- [CTEs (`WITH` / `WITH RECURSIVE`)](https://db.mqlang.org/book/reference/cte) -- [Full-Text Search (`match()` / `score()` / `bm25()`)](https://db.mqlang.org/book/reference/full-text-search) -- [`EXPLAIN` / `EXPLAIN ANALYZE`](https://db.mqlang.org/book/reference/explain) -- [External Files (`read_csv()` / `read_json()`)](https://db.mqlang.org/book/reference/external-files) -- [DDL Statements (custom tables, views, `ATTACH`/`DETACH`)](https://db.mqlang.org/book/reference/sql-ddl) -- [Write-Back (`INSERT`/`UPDATE`/`DELETE`)](https://db.mqlang.org/book/reference/write-back) - -### mq queries - -```bash -mq-db mq ".h1" --db store.mq-db mq-db mq 'select(.code_lang == "rust")' --db store.mq-db -mq-db mq ".h1" --db store.mq-db --format markdown # also: json, csv, tsv, html -``` - -### Interactive REPL - -```bash -mq-db repl --db store.mq-db --mode sql -``` - -``` -mq-db (.help for commands .quit to exit) -mode: sql (.mode mq | .mode sql) - -sql> SELECT content FROM blocks WHERE block_type = 'heading' LIMIT 3; -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ content β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Overview β”‚ -β”‚ Architecture β”‚ -β”‚ Query Engine β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -(3 rows) - -sql> .mode mq -β†’ mq mode -mq> .h2 -## Architecture -## Query Engine -``` - -Pass `--write-back` to `repl` to allow `UPDATE`/`DELETE` on `blocks` in SQL mode: - -```bash -mq-db repl --db store.mq-db --mode sql --write-back -``` - -### HTTP server - -```bash -mq-db serve --db store.mq-db # listens on 127.0.0.1:7878 -mq-db serve --db store.mq-db --port 8080 # custom port -mq-db serve --db store.mq-db --host 0.0.0.0 --port 8080 -``` - -`--host 0.0.0.0` exposes the query endpoints beyond localhost. When doing so, secure the server with an API key or Basic auth, and consider TLS and a rate limit: - -```bash -mq-db serve --db store.mq-db --host 0.0.0.0 \ - --api-key "$MQ_DB_API_KEY" \ - --rate-limit 20 \ - --timeout 10 \ - --tls-cert cert.pem --tls-key key.pem -``` - -| Option | Description | -| -------------------------- | -------------------------------------------------------------------------------- | -| `--timeout ` | Abort a request and return `408` if it runs longer than this many seconds | -| `--rate-limit ` | Max requests per second per client IP; excess requests get `429` | -| `--api-key ` | Require `Api-Key: ` or `Authorization: Bearer ` (env `MQ_DB_API_KEY`) | -| `--basic-auth ` | Require HTTP Basic auth (env `MQ_DB_BASIC_AUTH`) | -| `--tls-cert` / `--tls-key` | PEM certificate/key pair to serve over HTTPS instead of plain HTTP | - -If both `--api-key` and `--basic-auth` are set, either credential grants access. `--tls-cert` and `--tls-key` must be provided together. - -Three endpoints are available: - -| Method | Path | Body | Description | -| ------ | --------- | ---------------------- | ---------------------------------------------------- | -| `GET` | `/health` | (none) | `{"status":"ok","documents":}` | -| `POST` | `/sql` | `{"query":"SELECT …"}` | Execute a SQL query, returns JSON rows | -| `POST` | `/mq` | `{"code":".h1"}` | Evaluate an mq expression, returns `{"results":[…]}` | - -```bash -# Health check -curl http://127.0.0.1:7878/health - -# SQL via HTTP -curl -s -X POST http://127.0.0.1:7878/sql \ - -H 'Content-Type: application/json' \ - -d '{"query":"SELECT block_type, count(*) FROM blocks GROUP BY block_type"}' - -# mq via HTTP -curl -s -X POST http://127.0.0.1:7878/mq \ - -H 'Content-Type: application/json' \ - -d '{"code":".h1"}' -``` - -### Structural linting - -```bash -mq-db lint --db store.mq-db --depth 2 -``` - -``` -βœ— 1 violation (H2 immediately followed by list) - - file heading - ──────────────────────────────────────── ────────────────────────────── - docs/DESIGN.md "Quick Start" -``` - -### Statistics - -```bash -mq-db stats --db store.mq-db -``` - -``` - Documents 5 - Blocks 632 - - Block types - ──────────────────────────────────────────────────────── - ΒΆ paragraph β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘ 241 (38%) - # heading β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 89 (14%) - {} code β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 73 (12%) - β€’ list β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 58 (9%) - - Code languages - ──────────────────────────────────────────────────────── - {} rust β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ 41 (57%) - {} python β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 18 (25%) - {} bash β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 14 (19%) -``` - -### Compaction (VACUUM) - -`UPDATE`/`DELETE` write-back, `DROP TABLE`/`DROP VIEW`, and re-indexing a changed file all replace or remove data by writing a fresh page chain and abandoning the old one, so the `.mq-db` file only grows. `vacuum` rewrites the file from scratch (same compaction `save`/`index` already do for a brand-new file) and reclaims that dead space: - -```bash -mq-db vacuum --db store.mq-db -``` - -``` - Pages before 190 - Pages after 123 - Reclaimed 536.0 KB -``` - -`VACUUM` as a SQL statement is recognized but redirects to this command rather than doing something different or silently failing: mq-db's compaction operates on the whole store file, not per-table, so it doesn't fit the `--write-back`/`execute_sql_mut` path the way `UPDATE`/`DELETE` do. - -### Show document structure - -```bash -mq-db show 0 --db store.mq-db -``` - -``` - docs/DESIGN.md - title Design Document - blocks 142 - - pre post type content - ──── ──── ──────────────── ────────────────────────────────────────── - 0 141 heading H1 Design Document - 2 55 heading H2 Architecture - 4 21 paragraph The system is built on… - 22 37 heading H3 Query Engine - 24 36 code fn main() { … } +mq-db repl --db store.mq-db --mode sql # interactive REPL +mq-db tui --db store.mq-db # terminal UI +mq-db serve --db store.mq-db # HTTP server on 127.0.0.1:7878 ``` -### TUI +Find everything inside a section with `under()`: -```bash -mq-db tui --db store.mq-db -``` - -``` - mq-db SQL Tab:switch i:input j/k:nav d/u:scroll q:quit -β”Œβ”€ Documents ──────────┬─ SQL ────────────────────────────────────────────────┐ -β”‚ DESIGN.md β”‚ SELECT block_type, count(*) FROM blocks GROUP BY b_ β”‚ -β”‚ 142 blocks β”œβ”€ Results ───────────────────────────────────────────── -β”‚ API.md β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ -β”‚ 87 blocks API β”‚ β”‚ block_type β”‚ count(*) β”‚ β”‚ -β”‚ README.md β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ -β”‚ 34 blocks β”‚ β”‚ paragraph β”‚ 48 β”‚ β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - 5 docs 632 blocks 3 rows +```sql +SELECT b.block_type, b.content +FROM blocks b +WHERE under(b.pre, b.post, + (SELECT pre FROM blocks WHERE block_type = 'heading' AND content = 'Architecture'), + (SELECT post FROM blocks WHERE block_type = 'heading' AND content = 'Architecture')) +ORDER BY b.pre; ``` -`Tab` cycles the input mode through SQL β†’ **find** β†’ mq, giving the [`find` CLI command](#quick-full-text-search)'s highlighted results right in the TUI. - -**Keys:** - -| Key | Action | -| -------------- | ------------------------ | -| `i` | Focus query input | -| `Esc` | Blur input | -| `Enter` | Run query | -| `Tab` | Cycle SQL / find / mq mode | -| `j` / `k` | Navigate document list | -| `d` / `u` | Scroll results down / up | -| `g` / `G` | Jump to top / bottom | -| `q` / `Ctrl+C` | Quit | - -## Library API +Use it as a library: ```rust -use mq_db::{DocumentStore, SqlEngine, MqEngine, block::BlockType}; +use mq_db::{DocumentStore, SqlEngine}; -// Build in memory let mut store = DocumentStore::new(); store.add_file("docs/DESIGN.md")?; -store.add_str("# Hello\n\n## Architecture\n\nDetails\n")?; -// Chainable query API: zone-map skip + interval scope + block predicates -let chunks = store.query() - .documents(|doc| doc.zone_maps.heading_contents.contains("Architecture")) - .under_heading("Architecture", Some(2)) - .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code)) - .blocks(); - -// SQL engine (custom sqlparser-based evaluator, no SQLite dependency) let engine = SqlEngine::new(&store)?; -let out = engine.execute( - "SELECT content FROM blocks WHERE block_type = 'heading' ORDER BY pre" -)?; +let out = engine.execute("SELECT content FROM blocks WHERE block_type = 'heading' ORDER BY pre")?; print!("{}", out.to_table()); - -// mq engine -let results = MqEngine::eval_store(".h1", &store)?; - -// Structural lint -let violations = store.query().lint_heading_followed_by(2, &[BlockType::List]); - -// Incremental re-index: skips unchanged files (content-hash based), replaces -// changed ones in place (same DocumentId), adds new ones; prune=true drops -// missing paths. -let report = store.reindex_paths(&[std::path::PathBuf::from("docs/DESIGN.md")], false)?; -println!("{} added, {} updated, {} unchanged", report.added.len(), report.updated.len(), report.unchanged); - -// UPDATE/DELETE with write-back: rewrites the affected block's source file -// (heading/paragraph content only), then re-parses it in place. -store.execute_sql_mut( - "UPDATE blocks SET content = 'New Title' WHERE block_type = 'heading' AND content = 'Old Title'" -)?; - -// Persist / load -store.save("store.mq-db")?; - -// Full load: all blocks read into memory, indexes built on first SqlEngine use -let store = DocumentStore::load("store.mq-db")?; - -// Lazy open: catalog only; call load_all_blocks() + load_all_indexes() before SQL -let mut store = DocumentStore::open("store.mq-db")?; -store.load_all_blocks()?; -store.load_all_indexes()?; - -// Catalog-only: for metadata commands (list, stats) that don't need block data -let store = DocumentStore::load_catalog_only("store.mq-db")?; ``` -See the book's [Library API](https://db.mqlang.org/book/start/library) page for the loading-strategy comparison and the full query-builder reference. - -## SQL Reference - -The virtual schema (`documents`/`blocks`), the full built-in function library (mq-db-specific, string, numeric, date/time, null-handling, and aggregate functions), and the DDL statement list live in the book, along with a set of worked example queries: - -- [Virtual Schema](https://db.mqlang.org/book/reference/sql-schema) -- [Built-in Functions](https://db.mqlang.org/book/reference/sql-functions) -- [DDL Statements](https://db.mqlang.org/book/reference/sql-ddl) -- [Example Queries](https://db.mqlang.org/book/reference/sql-examples) +## Documentation -A quick taste: - -```sql --- H2 headings immediately followed by a list (structural lint) -SELECT d.path, h.content AS heading -FROM blocks h -JOIN blocks nxt ON nxt.document_id = h.document_id AND nxt.pre = h.pre + 1 -JOIN documents d ON d.id = h.document_id -WHERE h.block_type = 'heading' AND depth = 2 AND nxt.block_type = 'list'; - --- Bucket headings by depth and summarize with string/numeric functions -SELECT - CASE WHEN depth <= 1 THEN 'top-level' ELSE 'nested' END AS bucket, - count(*), - group_concat(initcap(trim(content)), ', ') AS headings -FROM blocks -WHERE block_type = 'heading' -GROUP BY CASE WHEN depth <= 1 THEN 'top-level' ELSE 'nested' END; -``` - -## Architecture - -Every Markdown element becomes a `Block` (a typed struct with `pre`/`post` interval-index bounds and row-polymorphic `properties`), indexed through three complementary layers, cheapest-first: **Zone Maps** (document-level skip), the **Interval Index** (section hierarchy, `pre`/`post` containment), and **Secondary Indexes** (`BitmapIndex`/`BTreeIndex`/`HashIndex`/`TermIndex`). Documents persist to a custom 8 KB page file with atomic writes. - -See the book for the full architecture, with diagrams and the on-disk byte layout: - -- [Block Model](https://db.mqlang.org/book/reference/block-model) -- [Index Layers](https://db.mqlang.org/book/reference/index-layers) -- [Storage Format](https://db.mqlang.org/book/reference/storage-format) +- [Install](https://db.mqlang.org/book/start/install) and [CLI](https://db.mqlang.org/book/start/cli) (`index`, `find`, `lint`, `stats`, `vacuum`, …) +- [Library API](https://db.mqlang.org/book/start/library), [HTTP server](https://db.mqlang.org/book/start/http-server) +- [SQL schema](https://db.mqlang.org/book/reference/sql-schema), [functions](https://db.mqlang.org/book/reference/sql-functions) and [examples](https://db.mqlang.org/book/reference/sql-examples) +- [Block model](https://db.mqlang.org/book/reference/block-model), [index layers](https://db.mqlang.org/book/reference/index-layers) and [storage format](https://db.mqlang.org/book/reference/storage-format) ## Support @@ -513,4 +100,3 @@ Contributions are welcome! Feel free to open an issue or submit a pull request. ## License [MIT](LICENSE) - diff --git a/docs/books/src/reference/vector-search.md b/docs/books/src/reference/vector-search.md index 778e236..3082bfb 100644 --- a/docs/books/src/reference/vector-search.md +++ b/docs/books/src/reference/vector-search.md @@ -1,30 +1,58 @@ # Vector Search -mq-db does not compute embeddings. Compute them with any model, store them in a custom table as JSON arrays, and rank with `cosine_similarity`, `dot_product` or `l2_distance`. +mq-db does not compute embeddings. Compute them with any model, keep them in a custom table as JSON arrays, and rank with `cosine_similarity`, `dot_product` or `l2_distance`. -Key each embedding on `blocks.content_hash`, not `blocks.id`. The hash depends only on the block's type and content, so an embedding keeps matching its block when the block moves or other documents change. +## Key embeddings on `content_hash` + +`blocks.id` shifts when an earlier document changes, so do not key embeddings on it. [`blocks.content_hash`](sql-schema.md) depends only on a block's type and content: an unchanged block keeps its embedding across re-indexing and moves, and an edited block has a new hash, so it shows up as missing. + +## Workflow + +```bash +# 1. Export the blocks that have no embedding yet (all of them the first time). +mq-db sql --db store.mq-db --format json \ + "SELECT DISTINCT content_hash, content FROM blocks WHERE block_type <> 'yaml'" > todo.json + +# 2. Embed `content` with your model and write one JSON object per line: +# {"content_hash": "1c8d281685b6b265", "embedding": [0.12, -0.03, ...]} +# -> embeddings.jsonl + +# 3. Load them. +mq-db sql --db store.mq-db \ + "CREATE TABLE emb AS SELECT * FROM read_json('embeddings.jsonl')" +``` + +After documents change and are re-indexed, list what needs embedding and append it: + +```sql +SELECT DISTINCT content_hash, content FROM blocks +WHERE content_hash NOT IN (SELECT content_hash FROM emb); + +INSERT INTO emb SELECT * FROM read_json('new-embeddings.jsonl'); +``` + +Drop embeddings of blocks that no longer exist: ```sql --- Hand these to your embedding model. -SELECT DISTINCT content_hash, content FROM blocks; +CREATE OR REPLACE TABLE emb AS +SELECT * FROM emb WHERE content_hash IN (SELECT content_hash FROM blocks); +``` --- One JSON object per line: {"content_hash": "9f86d0...", "embedding": [0.12, -0.03, ...]} -CREATE TABLE emb AS SELECT * FROM read_json('embeddings.jsonl'); +## Searching -SELECT b.content -FROM emb e JOIN blocks b ON b.content_hash = e.content_hash +```sql +SELECT d.path, b.anchor, b.content +FROM emb e +JOIN blocks b ON b.content_hash = e.content_hash +JOIN documents d ON d.id = b.document_id ORDER BY cosine_similarity(e.embedding, '[0.1, 0.2, 0.3]') DESC LIMIT 5; ``` -A block whose content changed gets a new hash, so it simply stops matching. Find the blocks that still need an embedding: +Identical text in several places shares one embedding and appears once per location. Use `(documents.path, anchor)` instead when each location needs its own row. Integers compare equal to their text form (`1` and `'1'`), so a text column can join an integer key. -```sql -SELECT DISTINCT b.content_hash, b.content -FROM blocks b LEFT JOIN emb e ON b.content_hash = e.content_hash -WHERE e.content_hash IS NULL; -``` +Pass the query vector as a string literal. A row whose vector is invalid or has a different length scores `NULL`; `NULL` sorts lowest, so those rows come last under `DESC`. -Equal content shares one hash, so one embedding covers every copy. Use `(documents.path, anchor)` instead when each location needs its own row. `blocks.id` also works for a throwaway table, but it shifts whenever an earlier document gains or loses blocks. Integers compare equal to their text form (`1` and `'1'`), so a text column can join an integer key. +## Limits Every query scans the whole table, which is fast enough for tens of thousands of vectors (about one second for 20,000 vectors of 384 dimensions) but has no index. diff --git a/src/sql/aggregate.rs b/src/sql/aggregate.rs index 88437cf..2b79e51 100644 --- a/src/sql/aggregate.rs +++ b/src/sql/aggregate.rs @@ -521,7 +521,7 @@ impl<'a> SqlEngine<'a> { for (expr, desc) in &sort_keys { let va = eval_agg_expr_value(expr, group_by_exprs, &a.0, &a.1); let vb = eval_agg_expr_value(expr, group_by_exprs, &b.0, &b.1); - let mut ord = va.cmp_val(&vb).unwrap_or(std::cmp::Ordering::Equal); + let mut ord = va.sort_cmp(&vb); if *desc { ord = ord.reverse(); } diff --git a/src/sql/result_ops.rs b/src/sql/result_ops.rs index aa67893..b28b94c 100644 --- a/src/sql/result_ops.rs +++ b/src/sql/result_ops.rs @@ -24,9 +24,7 @@ pub(super) fn apply_order_by_exprs(rows: &mut [(Row, Vec)], exprs: &[Ord let mut order: Vec = (0..rows.len()).collect(); order.sort_by(|&a, &b| { for (i, ob) in exprs.iter().enumerate() { - let ord = keys[a][i] - .cmp_val(&keys[b][i]) - .unwrap_or(std::cmp::Ordering::Equal); + let ord = keys[a][i].sort_cmp(&keys[b][i]); // sort=None or Asc β†’ ascending; Desc β†’ descending let ord = if ob.options.sort == Some(OrderBySort::Desc) { ord.reverse() diff --git a/src/sql/tests.rs b/src/sql/tests.rs index c5c7589..99a3a35 100644 --- a/src/sql/tests.rs +++ b/src/sql/tests.rs @@ -3456,3 +3456,56 @@ fn test_sql_distinct_content_hash() { .unwrap(); assert_eq!(out.rows.len(), 1); } + +#[test] +fn test_sql_order_by_mixed_null_and_numbers_is_total() { + let store = DocumentStore::new(); + let engine = SqlEngine::new(&store).unwrap(); + engine + .execute("CREATE TABLE v (name TEXT, embedding TEXT)") + .unwrap(); + let values: Vec = (0..200) + .map(|i| { + if i % 3 == 0 { + format!("('n{i}', 'bad')") + } else { + format!("('n{i}', '[1,{i}]')") + } + }) + .collect(); + engine + .execute(&format!("INSERT INTO v VALUES {}", values.join(","))) + .unwrap(); + let out = engine + .execute("SELECT name, cosine_similarity(embedding, '[1,0]') AS s FROM v ORDER BY s DESC") + .unwrap(); + assert_eq!(out.rows.len(), 200); + let nulls = out.rows.iter().filter(|r| r[1] == "NULL").count(); + assert_eq!(nulls, 67); + // Every number must come before every NULL. + let first_null = out.rows.iter().position(|r| r[1] == "NULL").unwrap(); + assert!(out.rows[first_null..].iter().all(|r| r[1] == "NULL")); + // And the numbers must be in descending order. + let nums: Vec = out.rows[..first_null] + .iter() + .map(|r| r[1].parse().unwrap()) + .collect(); + assert!(nums.windows(2).all(|w| w[0] >= w[1])); +} + +#[test] +fn test_sql_order_by_mixed_integer_and_numeric_string_keys() { + let store = DocumentStore::new(); + let engine = SqlEngine::new(&store).unwrap(); + engine.execute("CREATE TABLE v (name TEXT)").unwrap(); + engine + .execute("INSERT INTO v VALUES ('first'), ('second'), ('third')") + .unwrap(); + let out = engine + .execute( + "SELECT name FROM v ORDER BY CASE WHEN name = 'first' THEN 10 ELSE '2' END ASC, name", + ) + .unwrap(); + let names: Vec<&str> = out.rows.iter().map(|r| r[0].as_str()).collect(); + assert_eq!(names, vec!["second", "third", "first"]); +} diff --git a/src/sql/value.rs b/src/sql/value.rs index 2fd0758..6388df6 100644 --- a/src/sql/value.rs +++ b/src/sql/value.rs @@ -61,6 +61,29 @@ impl Value { } } + /// A total order for `ORDER BY`: `NULL`, then booleans, then numbers + /// (integers, floats and canonical integer strings, by exact numeric + /// value; `NaN` last), then other strings. `NULL` sorts lowest, so it + /// comes last under `DESC`. + pub(super) fn sort_cmp(&self, other: &Value) -> std::cmp::Ordering { + fn rank(v: &Value) -> u8 { + match v { + Value::Null => 0, + Value::Bool(_) => 1, + _ if sort_number(v).is_some() => 2, + _ => 3, + } + } + match (self, other) { + (Value::Bool(a), Value::Bool(b)) => a.cmp(b), + (Value::Str(a), Value::Str(b)) if rank(self) == 3 && rank(other) == 3 => a.cmp(b), + _ => match (sort_number(self), sort_number(other)) { + (Some(a), Some(b)) => cmp_num(a, b), + _ => rank(self).cmp(&rank(other)), + }, + } + } + pub(super) fn cmp_val(&self, other: &Value) -> Option { match (self, other) { (Value::Int(a), Value::Str(s)) => canonical_int(s).map(|b| a.cmp(&b)), @@ -76,6 +99,54 @@ impl Value { } } +/// A numeric `ORDER BY` key. +#[derive(Clone, Copy)] +enum SortNum { + Int(i64), + Float(f64), +} + +fn sort_number(v: &Value) -> Option { + match v { + Value::Int(i) => Some(SortNum::Int(*i)), + Value::Float(f) => Some(SortNum::Float(*f)), + Value::Str(s) => canonical_int(s).map(SortNum::Int), + _ => None, + } +} + +/// Exact numeric comparison, with `NaN` above everything (including infinity). +fn cmp_num(a: SortNum, b: SortNum) -> std::cmp::Ordering { + match (a, b) { + (SortNum::Int(a), SortNum::Int(b)) => a.cmp(&b), + (SortNum::Int(a), SortNum::Float(b)) => cmp_i64_f64(a, b), + (SortNum::Float(a), SortNum::Int(b)) => cmp_i64_f64(b, a).reverse(), + (SortNum::Float(a), SortNum::Float(b)) => match (a.is_nan(), b.is_nan()) { + (true, true) => std::cmp::Ordering::Equal, + (true, false) => std::cmp::Ordering::Greater, + (false, true) => std::cmp::Ordering::Less, + (false, false) => a.partial_cmp(&b).unwrap_or(std::cmp::Ordering::Equal), + }, + } +} + +fn cmp_i64_f64(i: i64, f: f64) -> std::cmp::Ordering { + use std::cmp::Ordering::*; + const TWO_POW_63: f64 = 9_223_372_036_854_775_808.0; + if f.is_nan() || f >= TWO_POW_63 { + return Less; + } + if f < -TWO_POW_63 { + return Greater; + } + // In range, so the truncation is exact. + let t = f.trunc(); + match i.cmp(&(t as i64)) { + Equal => 0.0f64.partial_cmp(&(f - t)).unwrap_or(Equal), + o => o, + } +} + /// The integer `s` spells in canonical form (no sign prefix, padding or spaces). fn canonical_int(s: &str) -> Option { s.parse::().ok().filter(|n| n.to_string() == s) @@ -138,3 +209,98 @@ impl Row { .and_then(|i| self.values.get(i)) } } + +#[cfg(test)] +mod tests { + use super::Value; + use std::cmp::Ordering::*; + + fn s(x: &str) -> Value { + Value::Str(x.to_string()) + } + + #[test] + fn sort_cmp_orders_integers_and_canonical_strings_numerically() { + assert_eq!(Value::Int(10).sort_cmp(&s("2")), Greater); + assert_eq!(s("2").sort_cmp(&Value::Int(10)), Less); + assert_eq!(Value::Int(2).sort_cmp(&s("2")), Equal); + assert_eq!(Value::Float(2.5).sort_cmp(&s("10")), Less); + // Non-canonical and non-numeric strings stay above all numbers. + assert_eq!(Value::Int(10).sort_cmp(&s("02")), Less); + assert_eq!(s("02").sort_cmp(&s("abc")), Less); + } + + #[test] + fn sort_cmp_is_exact_beyond_f64_precision() { + let a = Value::Int(9_007_199_254_740_992); + let b = Value::Int(9_007_199_254_740_993); + let f = Value::Float(9_007_199_254_740_992.0); + assert_eq!(a.sort_cmp(&f), Equal); + assert_eq!(b.sort_cmp(&f), Greater); + assert_eq!(f.sort_cmp(&b), Less); + assert_eq!(a.sort_cmp(&b), Less); + assert_eq!( + Value::Int(i64::MAX).sort_cmp(&Value::Float(2f64.powi(63))), + Less + ); + assert_eq!( + Value::Int(i64::MIN).sort_cmp(&Value::Float(-(2f64.powi(63)))), + Equal + ); + assert_eq!(Value::Int(0).sort_cmp(&Value::Float(0.5)), Less); + assert_eq!(Value::Int(-1).sort_cmp(&Value::Float(-0.5)), Less); + } + + #[test] + fn sort_cmp_handles_nan_and_infinities() { + let nan = Value::Float(f64::NAN); + let inf = Value::Float(f64::INFINITY); + assert_eq!(nan.sort_cmp(&inf), Greater); + assert_eq!(nan.sort_cmp(&Value::Int(i64::MAX)), Greater); + assert_eq!(Value::Int(i64::MAX).sort_cmp(&inf), Less); + assert_eq!( + Value::Int(i64::MIN).sort_cmp(&Value::Float(f64::NEG_INFINITY)), + Greater + ); + assert_eq!(nan.sort_cmp(&Value::Float(f64::NAN)), Equal); + assert_eq!(nan.sort_cmp(&s("x")), Less); + } + + #[test] + fn sort_cmp_is_transitive_over_mixed_values() { + let vals = vec![ + Value::Null, + Value::Bool(false), + Value::Bool(true), + Value::Int(-3), + Value::Int(2), + Value::Int(10), + Value::Int(9_007_199_254_740_992), + Value::Int(9_007_199_254_740_993), + Value::Int(i64::MAX), + Value::Int(i64::MIN), + Value::Float(-0.0), + Value::Float(2.0), + Value::Float(2.5), + Value::Float(9_007_199_254_740_992.0), + Value::Float(f64::INFINITY), + Value::Float(f64::NEG_INFINITY), + Value::Float(f64::NAN), + s("2"), + s("10"), + s("02"), + s("abc"), + s(""), + ]; + for a in &vals { + for b in &vals { + assert_eq!(a.sort_cmp(b), b.sort_cmp(a).reverse(), "{a:?} {b:?}"); + for c in &vals { + if a.sort_cmp(b).is_le() && b.sort_cmp(c).is_le() { + assert!(a.sort_cmp(c).is_le(), "{a:?} {b:?} {c:?}"); + } + } + } + } + } +} diff --git a/src/store.rs b/src/store.rs index 9fa63d5..11eb019 100644 --- a/src/store.rs +++ b/src/store.rs @@ -270,6 +270,14 @@ impl Default for DocumentStore { /// Hash of file/document content used for change detection (not /// cryptographic: this only needs to detect "did this file change since /// last index", not resist adversarial collisions). +/// The directory holding `path`; `.` for a bare file name, whose `parent()` +/// is the empty path rather than `None`. +fn containing_dir(path: &Path) -> &Path { + path.parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")) +} + fn hash_bytes(bytes: &[u8]) -> u64 { let mut hasher = std::collections::hash_map::DefaultHasher::new(); bytes.hash(&mut hasher); @@ -1159,8 +1167,7 @@ impl DocumentStore { std::fs::rename(&tmp_path, path)?; // A rename is only power-loss durable after the containing directory // is synced as well. - let parent = path.parent().unwrap_or_else(|| Path::new(".")); - std::fs::File::open(parent)?.sync_all()?; + std::fs::File::open(containing_dir(path))?.sync_all()?; drop(save_lock); Ok(temp_storage) } @@ -1876,3 +1883,22 @@ mod tokenizer_tests { assert_eq!(loaded.tokenizer(), TokenizerKind::Word); } } + +#[cfg(test)] +mod containing_dir_tests { + use super::containing_dir; + use rstest::rstest; + use std::path::Path; + + #[rstest] + #[case("store.mq-db", ".")] + #[case("./store.mq-db", ".")] + #[case("dir/store.mq-db", "dir")] + #[case("/abs/store.mq-db", "/abs")] + fn bare_file_names_resolve_to_the_current_directory( + #[case] path: &str, + #[case] expected: &str, + ) { + assert_eq!(containing_dir(Path::new(path)), Path::new(expected)); + } +}