perf(fullmap): upgrade redb 2→4, open fullmap readers with shared locks

CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+### Breaking Changes
+- **Fullmap databases built by older releases must be rebuilt.** The Rust extension upgraded its embedded database engine from redb 2.6 to redb 4.1, and redb ≥ 3 dropped the old v2 file format. Existing `fullmap.redb` (and sibling `fullmap.s*.redb`) files fail to open with `fullmap DB is outdated or needs repair; rebuild with 'tablassert build-fullmap'`. Run `tablassert build-fullmap` once after upgrading. BABEL downloads stay cached, but the command rebuilds the fullmap files. The on-disk fullmap schema is now `tablassert.fullmap.v5` (the table layout is unchanged; the bump makes the redb-4 rebuild explicit and lets an older extension reject new files loudly).
+
+### Changed
+- **Fullmap reads no longer serialize across processes.** The lookup path (`lookup_fullmap_terms` and the `hydrate_*` helpers) now opens the fullmap redb files READ-ONLY with a SHARED file lock (redb ≥ 3 `ReadOnlyDatabase`) instead of an exclusive lock: concurrent readers — the agent supervisor, its code-executor subprocesses, and parallel `agent run` processes — no longer contend on the fullmap lock ("Database already open"); only a running `build-fullmap` rebuild can briefly block readers. Read-only opens also never touch the file mtime, making the mtime-keyed Python lookup caches fully stable. The redb 4.1 upgrade additionally speeds up multi-threaded shard reads (~15% on upstream benchmarks) and the fullmap build's redb write phase (~1.5x on upstream write benchmarks).
+
 ## 8.1.0 - 2026-08-03
 
 ### Breaking Changes

docs/fullmap.md

@@ -109,7 +109,7 @@ they hold six tables (see `rust/src/fullmap.rs`):
 | `categories` | Compact `u16` id → Biolink category string (primary file) |
 | `sources` | Compact `u8` id → source metadata (name/version) (primary file) |
 | `curies` | Compact `u32` id → CURIE record (CURIE, preferred name, category, taxon, source) (primary file) |
-| `meta` | Schema version tag (`tablassert.fullmap.v4`), the shard count (`shards`), and the BABEL `source_version` used to build the file (primary file) |
+| `meta` | Schema version tag (`tablassert.fullmap.v5`), the shard count (`shards`), and the BABEL `source_version` used to build the file (primary file) |
 
 The shard files must remain alongside the primary file — lookups discover them as siblings of the
 resolved primary path.
@@ -118,8 +118,15 @@ Lookups (`lookup_fullmap_terms`) check the primary's `meta` schema tag before re
 `shards` count to open exactly that many shard files, and fan the query terms out across the shards in
 parallel (releasing the GIL, one reader per non-empty shard, re-merged into input order); a mismatched or
 missing tag raises rather than silently reading incompatible data. Databases built under the older
-`v1`/`v2`/`v3` schemas are rejected — there is no automatic schema migration, so a schema bump (including
-the v3→v4 move to sharded files) requires rebuilding via `tablassert build-fullmap`.
+`v1`/`v2`/`v3`/`v4` schemas are rejected — there is no automatic schema migration, so a schema bump
+(including the v3→v4 move to sharded files and the v4→v5 move to the redb 4 engine) requires rebuilding
+via `tablassert build-fullmap`.
+
+Readers open every fullmap file READ-ONLY with a SHARED file lock (redb ≥ 3 `ReadOnlyDatabase`), so any
+number of processes can run lookups against the same fullmap concurrently; only a `build-fullmap` rebuild
+(an exclusive-lock writer) briefly blocks readers. Each lookup pins one primary-plus-shards file
+generation — cached handles are validated against the file's `(dev, ino)` on every use — so a reader
+follows a rebuild on the next lookup.
 
 ## Usage in Graph Config
 

rust/Cargo.toml

@@ -18,7 +18,7 @@ memmap2 = "0.9"
 mimalloc = { version = "0.1", default-features = false }
 pyo3 = "0.29"
 rayon = "1"
-redb = "2"
+redb = "4"
 rlimit = "0.10"
 rustc-hash = "1"
 serde = { version = "1", features = ["derive"] }
@@ -29,7 +29,7 @@ xxhash-rust = { version = "0.8", features = ["xxh64", "xxh3"] }
 [dev-dependencies]
 bincode = "1"
 flate2 = "1"
-redb = "2"
+redb = "4"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 tempfile = "3"

rust/examples/count_tables.rs

@@ -1,5 +1,6 @@
 //! Count rows in each table of one or more fullmap redb files.
 //! Usage: cargo run --release --example count_tables -- <db1> [db2 ...]
+use redb::ReadableDatabase;
 use redb::ReadableTableMetadata;
 use redb::TableDefinition;
 
@@ -11,7 +12,7 @@ const CURIES: TableDefinition<u32, &[u8]> = TableDefinition::new("curies");
 
 fn main() {
     for arg in std::env::args().skip(1) {
-        let db = redb::Database::open(&arg).expect("open db");
+        let db = redb::ReadOnlyDatabase::open(&arg).expect("open db");
         let read = db.begin_read().expect("begin read");
         let records = read
             .open_table(RECORDS)