fix(lancedb): detect schema type drift and add cascade rebuild recovery

[gloryfromca]

Problem — the root cause behind #337

verify_business_schemas (the startup guard) only compared column names against the current schema. A column whose on-disk Arrow type had drifted — name unchanged — sailed through and detonated much later, deep inside merge_insert, as the notoriously opaque:

RuntimeError: lance error: LanceError(IO): Execution error: Spill has sent an error, poll.rs:291:44

This is exactly #337: an episode.subject_vector column left as string (or null) by an older build, while the current schema declares a 1024-d fixed_size_list. Writing a real vector into that column is a pyarrow type error that lance wraps as the "Spill" IO error.

Verified

• Reproduced byte-identically: a drifted subject_vector (string and null) column + a real 1024-d vector through the real LanceRepoBase.upsert() path yields the exact cascade worker fails to index episodes: LanceDB Spill has sent an error (IO) #337 error; a correctly-typed table accepts the same records.
• No lancedb version produces the bad column (checked 0.13→0.34: ≤0.19 raise TypeError, 0.20+ render the correct fixed_size_list), and lance never silently adds a mistyped column — so the drift originates from a build that declared the field differently, and the guard is what should have caught it at startup.

Changes

1. Type-aware verify_business_schemas — compare each shared column's Arrow type against schema.to_arrow_schema() (the exact schema get_table builds tables from, so a healthy table never false-positives). On drift the startup error now names the drifted column and both types instead of the runtime Spill.

2. everos cascade rebuild — the safe recovery from a drifted/corrupt index:

• drops the business LanceDB tables and re-indexes from markdown (the source of truth);
• skips the verify guard (whose whole point is the drift it would otherwise trip on at startup — chicken-and-egg);
• unlike rm -rf ~/.everos/.index/lancedb, re-populates already-done entries (reset_all clears the cascade queue so every md file re-enqueues) — a bare rm of the lancedb dir leaves the queue marked done and the index comes back empty;
• unlike rm -rf ~/.everos/.index, preserves unprocessed_buffer (messages received but not yet extracted — not rebuildable from md).

Tests

• test_verify_schemas.py — healthy tables pass (false-positive guard); subject_vector as string/null raises with type_drift; missing column raises (unchanged); drop_business_tables drops + recreates the correct vector type. The type-drift tests were confirmed to fail without the fix (old guard DID NOT RAISE).
• test_md_change_state.py — reset_all clears every row regardless of status; noop on empty.
• test_cascade_cli_integration.py — cascade rebuild runs despite a drift that trips a normal command's startup verify, recreates the table with the correct type, and re-indexes md from scratch.

Full unit suite + make lint (ruff, import-linter, datetime, openapi) green.

Fixes #337.

[arelchan]

在 1.1.3 上独立复现,确认这个 PR 的诊断和恢复路径都对。补几个数据点,顺便催一下合并。

1.1.3 仍复现,且已发布版本都还没带上这个修复

• everos 1.1.3 + lancedb 0.33.0(0.34.0 也一样)/ pyarrow 25.0.0 / Python 3.12.13 / macOS arm64
• 表结构确认:episode.subject_vector 磁盘上是 string,而 Episode.to_arrow_schema() 要求 fixed_size_list<float>[1024] —— 与 cascade worker fails to index episodes: LanceDB Spill has sent an error (IO) #337 结论一致
• 现存 19 行的 subject_vector 全部为 null,符合"该列在一直为空时被建成 string"的推断
• 最新 release 1.2.0(7-24)不含本 PR(仍 OPEN),所以现在踩到的人依旧只能看到那个不可读的 Spill has sent an error

没有本 PR 的排查成本(正是它要消灭的)

在不知道 #337 的情况下,从症状定位到根因需要逐一排除:磁盘空间(剩余 237G)、文件描述符(116/1048576)、TMPDIR 可写性、双实例抢锁、everos 版本(1.1.3 与 1.2.0.dev1 均失败)、lancedb 版本(0.33.0 与 0.34.0 均失败)、数据损坏(读取/扫描/向量检索全正常)、批量大小(10/50/200 全部成功)。

最具误导性的一点:手动对同一张表做 merge_insert 是成功的(因为复制的老行 subject_vector 为 null),只有写入带真实 subject 向量的新行才炸。这让人很容易误判成"表是好的、是并发/资源问题"。

本 PR 的启动期类型比对能把这一整段直接变成一条明确报错,价值很大。

手动恢复已验证可行(给还没等到合并的人)

在 1.1.3 上按本 PR 的思路手动执行,结果正常:

1. 停服务,备份 .index
2. 按 Episode.to_arrow_schema() 重建 episode 表,把原有行拷回(subject_vector 全为 null,无损)
3. 清掉卡住的 md_change_state(status in ('processing','failed'))
4. 重启

结果:episode 19 行 -> 88 行(补回了积压 18 天的记录),其中 69 行带 subject_vector,cascade_worker_unrecoverable 从 330 次 -> 0 次,无数据丢失。

这正是本 PR 的 cascade rebuild 要自动化的流程,可以佐证该恢复路径是有效的。

两个本 PR 未覆盖的点

不影响本 PR 合并,单独说明(另开 issue 跟踪):

1. 写入侧 API 仍会谎报成功:cascade worker 处于不可恢复失败时,POST /api/v1/memory/add 返回 200、POST /api/v1/memory/flush 返回 {"status": "extracted"},但实际什么都没落库。本 PR 通过"启动期拒绝启动"堵住了 schema 漂移这一类,但其他原因导致的写入失败仍会被静默吞掉,宿主应用无从察觉。
2. 重试没有上限:单次运行观察到 330 条完全相同的 cascade_worker_unrecoverable kind=episode,只带来噪声和负载。

能否优先合并?目前每个新装/老库用户遇到这个问题,拿到的仍是那条看不懂的 IO 错误。

[gloryfromca]

Folded into #379 (consolidated LanceDB hardening PR, rebased on latest main + validated by the storage soak). Commit preserved there.