refactor(storage): separate tool bodies and compact semantic directories

Cargo.toml

@@ -92,6 +92,7 @@ publish = false
 [workspace.dependencies]
 mxc-sdk = { git = "https://github.com/microsoft/mxc", rev = "05a430bc17a3e1b3fa2131524fc52e4319e07298" }
 flatbuffers = "25.12.19"
+postcard = { version = "1.1.3", default-features = false, features = ["alloc"] }
 maka-python-runtime = { path = "platform/python-runtime" }
 maka-mcp = { path = "plugins/agent/mcp" }
 hex = "0.4.3"

platform/event-log/Cargo.toml

@@ -25,6 +25,7 @@ autotests = false
 
 [dependencies]
 hex.workspace = true
+postcard.workspace = true
 maka-agent-facts.workspace = true
 maka-packages.workspace = true
 maka-agent-ports.workspace = true

platform/event-log/README.md

@@ -69,7 +69,17 @@ New runtime events use UUIDv7 identities to keep event and derived message index
 
 Canonical runtime-event ingress inserts each event, captures its sequence, then maintains safety, execution, source and semantic projections as separate statements on the same SQLite worker and writer transaction. The shared insert statement derives indexed accounting and owner fields from those same JSON bytes; collection preserves their published values when removing retired bodies. Typed facts select the safety and execution statements to run. The next event cannot begin before those projections finish. Projection storage failures roll back the entire batch, including imported events; notifications remain tied to successful commit. Copied-prefix projections are maintained by their current copy paths.
 
-Canonical databases use UTF-8. Semantic fragments for successful tool results reference checked byte spans in their original event JSON. Resolution reads and decodes the selected projection on the original SQLite worker within the caller's read snapshot; the source collector retains it for live history owners. Derived archive placeholders and other fragment bodies keep their chunked storage.
+Canonical databases use UTF-8. Successful tool results store their original `model_projection` token separately: `event_header` preserves every surrounding byte with a literal `null` placeholder, and the event row records the original byte offset and immutable body reference. Reconstructing the header and original token reproduces the exact canonical event, including imported whitespace, number spelling and Unicode escapes. `canonical_bytes` supplies logical byte budgets; encoded size never substitutes for admitted canonical size. Other event kinds remain inline.
+
+Bodies are directly seekable raw bytes, shared by SHA-256 of their original token with exact byte comparison on a repeated identity. Reads validate logical lengths and selected ranges. Framed compression was evaluated against raw storage and rejected because it repeatedly slowed supported window and sequential reads; it is not a persisted format. Space savings come from compact directories and sharing identical projection tokens, not from compressing unique bodies.
+
+Complete-event JSON queries join the referenced body into the selected SQL row within the original snapshot and reconstruct the original bytes. Typed prefix readers order headers and body references, keeping large payloads out of SQLite's sort records. One prepared primary-key lookup resolves each referenced body on the same connection and transaction; inline events require no body lookup. Readers decode the borrowed header and original projection bytes directly on the original SQLite worker. They avoid copying the body into a temporary string and finish decoding before advancing either row. The existing event/byte preflight and transaction cover the complete result; no SQLite borrow or unfinished decoder leaves that worker. Typed headers reuse fact field types and defaults, ignore unknown metadata, and verify that the projection placeholder is exactly at its stored offset. Root evidence hashes the three original byte slices as one length-prefixed JSON field. Metadata predicates use the header and retain their indexes. Semantic spans retain original canonical coordinates; tool spans translate those logical coordinates into offsets within their immutable body, while inline model parts retain physical SQLite BLOB bounds. Resolution stays on the original SQLite worker. Hot range reads use its existing prepared-statement cache. Body references follow the existing history-owner collection proof; clearing or deleting the final reference removes its body in the same transaction. The event row publishes the header, logical length, offset and reference atomically; live references cannot be redirected. Neither a body digest nor a derived directory grants access or extends lifetime. Derived archive placeholders and other fragment bodies keep their chunked storage.
+
+Semantic row, payload and child directories are finite derived descriptors encoded with Postcard. Writers encode them once in the publication transaction. Pages copy their stored bytes alongside fixed little-endian locator fields into the caller's charged allocation; consumers decode the original typed descriptors without navigating whole bodies. Original byte positions and leaf commitments remain unchanged. Directory encoding never supplies a canonical fact or source identity. The SQL migration converts former JSON descriptors atomically; its conversion functions exist only during migration.
+
+`event_storage_bench` exercises real Root writes, captured directories, child pages, first/tail and repeated 64-byte windows, sequential bodies, event prefixes and bundle export. Build the same example in release against both revisions and retain each binary, then run `python3 scripts/bench-event-storage.py --output /tmp/storage-comparison original=/path/to/original candidate=/path/to/candidate`. It alternates at least five runs and validates event, descriptor, body and export digests. The first read pass validates and warms caches; reopened measurements are OS-cache-warm. Raw distributions, process CPU/RSS/I/O availability, SQLite cache counters and separate main/WAL/SHM file sizes are recorded. Live prepared-statement snapshots do not represent total SQL work, and JSON/text/content leaf reads and settlement writes are also reported separately. Use `--body-pattern entropy` to include deterministic low-compressibility printable input.
+
+`prefix_read` stops when the prefix API returns; `prefix_all` also includes reserialization and full result digest verification. Keep both when interpreting a difference. `--operation prefix` restricts measured iterations to consecutive prefix reads while retaining the first complete validation/warmup pass and every prefix oracle; use the same option in both binaries. This is an OS-warm focused workload, distinct from the default mixed-operation loop.
 
 A statement failure rolls back the entire group unless it is a domain rejection isolated by its own savepoint. Former ordinary writer requests can continue recovery on the same owner after acknowledged rollback and fresh authority validation; pure observation batches retain their stricter retirement policy. Failed commits always return unknown outcomes without execution claims. Reusing a former ordinary writer group after a failed commit additionally requires acknowledged cleanup and zero transaction depth. Savepoint, authority and unconfirmed cleanup failures retire the writer. Single-key and paged plugin-data reads use the independent reader owner and see committed snapshots; dependent reads must still await their preceding write receipt.
 

platform/event-log/README.zh-CN.md

@@ -63,7 +63,17 @@ Control 中的调用意图、项目登记、workspace attachment、Session 路
 
 规范运行时事件入口逐条插入事件并取得序号，再在同一 SQLite worker、同一写事务内，用独立语句维护安全、执行、来源和语义投影。用于索引的记账和所有者字段由统一插入语句从同一 JSON 计算，正文回收后保留其发布值。安全与执行维护由类型化事实选择相应语句。当前事件投影完成后才开始下一条。投影存储错误使整个批次回滚，导入事件同样如此；通知仍只在提交成功后发布。复制前缀投影由当前复制路径维护。
 
-规范数据库使用 UTF-8。成功工具结果的语义片段引用原始事件 JSON 中经过校验的字节区间。解析在调用者的读取快照内，由原 SQLite worker 读取并解码选中的工具投影；来源收集器为仍有历史所有者的正文保留存活期。归档占位内容和其他片段正文继续使用分块存储。
+规范数据库使用 UTF-8。成功工具结果的原始 `model_projection` token 单独保存：`event_header` 用字面量 `null` 占位，保留周围全部原始字节；事件行记录原字节偏移和不可变正文引用。两者重建出精确的规范事件，包括导入内容的空白、数字拼写和 Unicode 转义。`canonical_bytes` 用于逻辑字节预算，压缩大小不替代已接受的规范大小。其他事件类型仍内联保存。
+
+正文保存为可直接 seek 的原始字节，按原始 token 的 SHA-256 共享，重复身份还需逐字节核对。读取校验逻辑长度与选中区间。对照实验中，分帧压缩稳定拖慢了受支持的窗口和顺序读取，因此未作为持久化格式保留。空间收益来自紧凑目录与相同投影 token 的共享，不能把压缩实验中独有正文的压缩率套到最终实现。
+
+完整事件的 JSON 查询在原快照内的同一 SQL 行 JOIN 引用的正文，并重建原始字节。类型化前缀 reader 只排列 header 和正文引用，避免大正文进入 SQLite 排序记录；随后在同一连接、同一事务内复用一条 prepared 主键查询读取引用的正文，内联事件无需查询正文。Reader 在原 SQLite worker 上直接解码借用的 header 和原始投影字节，避免把正文复制到临时字符串中，并在推进任一行之前完成解码。原有事件／字节预检和事务覆盖完整结果；SQLite 借用和未完成的解码都不会离开该 worker。Header 复用事实字段的类型和默认值、忽略未知元数据，并验证投影占位符恰好位于已存偏移。Root 证据把原始三段字节作为一个带长度前缀的 JSON 字段计算摘要；元数据谓词继续读取 header 并使用原索引。语义区间保留原规范坐标，工具区间把原规范坐标换算为不可变正文内部的偏移，内联 model part 则保留 SQLite BLOB 的物理边界。读取仍由原 SQLite worker 执行。热区间读取复用其已有 statement cache。正文引用的存活由既有历史所有者回收证明决定，最后一个引用清除或删除时，在同一事务删除正文。事件行原子发布 header、逻辑长度、偏移和引用，存活引用不可重定向；正文摘要和派生目录都不授予访问权，也不延长存活期。归档占位内容和其他片段正文继续使用分块存储。
+
+语义行、payload 和子项目录是有限的派生描述符，采用 Postcard 编码。writer 在发布事务内编码一次；分页把已存字节和固定小端序定位字段直接复制到调用者计费的缓冲区，消费端恢复原有类型，无需读取整个正文。原始字节位置与叶子摘要保持不变，目录编码不参与规范事实或来源身份。SQL 迁移在同一事务内转换原 JSON 描述符；转换函数仅在迁移期间存在。
+
+`event_storage_bench` 测量真实 Root 写入、冻结目录、子项分页、首尾及重复 64 字节窗口、顺序正文、事件前缀和 bundle 导出。在两个版本上用 release 构建同一 example 并各自保存二进制后，运行 `python3 scripts/bench-event-storage.py --output /tmp/storage-comparison original=/path/to/original candidate=/path/to/candidate`。脚本交错执行至少五轮，校验事件、目录、正文与导出摘要。首次读取用于校验和预热；重开测量保留操作系统缓存。记录保留原始分布、进程 CPU/RSS/I/O 可用性、SQLite 缓存计数，以及分列的主库、WAL、SHM 大小。存活 prepared statement 的快照不能代表全部 SQL 工作；另外分别记录 JSON／文本／多段内容的叶子读取和结算写入耗时。
+
+`prefix_read` 在前缀 API 返回时停止计时；`prefix_all` 还包含重新序列化及完整结果摘要验证，解释差异时保留两者。`--operation prefix` 仅将计量迭代切换为连续前缀读取，首次完整校验／预热及每次前缀校验仍然执行；两个二进制必须使用相同选项。这是保留操作系统缓存的定向工作负载，与默认混合操作循环分别报告。
 
 语句失败会回滚整组，只有已由自身保存点隔离的领域拒绝例外。原先使用普通写入路径的请求，在确认回滚并重新校验所有权后，可以继续通过同一 owner 恢复；纯 observation 批次保留更严格的退出策略。提交失败始终返回结果未知，不产生执行凭据；此后复用原普通写入组，还要求确认清理完成且事务深度为零。保存点、所有权或无法确认的清理失败都会让写入器退出。单键与分页插件数据读取使用独立 reader，读取已提交的快照；依赖写入的读取仍需先等待相应回执。