[upstream issue Ephemeral-AI-Lab#510] Ephemeral-AI-Lab#506 Live view: browse a Store in S3 without loading it

• Upstream: #506 Live view: browse a Store in S3 without loading it Ephemeral-AI-Lab/layerfs#510
• Type: issue
• State: OPEN
• Updated: 2026-10-30
• Upstream labels: -

Original body (excerpt)

Sub-issue of Ephemeral-AI-Lab#506, building on Ephemeral-AI-Lab#509. Owner direction 2026-10-30: a SQLite Store saved in S3 (MinIO, Cloudflare R2) must be viewable without loading it: its folder tree, history and file contents, read-only. Downloading a temporary copy first is rejected, because it does not work for large Stores.

Spec: core/docs/issues/506/live-view.md on branch issue506-store-backends-remote (PR Ephemeral-AI-Lab#507). The Connect part lives in Connect's own repository (docs/ui/storage-links.md §8).

Why it is feasible (diagnostic, one sample, r2-examples-clone, 251 MB)

Ranged requests needed at 64 KiB blocks, adjacent blocks merged:

Parts

• A. Push layout: the push snapshot uses VACUUM INTO, so each table is contiguous. This is blocked on a rowid audit: VACUUM may renumber rows of tables without an explicit integer key.
• B. Remote reader:
  • A read-only, immutable SQLite VFS (layerfs-remote) through rusqlite::ffi, with no new dependency.
  • 64 KiB blocks, merged ranged GETs, and a bounded LRU cache.
  • Every GET carries If-Match: <ETag>; a 412 becomes Changed.
  • Pieces are mapped through the manifest. Each request is one attempt, with no retry.
• C. Store path remote:<connection>/<file>: every read works and every write refuses ReadOnly. It is not registered or linked.
• D. Runtime calls: remote.open_view, reads with { view }, close_view and view_stats.
• E. Connect: a Browse button on remote files and linked Stores opens read-only History, Files and Timeline, with a banner and Reopen when the file changed.

Checklist

• A: rowid audit recorded here; snapshot via VACUUM INTO if the audit allows it
• B: remote VFS, block cache, ETag pinning, pieces, counters
• C and D: remote store path and runtime calls; README §6 updated
• E: Connect Browse, read-only project pages, banner, Changed and Reopen
• Verification: unit tests; native comparison against the same Store opened locally (lists, heads, Commits, a directory, file bytes); Changed when a push lands mid-view; end to end on MinIO and R2
• Measurement (agent policy, cold cache, one sample): push time and size, backup API vs VACUUM INTO; first History view and a folder listing through a view (wall time, requests, bytes)

Acceptance

For a Store pushed with A:

• The first History view takes at most 5 ranged GETs.
• A folder takes at most 4 per directory level.
• On r2-examples-clone, History plus one folder plus one small file fetches under 10% of the file.
• Every read is identical to the same Store opened locally.

Sync note

这是只读镜像，不是上游仓库的替代。上游 issue/PR 仍是唯一真源；本镜像不复制评论、不复制代码分支。