Skip to content

Add DuckDB snapshots and local Datalog federation POC - #16

Open
bjacobso wants to merge 2 commits into
mainfrom
shanghai-v5
Open

bjacobso wants to merge 2 commits into
mainfrom
shanghai-v5

Conversation

@bjacobso

@bjacobso bjacobso commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Adds a private Effect service for DuckDB analytical snapshots and federated Datalog queries across isolated local SQLite databases, using unmodified DuckDB.

Local worker processes send visible facts in bounded batches to a DuckDB coordinator; source shortcuts and ordinary database/transaction membership patterns share the existing compiler, with source-qualified identities and independent pinned snapshot cuts.

Includes 13 correctness tests, a runnable two-database demo, the opt-in SQLite comparison benchmark and documented results, and clean-install verification of the packaged workers; S3 access, tenant authorization, and distributed recursive rounds remain deferred.

Validation: pnpm check and pnpm pack:check pass, including 11 package tarballs and the federated consumer smoke test.


Open workspace in Conductor

@bjacobso bjacobso changed the title Add DuckDB analytical snapshot proof of concept Add DuckDB snapshots and local Datalog federation POC Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant