Skip to content

feat: partial and incremental scalar index builds via fragment_ids - #72

Merged
everySympathy merged 1 commit into
daft-engine:mainfrom
everySympathy:feat/fragment-partial-indexing
Sep 20, 2026
Merged

everySympathy merged 1 commit into
daft-engine:mainfrom
everySympathy:feat/fragment-partial-indexing

Conversation

@everySympathy

@everySympathy everySympathy commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Part of #27 (distributed segment-index creation tracking). Builds on #73 (distributed-only scalar indexing).

What

create_scalar_index(..., fragment_ids=[...]) schedules only the listed fragments, enabling two workflows on top of the single distributed segment path:

  • Partial first builds: commit an index that covers a subset of fragments. Partial indexes are already query-correct — uncovered fragments fall back to scans.
  • Incremental backfill: re-invoking with the same index name skips fragments already covered by committed segments, builds only the remainder, and appends the new segments atomically via commit_existing_index_segments — non-overlapping segments are appended, so committed segments for other fragments are preserved untouched.

This is the building block the support matrix in #27 needs for keeping indexes fresh on append-only tables without full rebuilds.

Semantics

Validation (driver-owned):

  • empty list / unknown fragment IDs → ValueError (available IDs listed)
  • duplicate IDs → de-duplicated with a logged warning (asserted via caplog)
  • indexing an empty dataset → ValueError (no silent no-op, matching lance-ray)
  • replace=False without fragment_ids on an existing name → ValueError. This is the one same-name check the driver keeps: replace is a daft-lance parameter and no downstream API knows about it — without the check, a replace=False call would silently rebuild-and-swap the index, violating the parameter's contract.

Column/type compatibility (Lance-owned, not duplicated here):

  • a different column under an existing name is rejected by Lance's build API ("already exists with different fields");
  • appending segments of a different type is rejected by Lance's commit API ("cannot change index ... with partial fragment coverage");
  • a full rebuild without fragment_ids may change the index type via the atomic overlap swap.

An earlier revision of this PR carried driver-side column/type pre-checks. They duplicated Lance's validation in string space and caused two verified regressions — a false rejection of LABEL_LIST (pylance reports LabelList) and a blockade of type-changing full rebuilds — so they were removed entirely, matching lance-ray and lance-spark, which never pre-check. The principle: check what you own (our parameters), don't copy what Lance already enforces at the source of truth.

Other behavior:

  • replace governs replacement only; backfill appends and never replaces, so the flag does not apply on the fragment_ids path (documented, pinned by test)
  • a fully covered request is a no-op that leaves the dataset version unchanged
  • generated index names follow pylance's convention (<column>_idx, previously <column>_<type>_idx), so mixing daft-lance and pylance on the same table no longer creates duplicate indexes on one column

Safety

A pre-commit guard validates three invariants (mirroring lance-spark's validateDisjointCoverage), failing loudly instead of committing a broken index:

  • no dead fragments — segments referencing fragments rewritten by a concurrent compaction (Lance would still accept the commit; verified experimentally);
  • no overlapping coverage — every fragment covered at most once;
  • complete coverage — the union of built segments equals the scheduled fragment set, so a silently lost worker result cannot produce a partial index.

Equivalence

Partial-then-backfill builds are verified query-equivalent to one-shot full builds for INVERTED, BTREE, BITMAP, and LABEL_LIST, including multi-segment backfills (fragment_group_size=1). The intermediate partial state (half-covered index) is also query-verified, including predicates that hit only uncovered fragments and zero-hit predicates.

Tests

21 tests in tests/io/lance/test_lance_partial_fragment_index.py: per-type equivalence, backfill after appends, validation semantics, Lance-side rejection messages for column/type mismatches, the replace interaction, no-op on full coverage, multi-segment backfills, the duplicate-ID warning, and the three-invariant commit guard (including that it runs before the commit). Full suite passes on pylance 11.0.0 (413 passed); mypy exactly at the pre-PR baseline.

@everySympathy
everySympathy force-pushed the feat/fragment-partial-indexing branch 4 times, most recently from 892afe4 to 2a94da9 Compare September 20, 2026 12:00
create_scalar_index(..., fragment_ids=[...]) schedules only the listed
fragments:

- Partial first builds commit an index that covers just those fragments;
  partial indexes are already query-correct (uncovered fragments scan).
- Re-invoking with the same index name backfills: fragments already
  covered by committed segments are skipped and the remainder is appended
  atomically (non-overlapping segments are appended, so untouched
  committed segments are preserved). Verified query-equivalent to a
  one-shot full build (INVERTED, BTREE, BITMAP, LABEL_LIST), including
  multi-segment backfills (fragment_group_size=1).
- Deterministic validation: empty list and unknown fragment IDs raise
  ValueError (available IDs listed); duplicates are de-duplicated with a
  logged warning; indexing an empty dataset raises ValueError instead of
  silently doing nothing (matching lance-ray).
- Column/type compatibility of a same-name index is validated by Lance's
  own APIs, not duplicated here: the build API rejects a different column
  and the commit API rejects appending a different type. A string-space
  pre-check previously caused false rejections (pylance reports
  'LabelList' for LABEL_LIST), so no such check is kept — matching both
  lance-ray and lance-spark, which never pre-check either.
- replace governs replacement only: replace=False rejects an existing
  name on the full-build path; backfill appends and never replaces, so
  the flag does not apply (documented in README and docstrings).
- A fully covered request is a no-op that leaves the dataset version
  unchanged.
- Generated index names follow pylance's convention ('<column>_idx',
  previously '<column>_<type>_idx'), so mixing daft-lance and pylance no
  longer creates duplicate indexes on one column.
- The pre-commit guard validates three invariants (mirroring
  lance-spark's validateDisjointCoverage): no dead fragments (concurrent
  compaction), no overlapping coverage, and complete coverage of every
  scheduled fragment — a lost worker result can no longer produce a
  partial index.
- _existing_index_coverage keeps the legacy list_indices fallback so
  same-name checks still see indexes on manifests whose details cannot
  be described.
@everySympathy
everySympathy force-pushed the feat/fragment-partial-indexing branch from 2a94da9 to 0285371 Compare September 20, 2026 12:52
@everySympathy
everySympathy merged commit aabd636 into daft-engine:main Sep 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant