feat: partial and incremental scalar index builds via fragment_ids - #72
Merged
everySympathy merged 1 commit intoSep 20, 2026
Merged
Conversation
everySympathy
force-pushed
the
feat/fragment-partial-indexing
branch
4 times, most recently
from
September 20, 2026 12:00
892afe4 to
2a94da9
Compare
create_scalar_index(..., fragment_ids=[...]) schedules only the listed
fragments:
- Partial first builds commit an index that covers just those fragments;
partial indexes are already query-correct (uncovered fragments scan).
- Re-invoking with the same index name backfills: fragments already
covered by committed segments are skipped and the remainder is appended
atomically (non-overlapping segments are appended, so untouched
committed segments are preserved). Verified query-equivalent to a
one-shot full build (INVERTED, BTREE, BITMAP, LABEL_LIST), including
multi-segment backfills (fragment_group_size=1).
- Deterministic validation: empty list and unknown fragment IDs raise
ValueError (available IDs listed); duplicates are de-duplicated with a
logged warning; indexing an empty dataset raises ValueError instead of
silently doing nothing (matching lance-ray).
- Column/type compatibility of a same-name index is validated by Lance's
own APIs, not duplicated here: the build API rejects a different column
and the commit API rejects appending a different type. A string-space
pre-check previously caused false rejections (pylance reports
'LabelList' for LABEL_LIST), so no such check is kept — matching both
lance-ray and lance-spark, which never pre-check either.
- replace governs replacement only: replace=False rejects an existing
name on the full-build path; backfill appends and never replaces, so
the flag does not apply (documented in README and docstrings).
- A fully covered request is a no-op that leaves the dataset version
unchanged.
- Generated index names follow pylance's convention ('<column>_idx',
previously '<column>_<type>_idx'), so mixing daft-lance and pylance no
longer creates duplicate indexes on one column.
- The pre-commit guard validates three invariants (mirroring
lance-spark's validateDisjointCoverage): no dead fragments (concurrent
compaction), no overlapping coverage, and complete coverage of every
scheduled fragment — a lost worker result can no longer produce a
partial index.
- _existing_index_coverage keeps the legacy list_indices fallback so
same-name checks still see indexes on manifests whose details cannot
be described.
everySympathy
force-pushed
the
feat/fragment-partial-indexing
branch
from
September 20, 2026 12:52
2a94da9 to
0285371
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #27 (distributed segment-index creation tracking). Builds on #73 (distributed-only scalar indexing).
What
create_scalar_index(..., fragment_ids=[...])schedules only the listed fragments, enabling two workflows on top of the single distributed segment path:commit_existing_index_segments— non-overlapping segments are appended, so committed segments for other fragments are preserved untouched.This is the building block the support matrix in #27 needs for keeping indexes fresh on append-only tables without full rebuilds.
Semantics
Validation (driver-owned):
ValueError(available IDs listed)ValueError(no silent no-op, matching lance-ray)replace=Falsewithoutfragment_idson an existing name →ValueError. This is the one same-name check the driver keeps:replaceis a daft-lance parameter and no downstream API knows about it — without the check, areplace=Falsecall would silently rebuild-and-swap the index, violating the parameter's contract.Column/type compatibility (Lance-owned, not duplicated here):
fragment_idsmay change the index type via the atomic overlap swap.An earlier revision of this PR carried driver-side column/type pre-checks. They duplicated Lance's validation in string space and caused two verified regressions — a false rejection of
LABEL_LIST(pylance reportsLabelList) and a blockade of type-changing full rebuilds — so they were removed entirely, matching lance-ray and lance-spark, which never pre-check. The principle: check what you own (our parameters), don't copy what Lance already enforces at the source of truth.Other behavior:
replacegoverns replacement only; backfill appends and never replaces, so the flag does not apply on thefragment_idspath (documented, pinned by test)<column>_idx, previously<column>_<type>_idx), so mixing daft-lance and pylance on the same table no longer creates duplicate indexes on one columnSafety
A pre-commit guard validates three invariants (mirroring lance-spark's
validateDisjointCoverage), failing loudly instead of committing a broken index:Equivalence
Partial-then-backfill builds are verified query-equivalent to one-shot full builds for
INVERTED,BTREE,BITMAP, andLABEL_LIST, including multi-segment backfills (fragment_group_size=1). The intermediate partial state (half-covered index) is also query-verified, including predicates that hit only uncovered fragments and zero-hit predicates.Tests
21 tests in
tests/io/lance/test_lance_partial_fragment_index.py: per-type equivalence, backfill after appends, validation semantics, Lance-side rejection messages for column/type mismatches, thereplaceinteraction, no-op on full coverage, multi-segment backfills, the duplicate-ID warning, and the three-invariant commit guard (including that it runs before the commit). Full suite passes on pylance 11.0.0 (413 passed); mypy exactly at the pre-PR baseline.