Support distributed validation of selected Lance fragments

[hello-peter-tang]

Problem and business value

Large ingestion and column-enrichment pipelines need to check the fragments they touched and identify failures without repeatedly validating the entire table. Operators also need bounded concurrency and a report identifying the affected snapshot and fragment.

PyLance already provides dataset validation, and users can compose fragment validation with a custom Daft UDF. The missing feature is a supported, read-only daft-lance operation that distributes selected fragment checks and reports their results consistently.

Existing support and upstream primitive

At main 1b1fa3c, daft-lance exports no validation operation and contains no fragment .validate() calls. Its distributed compaction implementation and DatasetOpenContext already provide the planning, worker-reopen, and snapshot-pinning pattern needed here.

Lance v11 exposes LanceFragment.validate() in Python:
lance-format/lance#8428

This checks fragment-level file/metadata consistency, including deletion-vector consistency. It does not repair data, establish application-level correctness, or replace every dataset-wide validation check.

Proposed scope

• Accept a URI or namespace-addressed table and an optional set of fragment IDs. Resolve one dataset version on the driver.
• Enumerate fragment metadata, not a row scan, so empty/fully deleted fragments are not accidentally omitted.
• Reopen the pinned dataset on workers using the existing serializable context; call the native fragment validator with bounded concurrency.
• Produce one report row per selected fragment, including the resolved version, fragment ID, and validation outcome. Keep table-open/authentication failures distinguishable from detected corruption; never silently skip missing fragments.
• Remain read-only: no commits, automatic repairs, or unconditional validation on every write.

The public API name/report shape can be agreed during review. This should reuse Lance's validator, not implement a second one. Document the PyLance version requirement; the current dependency floor still permits v8.

Acceptance criteria

• Healthy multi-fragment datasets, fragments with multiple column files, and deletion-vector cases pass.
• A missing data file or inconsistent fixture is attributed to the correct fragment using fresh worker opens.
• Selecting a subset validates only that subset; empty selections and unknown IDs have explicit behavior.
• Local and distributed execution cover pinned snapshots, namespace reopening, bounded concurrency, and an unchanged dataset version after validation.
• A concurrent append after planning does not change the validation snapshot.

[hello-peter-tang]

To clarify, this is an integration gap, not a claim that Lance lacks validation. Lance v11 exposes Python fragment validation, but daft-lance has no supported operation for distributing selected-fragment checks and collecting results. Users must currently assemble that workflow themselves.

The practical value is a read-only health check after ingestion or column enrichment: validate the affected fragments, identify file/metadata inconsistencies before downstream jobs fail, and report which fragments need attention without repeatedly validating the whole table. This does not validate business content or repair corruption.

[hello-peter-tang]

PyLance version requirement

The proposed implementation using Python LanceFragment.validate() requires pylance>=11.0.0. The current pylance>=8.0.0 dependency floor does not guarantee this API is available; the implementation should explicitly enforce/document the new requirement and test the minimum supported version.

Both driver and worker environments must satisfy the requirement. This refers to the installed PyLance library (import lance), not dataset snapshot version=11 or a requirement to rewrite existing data with v11. Older datasets can be checked when readable by the supported library. Upgrading PyLance alone does not add the proposed daft-lance integration.