feat(java): expose SourceDedupeBehavior in MergeInsertParams - #5
Open
sezruby wants to merge 459 commits into
Open
feat(java): expose SourceDedupeBehavior in MergeInsertParams#5sezruby wants to merge 459 commits into
sezruby wants to merge 459 commits into
Conversation
…ormat#7856) ## Summary `LANCE_CPU_THREADS` and `LANCE_IO_CORE_RESERVATION` were parsed with `.parse().unwrap()`. Two problems followed from a misconfigured value: 1. A non-integer value panicked inside the `LazyLock` initializer with a bare `ParseIntError` that named neither the variable nor the cause. A stray quote or trailing space (common in `.env` files, Docker, and k8s manifests) was enough to trigger it, and the process stayed panicked because the poisoned `LazyLock` re-panics on every subsequent access. 2. `LANCE_CPU_THREADS=0` parsed fine, then flowed into `create_runtime`'s `max_blocking_threads(0)`, which asserts inside tokio (`Max blocking threads cannot be set to 0`) on the first compute task. The panic surfaced deep in tokio, far from the actual misconfiguration, with no hint that the CPU-threads variable was at fault. Both variables are documented as user-tunable in the performance guide, so a hand-set bad value is a realistic, reachable failure — not a corrupt-input edge case. ## Change Route both through a small pure helper, `parse_env_usize(name, raw, min)`, which trims surrounding whitespace, names the offending variable in the error, and rejects values below a per-variable minimum. `LANCE_CPU_THREADS` must be at least 1 (the floor tokio requires); `LANCE_IO_CORE_RESERVATION` keeps allowing 0, meaning reserve no cores for IO. Unset still defaults to 2. Bad values still fail fast — the contract is unchanged — but now with an actionable message instead of a bare `ParseIntError` or an unrelated tokio assertion. The only inputs newly rejected are ones that never worked: `LANCE_CPU_THREADS=0` (previously a guaranteed tokio assertion) and values that already panicked as unparseable. Whitespace-padded values that used to panic now parse, so the change only widens the set of accepted configurations aside from those two. ## Test plan The variables feed process-global `LazyLock`s that read once and are read in parallel by other tests, so the environment cannot be mutated reliably from a test. The pure parser is unit-tested directly instead: - `parses_valid_value_and_trims_surrounding_whitespace` - `rejects_non_integer_naming_the_variable` - `rejects_value_below_minimum` (the `LANCE_CPU_THREADS=0` case) - `allows_zero_when_minimum_is_zero` (the `LANCE_IO_CORE_RESERVATION=0` case) `cargo fmt --all` and `cargo clippy -p lance-core --tests -- -D warnings` are clean.
…e-format#7883) ## Summary `SlotBackoff::next_backoff` doubled the slot count on every attempt with no ceiling, and computed the sleep as `(slot_i * self.unit) as u64` — a `u32 * u32` multiply that widens to `u64` only after the product is formed. Two problems followed: 1. At a high attempt count the `slot_i * unit` product overflows `u32`: a debug-build panic, or in release a wrapped, wrong sleep duration. 2. The slot count, and therefore the backoff, grew without bound as attempts climbed. This is reachable on the default commit-retry path (20 retries). `unit` is set to the first attempt's latency plus 10% (`commit.rs`), so once the first commit takes a few seconds, a deep retry drives `slot_i` into the range where the product overflows. ## Change Cap the slot count at `MAX_SLOTS` (128) and widen the operands to `u64` before multiplying. 128 slots already exceeds any realistic number of concurrent committers, so further doubling only lengthens the wait without reducing collisions. Every backoff is now bounded by `(MAX_SLOTS - 1) * unit`. Capping the slot *count* rather than the resulting duration is deliberate: `SlotBackoff` spreads contending writers across random slots so they don't collide, and a duration clamp would map every high-attempt writer onto the same instant, recreating the thundering herd the type exists to prevent. A fixed 128-slot grid keeps writers uniformly distributed. Attempts 0-4 are unchanged (their slot counts are already below 128); the cap only affects attempt 5 and beyond, where the extra slots bought no throughput anyway (commit throughput is one per `unit` regardless of slot count). ## Follow-up (out of scope) The cap is proportional to `unit`, not absolute, so a slow first attempt can still produce a multi-minute single sleep. The commit-path sleep has no timeout wrapper, unlike the write-retry path which bounds its sleep against a deadline. Bounding wall-clock regardless of `unit` is tracked separately in lance-format#7882. ## Test plan - `test_slot_backoff` (existing) — low-attempt slot distributions, unchanged. - `test_slot_backoff_high_attempt_is_bounded` — every backoff stays within `(MAX_SLOTS - 1) * unit` across many high attempts; fails if the cap is removed. - `test_slot_backoff_large_unit_does_not_overflow` — with `unit = u32::MAX`, asserts a lower bound that a reverted `u32` multiply would violate in release (wraps to a smaller value) as well as debug (panics). - `cargo fmt --all` and `cargo clippy -p lance-core --tests -- -D warnings` are clean.
## Summary Expose existing `LABEL_LIST` segment builds in Pylance. Core already supports building a `LABEL_LIST` index on `LargeList`, but its query parser only accepted `List` literals. A `LargeList` filter therefore fell back to a regular scan instead of producing `ScalarIndexQuery`. This change makes both list types use the same parser path and adds parser and dataset-plan coverage. ## Testing - `cargo test -p lance test_label_list_index_types` - `cargo test -p lance-index test_label_list_query_parser` - `uv run --frozen pytest python/tests/test_scalar_index.py::test_label_list_segment_index` - `uv run --frozen pytest python/tests/test_scalar_index.py -k label_list`
…t#8000) Close lance-format#7967 --------- Co-authored-by: fangbo <fangbo.0511@bytedance.com>
Legacy format paths are retained to preserve compatibility with existing data, but current writers no longer emit them in normal operation. Without an explicit boundary, new feature work can accidentally extend or refactor these paths and increase the risk of breaking historical reads. This establishes legacy code as a frozen compatibility surface. New features should target current format and write paths, while any unavoidable legacy change must stay isolated and be covered by released historical fixtures.
…ormat#7863) ## Context Modern FTS postings use dense per-partition document IDs, while the existing `LazyDocSet` also owns physical row addresses, document lengths, visibility, and final projection. This mixes identity domains with different lifetimes and makes query-time ownership difficult to reason about. This refactor keeps dense `DocId` values through modern WAND scoring and filtering, loads document lengths and address projection independently, and resolves physical row addresses only for the final top-k. Legacy single-file indexes continue through the existing row-address path. We no longer write new legacy indexes, so their observable behavior and on-disk handling remain unchanged, and the `metadata.lance` format is unchanged. The fully prewarmed path additionally publishes query-ready document projections and validated postings, bypasses already-resident async singleflight futures, specializes unfiltered scoring, and skips redundant corpus-stat synchronization. Cold queries retain bounded, concurrent top-k address resolution. ## Performance The benchmark used the same 100M-row S3 corpus, 29 FTS segments, and 10-query top-10 panel on an `r7i.12xlarge` in `us-east-1`. Results matched, and every fully prewarmed measured query reported zero object-store reads and writes. Values are candidate deltas versus `main`; higher QPS and lower latency are better. ### Without explicit index prewarm | Workload | QPS | Mean latency | P50 latency | P95 latency | | --- | ---: | ---: | ---: | ---: | | Sequential | **+61.98%** | **-38.32%** | **-49.71%** | **-25.87%** | | Concurrency 8 | **+108.76%** | **-52.07%** | **-61.73%** | **-59.19%** | Each variant ran twice in reversed order for 5,000 measured queries per workload. ### After `Dataset::prewarm_index` | Workload | QPS | Mean latency | P50 latency | P95 latency | | --- | ---: | ---: | ---: | ---: | | Sequential | **+0.32%** | **-0.22%** | **-0.31%** | **-1.55%** | | Concurrency 8 | **+1.48%** | **-1.25%** | **-1.16%** | **-1.78%** | The standard FTS prewarm API was used with a 192 GiB index cache. Each variant ran twice in reversed order for 20,000 measured queries per workload. Benchmark candidate `a03f98b36604adbdd1720c03256ca25eba8a97dc` is production-code equivalent to PR head `964bfe243e64de2382a5801389f7d5529a527806`; baseline `bb819936ca8090c602164e45907aef5db1c26a56` is based on `main` `aea6ded4822780812703605a88554addb839c9e3`. ## Verification - `cargo test -p lance-index --lib` (903 passed, 0 failed, 2 ignored) - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all -- --check` Benchmark-only source and scripts are kept on separate branches and are not part of this PR.
Closes lance-format#6398. A logical vector index can contain immutable physical segments trained with different IVF and quantizer models. The append path currently validates the entire segment set before selecting work and then tries to encode unindexed fragments as though every segment shared one model. This makes append fail on valid heterogeneous segment sets. This makes append segment-set-native. Append builds one new physical segment over only unindexed fragments using the complete model of the deterministic manifest suffix segment, while preserving all existing segment metadata and fragment coverage. Explicit merge selects its requested suffix before compatibility validation, and steady-state rebalance continues to rewrite only one segment. Explicit retrain remains the operation that source-rebuilds and unifies models. The compatibility guard remains fail-closed and now applies to legacy and V3 storage, including IVF, quantizer/codebook/rotation, metric, dimension, and index-type metadata. This intentionally leaves distributed merge planning in lance-format#7730 out of scope.
…mat#8049) ## Problem Arrow structs carry a parent validity bitmap in addition to child validity. Lance reconstructs separately stored struct fields with `lance_arrow::merge`, where parent validity is combined with OR: either input can make a row valid, and a missing bitmap means all-valid. `merge_struct_validity` first discarded any all-null bitmap to handle all-null placeholders from schema evolution. This conflated an explicit all-null batch with absent validity. When both inputs were all-null — as in a filtered batch containing only a null row, or an `AllNulls`-added column — both bitmaps were dropped and the result became all-valid. The scan then returned a valid struct with null children even though `IS NULL`, which reads on-disk definition levels directly, correctly reported null. ## Fix Merge the original bitmaps without normalization. An all-null bitmap is already the identity for OR, so it preserves filler behavior while allowing two all-null inputs to remain null. All-valid and all-null cases short-circuit to avoid unnecessary allocation. No file-format change is needed; the encoded validity was already correct. Reverting only this function reproduces the failure, so the loss was purely in the in-memory reconstruction. ## Affected read paths Measured with the fix reverted, on storage version 2.1: | Path | before | after | |---|---|---| | filtered scan of a nullable struct | valid struct, null children | null struct | | filtered scan of `List<Struct>` | all rows valid | validity preserved | | `add_columns(AllNulls)` struct column | valid struct, null children | null struct | | full scan, `take`, `count_rows(IS NULL)` | already correct | unchanged | Only the both-inputs-all-null combination changes behavior; every other combination produces exactly what it did before. Storage version 2.0 does not encode struct validity at all, so the test pins 2.1. ## Tests * `merge_struct_validity` combinations, including all-null/all-null and an all-null input acting as the merge identity * filtered scans preserve nullable `Struct` and `List<Struct>` validity, children agree with the parent, and both agree with `IS NULL` * a struct column added with `AllNulls` reads back as null structs ``` cargo nextest run -p lance-arrow cargo nextest run -p lance --lib test_filtered_scan_preserves ``` Each new assertion fails with the fix reverted. Fixes lance-format#7908
…gments (lance-format#7657) ## Summary `build_index_metadata_from_segments` listed each segment's index directory in a serial loop, so committing N segments cost N sequential LIST round trips on the object store. Distributed index builds can produce many segments, making this step latency-bound on remote stores (S3/GCS/Azure). This change runs the per-segment finalize + list work concurrently with `buffered(io_parallelism())`, preserving output order and all validation. ## Performance Micro-benchmark isolating the changed caller pattern: N index directories on an in-memory object store wrapped with `ThrottleConfig` to simulate per-call LIST latency; the *before* (serial loop) and *after* (`buffered(io_parallelism)`) patterns both call the unchanged `list_index_files_with_sizes` helper. `io_parallelism = 8`, 4 files per segment, multi-thread runtime. | segments | LIST latency | before: serial | after: concurrent | speedup | |---------:|-------------:|---------------:|------------------:|--------:| | 8 | 20ms | 175.3 ms | 22.5 ms | 7.78x | | 32 | 20ms | 705.9 ms | 88.2 ms | 8.00x | | 128 | 20ms | 2837.2 ms | 350.1 ms | 8.10x | | 128 | 50ms | 6659.1 ms | 832.5 ms | 8.00x | Serial cost grows linearly with `N × latency`; concurrent cost scales as `ceil(N / io_parallelism) × latency`, so the speedup tracks `io_parallelism` and is independent of the per-call latency. On local/in-process stores where LIST is nearly free the difference is negligible — the win is on high-latency remote stores, which is exactly where distributed index commits run. ## Test plan - [x] `cargo check -p lance --lib` - [x] Micro-benchmark above (throttled in-memory store) — both paths return identical file counts - [ ] Existing index commit tests remain green in CI Co-authored-by: xiaojiebao <xiaojiebao@xiaomi.com> Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
…dotfile-safe reserved marker (lance-format#7940) ## Summary Make `create_table_version` in the directory namespace (`lance-namespace-impls`) idempotent, strictly enforce version CAS, and make the `.lance-reserved` marker creation safe across object stores. ## Problem In the namespace directory provider, `create_table_version` is currently unsafe against the retries and races that happen with real object-store deployments: 1. **Non-idempotent retries.** If a version is successfully created but the client retries (network blip, or the coordinating node renamed staging → final while the caller lost the response), the second attempt sees the version already published and fails with `ConcurrentModification`. Callers cannot distinguish a genuine conflict from a benign retry. 2. **Loose version CAS.** There is no enforced "must be `latest + 1`" check, so a stale or duplicate write can fork the version chain. 3. **Reserved-marker staging breaks on some object stores.** The table is reserved by writing a `.lance-reserved` marker. Some backends implement `PutMode::Create` as a temp-file + rename, producing temporary names containing `..`. Object stores and unified file systems that reject `..` path components then fail the marker write and the table can never be created. ## Changes - **Strict CAS**: `create_table_version` only accepts `version == latest + 1` (`version == 1` on an empty chain); any other value maps to `NamespaceError::ConcurrentModification`. - **Idempotent retry on published versions**: if the requested version already exists with identical manifest content (`e_tag` / `size` / bytes), return success instead of `ConcurrentModification`. - **Content-conflict detection**: if a version with the same number exists but content differs, fail with `ConcurrentModification`. - **Dotfile-safe reserved marker**: stage via a non-dot sibling file then `rename_if_not_exists` (conditional no-replace) onto `.lance-reserved`; fall back to `PutMode::Create` when rename is unsupported. ## Test plan - Added tests for idempotent retry, content conflict, and CAS gap rejection, plus table-not-found and on-branch variants. - `cargo test -p lance-namespace-impls --lib create_table_version` passes. ## Scope Targets the `dir` namespace provider (`create_table_version` / reserved-marker lifecycle). The retry/conflict semantics apply to any object-store backend, not a specific one. Closes lance-format#7939
…e-format#7879) Part 1/12 of lance-format#7877. This is the foundation of the reviewable implementation stack for the final layout demonstrated in lance-format#7979. Unlike lance-format#7979, this PR is intended to merge independently. `LanceFileVersion` currently combines user-facing selectors (`Stable` and `Next`) with exact identities that can be persisted. It also exposes one numeric conversion for wire locations whose compatibility mappings are deliberately different. This PR introduces `ConcreteFileVersion` as the exact, unordered file-format identity while leaving selector ownership and higher-level dispatch unchanged. Selectors resolve before persisted metadata is constructed, and manifest strings, `DataFile` fields, standard footers, and embedded footers use location-specific codecs. Existing wire mappings, legacy empty-manifest recovery, reader compatibility, and mixed-version rejection remain unchanged. Version-specific dispatch and encoding/compression/dataset execution migration are intentionally deferred to later PRs in the stack. Validation: - `cargo fmt --all -- --check` - `cargo check -p lance-file -p lance-table -p lance --tests`
## Summary - Replace heap-allocated, repeatedly hashed logical string keys with one opaque, canonical 16-byte `InternalCacheKey`. - Derive keys from stable type/schema identity plus typed fields using domain-separated BLAKE3, and migrate every in-tree cache-key producer to allocation-free typed encoding. - Keep `CacheBackend` object-safe while simplifying it around one physical key type; preserve codecs, accounting, `clear`, and Moka single-flight behavior. - Add deterministic cache/concurrency contracts, allocation guards, persistent-backend restart coverage, and paired Criterion benchmarks. Closes lance-format#7832. ## Stable key format `CACHE_KEY_FORMAT` is `blake3-128-v1`: 1. The root 32-byte namespace is generated with BLAKE3 `derive_key` using a fixed Lance context. 2. Each `with_key_prefix` segment derives a new keyed namespace with explicit domain and length framing. 3. Each entry hashes a stable type ID, author-defined schema ID/version, and tagged logical fields under that namespace. 4. Field tags are defined by a `#[repr(u8)]` enum. Variable-width values are length-framed; integers are fixed-width little-endian; options, variants, sequences, fixed bytes, and variable bytes have distinct one-byte tags. 5. The first 128 bits become the canonical backend key. `InternalCacheKey::{as_bytes,into_bytes,from_bytes}` are the persistence boundary. Changing a key schema version intentionally produces a cold miss. Persistent backends should include `CACHE_KEY_FORMAT` in their physical namespace and allow entries from older formats to age out; there is no runtime legacy-key fallback. The digest is a cache identity, not an authentication or authorization mechanism. Deterministic namespace keys are not secret. A 128-bit digest has approximately 64 bits of generic birthday-collision resistance and 128 bits of targeted preimage resistance. This change does not add a FIPS mode: lance-format#7832 selects BLAKE3 and Lance has no existing FIPS configuration surface. ## Intentional backend API break This removes APIs that require retaining logical strings or a secondary inventory: - `CacheBackend::invalidate_prefix` - backend key inventory / `LanceCache::keys` - readable key and prefix accessors - session cache-key inventory methods Use `clear` for explicit invalidation, or derive/version a new namespace when a logical scope changes. Custom backend migration: - Store/copy `InternalCacheKey` directly, or persist `key.into_bytes()` as exactly 16 bytes. - Reconstruct keys with `InternalCacheKey::from_bytes` when needed. - Route serialized values with `CacheCodec::type_id()` instead of inferring value type from a logical key string. - Replace `with_backend_and_prefix` with `with_backend(...).with_key_prefix(...)`. - Remove prefix scans and key-string parsing. The `CacheBackend` trait remains non-generic and object-safe. Existing out-of-tree `CacheKey` / `UnsizedCacheKey` implementations retain a source-compatible default bridge through `key()`. Performance-sensitive implementations should define a stable `CacheKeySchema` and override `write_key`; all current in-tree producers do so. ## Correctness and persistence coverage The proof suite covers: - official and golden BLAKE3 vectors, exact builder output, framing boundaries, endianness, type/schema/namespace separation, and options/variants/sequences; - default sized and unsized string bridges plus schema-version cold misses; - shared strong/weak cache state, expired weak handles, no-cache behavior, custom backends, Moka weights scaled safely above 4 GiB, and contextual type-collision misses/errors; - deterministic single-flight success, error, and owner-cancellation behavior with contenders explicitly parked before release/abort; - zero allocations for complete production-shaped typed page and optional-UUID keys after warm-up; - deletion-file cache identity across distinct storage bases; - a shared serializing backend that retains only bytes and opaque keys across restart and always decodes with the lookup codec; - BTree and IVF restart queries that prove serialized state/partitions are reused, assert vector recall, and perform zero index I/O once non-serializable readers are reconstructed, plus existing FTS/metadata/scalar integration coverage. The removed unstable IVF `cache_key_prefix` protobuf payload is reserved by field number and name. Its codec version is unchanged because protobuf removal is wire-compatible and the new physical-key format already guarantees a cold miss. ## Benchmarks Three independent Criterion passes used `release-with-debug`, 100 samples, Rust 1.97.0, and an AMD Ryzen 9 3900X under x86_64 WSL2. Both paths include the backend's outer hash. Reported ranges compare median time for the fixed path against the benchmark-local implementation of the previous string-key path: - Long production-shaped key preparation: **2.8%–10.3% faster**. - Short isolated key preparation: **139%–156% slower**; this exposes BLAKE3's fixed setup cost instead of hiding it. The motivating long-prefix workload improves. - Strong warmed hits: **7.7%–16.6% faster**. - Weak warmed hits: **1.4%–6.0% faster**. - Bounded rotating inserts with prebuilt values: between **8.9% faster and 3.0% slower** (effectively neutral; median pass was 1.3% faster). - Typed key preparation: **0 allocations** after warm-up. Namespace derivation is benchmarked separately so one-time scope setup is not folded into per-entry preparation. ## Validation Base: `a3c6fce816befb7072505fbe05cc55cd205a171e` Passed after rebasing onto that base and again after review follow-ups: - `cargo fmt --all -- --check` - `CARGO_INCREMENTAL=0 cargo check --workspace --tests --benches --locked` - `CARGO_INCREMENTAL=0 cargo clippy --all --tests --benches -- -D warnings` - `CARGO_INCREMENTAL=0 cargo test --workspace --locked` - `CARGO_INCREMENTAL=0 cargo +1.91.0 check --workspace --tests --benches --locked` - `CARGO_INCREMENTAL=0 cargo check --manifest-path python/Cargo.toml --locked` - `CARGO_INCREMENTAL=0 cargo check --manifest-path java/lance-jni/Cargo.toml --locked` - targeted lance-core cache/allocation and Lance serializing-restart tests on Rust 1.91 - error-path single-flight stress test repeated 1,000 times - three complete, symmetrically hashed paired benchmark passes All three lockfiles contain only the intentional BLAKE3 dependency change (plus the upstream release-version updates already present in the base). ## Interaction with open cache work - lance-format#7818 currently relies on readable key inventory and prefix parsing. Cache diagnostics should instead consume aggregate accounting/component metadata that is independent of physical keys; this PR intentionally does not preserve raw key enumeration. - lance-format#7828 should expose the key as an opaque 16-byte ABI value and use codec type metadata for serialized-value routing rather than freezing legacy string fields into the ABI. - lance-format#7683's registry/configuration model remains applicable, but registered custom backends must adopt the trait migration above. URI/config selection does not need readable physical keys. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>
## Why The Python workflow rebuilt the same `cp310-abi3` Linux x86_64 wheel in the Python 3.10, 3.13, 3.14, and AWS jobs. Compatibility tests then waited for the entire Linux matrix before downloading one of those identical wheels, adding an unnecessary serial dependency to the workflow critical path. This change introduces explicit wheel and memtest artifact producers. The Linux Python matrix, compatibility tests, and AWS integration tests reuse the immutable wheel artifact, while the Linux matrix reuses one prebuilt memtest library. ARM, macOS, and Windows coverage remains unchanged. ## Benchmark Measured on PR head `15d816e28` with [Python workflow run 30424299228](https://github.com/lance-format/lance/actions/runs/30424299228). The same SHA was run once with a cold producer cache and rerun once after that cache was populated. | Metric | Baseline | Cold producer cache | Warm producer cache | | --- | ---: | ---: | ---: | | Python time-to-green | P50 37.6 min / P90 39.6 min | 27.7 min (-26.2% vs P50) | 24.1 min (-35.9% vs P50) | | Compatibility test start offset | median 19.9 min | 9.1 min | 5.4 min | | Total runner-minutes | 147.0 min | 128.9 min | 123.5 min | | Linux x86_64 Rust cache restored | ~9.6 GiB | none | ~1.2 GiB | Time-to-green baseline uses the latest 64 successful PR runs before this PR (2026-07-27 through 2026-07-29). Runner-minute and cache baselines use [run 30291213427](https://github.com/lance-format/lance/actions/runs/30291213427), the run nearest that P50. Both PR attempts completed all 11 Python workflow jobs successfully.
…at#8068) `ObjectStoreParams` currently uses the full fat pointer of `Arc<dyn WrappingObjectStore>` as the cache identity. Trait-object vtable addresses can differ across codegen units even when both values point to the same `Arc` allocation. Equivalent parameters can therefore compare unequal and split the object-store registry cache; for `memory://`, the miss creates a fresh empty store and concurrent dataset operations can fail with `DatasetNotFound` under optimized builds. Wrapper cache identity should follow the underlying `Arc` allocation rather than vtable metadata so the same wrapper reliably reuses the same store.
…lance-format#8070) Fixes lance-format#8067. Fixes lance-format#8069. Blob v2 empty descriptors at position zero are valid values, but Python lazy conversion treated their descriptor contents as a null sentinel even though Blob v2 nullness is already carried by Arrow validity. Storage 2.1 compaction can also change a blob encoder's input from an all-valid batch to a nullable batch after deletion filtering. The descriptor encoder buffered both batches while retaining the first batch's definition interpretation, so a surviving null's packed definition level (`1 << 16`) could be exposed as a valid zero-length descriptor. This keeps the Python and storage fixes at their separate semantic boundaries: lazy conversion relies on Arrow validity while the storage 2.1 writer closes a pending descriptor page before its repetition/definition interpretation changes. Blob v1 sentinel behavior and storage 2.0/2.2 semantics remain unchanged.
## Summary An index can be built from an old dataset handle. If the indexed column is rewritten before the index is committed, the old index may still claim that fragment and return wrong rows. When the dataset has changed during the commit, this PR checks each new index against the latest data. Fragments whose indexed columns changed are removed from the index coverage, so queries scan those fragments instead. The fix applies to all index types. RTree is only used for the end-to-end regression test. ## Tests - Commit an RTree segment from a stale dataset handle after `RewriteColumns`. - Verify only the index on the changed column loses coverage. - Focused Rust and Python tests, Clippy, Rust format, and Ruff. ## Related Follow-up to the stale index coverage problem found in the [review of lance-format#7884](lance-format#7884 (review)).
## What is the bug? `BooleanQuery` and `BoostQuery` recursively plan their children without a limit so outer score composition remains exact. Nested `MultiMatchQuery`, however, applied its final fetch from the ambient scanner limit instead of the recursively supplied FTS search parameters. Linear: [OSS-1599](https://linear.app/lancedb/issue/OSS-1599/fix-nested-multimatch-limit-propagation-in-compound-fts-queries) ## What issues or incorrect behavior does the bug cause? A nested MultiMatch could discard candidates before an outer MUST clause, SHOULD score accumulation, or BoostQuery demotion finished combining scores. This could omit the true top-k or return incomplete scores. Equal-score rows could also appear in a different order between bounded and exhaustive execution. ## How does this PR fix the problem? - Use `FtsSearchParams::limit` as the recursive planning contract for MultiMatch fetches. - Treat `None` as complete execution for compound parents and document that future competitive-score pruning needs a separate contract. - Order compound FTS ties by `_score DESC, _rowid ASC`. - Add a regression test covering MUST, SHOULD, BoostQuery, standalone MultiMatch, no limit, small k, multiple fields, three fragments/segments, and score ties. This is a correctness and planning-semantics fix. It does not replace the current MultiMatch Union/Aggregate/Sort execution. ## Validation - `cargo test -p lance --lib test_nested_multimatch_limit_propagation -- --nocapture` - `cargo test -p lance --lib io::exec::fts::tests -- --nocapture` (15 passed) - `cargo test -p lance --lib dataset::tests::dataset_index::test_fts_ -- --nocapture` (22 passed) - `cargo clippy -p lance -p lance-index --lib --tests -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` Co-authored-by: Yang Cen <yang@lancedb.com>
## Summary - Avoid the unconditional input copy in `InlineBitpacking::unchunk` by borrowing a typed view over the inline bitpacking chunk words. - Keep output zero-initialization unchanged; this PR only changes compressed input handling. - Add focused `unchunk` roundtrip coverage and a Criterion benchmark comparing the old copy path with the new typed-view path. ## Benchmark Local machine: WSL2 on Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz, 12 logical CPUs. Command: ```bash cargo bench -p lance-encoding --bench decoder decode_inline_bitpacking_unchunk --features bitpacking -- --noplot ``` Results: | Case | Path | Criterion time estimate | Throughput | | --- | --- | ---: | ---: | | `u32_bw12_1024` | `legacy_copy` | 285.46 ns | 5.0244 GiB/s | | `u32_bw12_1024` | `typed_view` | 247.35 ns | 5.7985 GiB/s | | `u64_bw23_1024` | `legacy_copy` | 434.11 ns | 6.3331 GiB/s | | `u64_bw23_1024` | `typed_view` | 340.27 ns | 8.0797 GiB/s | Path-to-path deltas: - `u32_bw12_1024`: `typed_view` was 13.4% faster than `legacy_copy` - `u64_bw23_1024`: `typed_view` was 21.6% faster than `legacy_copy` The benchmark uses aligned chunks built with `LanceBuffer::reinterpret_vec`, so it measures the aligned `borrow_to_typed_view` fast path. Misaligned buffers can still fall back to a copy. ## Test Plan - `cargo fmt --all` - `cargo test -p lance-encoding --features bitpacking` - `cargo clippy -p lance-encoding --all-features --tests --benches -- -D warnings` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo bench -p lance-encoding --bench decoder decode_inline_bitpacking_unchunk --features bitpacking -- --noplot` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an inline bitpacking decode benchmark covering additional integer and bit-width scenarios, with feature-gated coverage when supported. * **Bug Fixes** * Improved inline bitpacking decoding with zero-copy typed parsing and stronger upfront buffer validation. * Centralized corruption error handling and ensured decoded output matches expected payload details. * **Tests** * Expanded/updated roundtrip and corruption tests, including header sizing, alignment, payload length mismatches, invalid bit widths, and excessive value counts. * **Chores** * Updated crate configuration to use the shared workspace object storage dependency. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…dantic_model() classmethod (lance-format#7383) ## Summary Closes lance-format#1106 This PR adds first-class Pydantic model support to Lance so users no longer need to manually call `.model_dump()` before writing data. **Two changes:** - `write_dataset()` now auto-converts a list of Pydantic `BaseModel` instances by calling `.model_dump()` on each item and inferring the PyArrow schema from the resulting dicts - New `LanceDataset.from_pydantic_model(model_class, data, uri=None, **kwargs)` classmethod that infers the dataset URI from the model class name (snake_case) and delegates to `write_dataset()` **Before (required manual conversion):** ```python from pydantic import BaseModel import pyarrow as pa, lance class MyModel(BaseModel): name: str score: float data = [MyModel(name="alice", score=0.9)] schema = pa.schema([pa.field("name", pa.string()), pa.field("score", pa.float64())]) lance.write_dataset([m.model_dump() for m in data], "/tmp/test.lance", schema=schema) ``` **After (just works):** ```python lance.write_dataset(data, "/tmp/test.lance") # or ds = lance.LanceDataset.from_pydantic_model(MyModel, data, uri="/tmp/test.lance") ``` ## Files Changed - `python/python/lance/dependencies.py` — added `_PYDANTIC_AVAILABLE` flag and `_check_for_pydantic()` helper, following the same optional-dependency pattern used for pandas and HuggingFace - `python/python/lance/types.py` — added Pydantic branch in `_coerce_reader()` before the generic `Iterable` branch; handles both Pydantic v1 (`.dict()`) and v2 (`.model_dump()`) - `python/python/lance/dataset.py` — added `from_pydantic_model()` classmethod to `LanceDataset` - `python/python/tests/test_dataset.py` — added Pydantic instances to the existing `test_input_data` parametrized suite and added `test_from_pydantic_model` ## Test Results ``` python/python/tests/test_dataset.py: 196 passed, 1 skipped, 4 failed ``` The 4 failures (`test_random_dataset_recall_accelerated`, `test_random_dataset_recall_accelerated_one_pass`, `test_count_index_rows_accelerated`, `test_count_index_rows_accelerated_one_pass`) all fail with `PermissionError: [Errno 13] Permission denied: 'nvcc'` — these require the CUDA compiler toolkit and are pre-existing failures unrelated to this change. They do not touch any code modified in this PR. New tests added: - `test_input_data[NoneType-pydantic]` — Pydantic instances auto-convert correctly in `write_dataset()` - `test_from_pydantic_model` — classmethod writes data, infers schema, round-trips correctly ## Checklist - [x] Pydantic instances can be passed directly to `write_dataset()` without manual `.model_dump()` - [x] `LanceDataset.from_pydantic_model()` classmethod exists and works - [x] Pydantic is treated as an optional dependency (no unconditional import) - [x] Both Pydantic v1 and v2 are supported - [x] All existing `test_input_data` parametrized cases still pass - [x] New tests cover both new entry points - [x] Rebased cleanly on upstream/main with no conflicts
Part 3/12 of lance-format#7877. Builds on merged lance-format#8019. This is an independently reviewable step toward the final layout demonstrated in lance-format#7979. This PR makes `lance-file::versions::v1` the canonical owner of the frozen V1 reader, writer, metadata, page table, and encoding grammar. The ambiguous compiled `previous` namespace and `lance_io::encodings` compatibility layer are removed. V1 keeps its historical byte-buffer materialization private; legacy IVF payload I/O remains explicitly owned by its index format, while protobuf tensor conversion validates its own byte contract. No general Arrow conversion API is introduced. Manifest dictionary materialization now dispatches once on the exact manifest version and delegates V1 bytes to the V1 implementation. Current-format readers and writers are not reorganized here, and no later-version dispatch API is introduced. V1 wire behavior remains fixed by the checked-in fixtures from lance-format#8019. The historical `lance.file.previous.*` cache-key identifiers are intentionally retained because they are persisted cache identity, not source-level ownership.
## Summary - Remove the dead segment-commit branch from `CreateIndexBuilder::execute`. - Remove helpers used solely by that branch and rename the stale IVF-RQ test. - The branch became redundant in [lance-format#6889](lance-format#6889), which replaced its segment metadata conversion with the same `CreateIndex` transaction as the fallback branch but left the guard in place. ## Testing - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` - Focused `index::create` vector, IVF-RQ, and BTree tests
Rust PR CI's warm-cache Linux coverage job has a 34:04 critical path. In the reference run, the `Run tests` step took 32:42: compilation accounted for about 3:51, while serial test-binary execution consumed roughly 29 minutes, led by `lance` at about 18 minutes and `lance-encoding` at about 7 minutes. This change builds the canonical `ci`-profile coverage binaries once, fans the archived tests out across three Linux runners, merges each shard's coverage profile before transfer, and produces the existing single Codecov report in the final `linux-build` check. It preserves the coverage optimization level and required check name while moving serial test execution off the critical path. ## Benchmark - Successful warm reference: 34:04. - Sharded cold run: 24:46, at least 31.6% faster than the same-SHA control's 36:14 lower bound. - Final-head warm run: 20:49, 38.9% faster than the successful warm reference. The stages were 5:02 to build and upload once, 13:10 for the slowest shard, and 2:27 to merge, report, and upload. - Three warm sharded runs completed in 20:49, 22:03, and 22:39. The median is 22:03, a 35.3% reduction. The same-SHA serial controls reached the final `lance-linalg` test binary at 36:14 and 37:31, then hit the same two existing randomized f16 property-test failures. The successful historical warm run is therefore the primary speed baseline; the same-SHA controls provide conservative lower bounds rather than successful end-to-end samples. Coverage was compared from the same instrumented archive with three shards versus one unsharded nextest job. Both reports contain exactly 531 files, 334,985 line mappings, and 655,891 segment denominators, with zero missing or changed denominators. The sharded report covered 11 more segments (89.2064% versus 89.2048%, +0.0017 percentage points), smaller than the observed 44-segment variation between two sharded runs. Aggregate final-head work was 30:54 on 8x runners plus 2:27 on a standard runner, versus 34:04 on one 8x runner for the reference. The speedup comes from parallelism without increasing aggregate large-runner time. The main tradeoff is a 1.49 GB test archive and roughly 1–2 minutes of artifact transfer per stage; artifacts are retained for one day. Evidence: [successful warm reference](https://github.com/lance-format/lance/actions/runs/30464415673/attempts/2), [cold and warm sharded runs](https://github.com/lance-format/lance/actions/runs/30475524847), [same-archive coverage validation](https://github.com/lance-format/lance/actions/runs/30481552021), and [successful final-head workflow](https://github.com/lance-format/lance/actions/runs/30483893465).
…#8038) Stack 1 of the generic block compression series. Mini-block codecs need page framing information to compare container-level choices without creating a parallel planner API. This change passes one explicit context through the existing mini-block codec tree and distinguishes ordinary mini-block pages from SparseLayout callers. It intentionally changes no codec selection or persisted bytes: ordinary mini-block and SparseLayout writers keep their existing behavior, while later stack layers can consume the context when evaluating generic offset containers. Validation covered the complete lance-encoding test suite and workspace clippy with warnings denied. <!-- generic-block-stack-navigation --> ## Stack navigation - Umbrella / integration reference: lance-format#8002 - Next: lance-format#8040
Part 4/12 of lance-format#7877. Builds on merged lance-format#8020. This is an independently reviewable step toward the final layout demonstrated in lance-format#7979. This PR separates reusable encoding mechanisms from file-version composition. `lance-encoding` exposes array-encoding, structural encoding, decoding, and compression mechanisms without selecting a file version. Exact V2.1, V2.2, and V2.3 modules in `lance-file` own their compression and encoding composition instead. The goal is to make algorithm changes local and reusable while keeping the choice of algorithms isolated by exact file version. This removes the old `previous` encoding namespace and does not add a parallel compression abstraction. Validation: - `cargo clippy --all --tests --benches -- -D warnings` - encoding benchmarks compile - exact-version compatibility fixtures remain unchanged
lance-format#8102) ## Problem Two hot paths deep-copy the entire fragment descriptor list on every call, costing O(dataset fragments) regardless of how many fragments are actually touched: - `do_take_rows` called `dataset.get_fragments()` — cloning every `Fragment` descriptor (including `DataFile` path strings) — and then kept only the fragments addressed by the take. - `plan_match_query` / `plan_phrase_query` materialized `self.dataset.fragments().to_vec()` per query, even though the list is only read. On a 126M-row dataset with 3,150 fragments this costs ~1 ms per call (~300 ns per descriptor clone+drop). Under a take-heavy workload both FTS plan build and the take pay it on every request, which showed up as a hard ~1,000 qps per-process ceiling with the machine >95% idle: thread dumps showed active threads dominated by `Fragment::clone` / `drop_in_place<DataFile>` under `Scanner::plan_fts` and `do_take_rows`. ## Fix - `do_take_rows`: construct `FileFragment` handles only for the addressed fragment ids via `get_existing_fragments_from_ids` (same skip-missing semantics as the previous `filter_map`). - `plan_match_query` / `plan_phrase_query`: borrow the fragment slice (`&[Fragment]`); materialize only in the no-index flat fallback.
…at#7944) ## Summary - cache only portable IVF state and file metadata, rebuilding readers with the object store supplied by each dataset open - resolve the IVF_RQ index directory from the index metadata so shallow clones and external index bases read the source index path - preserve reusable IVF partition entries across credential rotations while routing reader I/O through the current object store - keep the existing in-memory caching of legacy (v0.1/v0.2) live vector indices; document that their store-bound readers assume internally refreshing credentials (e.g. a credentials provider), and that static-per-open credential deployments should use v0.3+ index formats ## Testing - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo test -p lance --lib test_vector_cache_uses_current_object_store -- --nocapture` (V3: readers rebind to the current store; Legacy: cached live index keeps its original store) - `cargo test -p lance --lib test_shallow_clone_ivf_rq_uses_resolved_index_directory -- --nocapture` - `cargo test -p lance --lib test_prewarm_ivf_legacy -- --nocapture` - `cargo test -p lance --lib test_prewarm_ivf_pq -- --nocapture` - `cargo test -p lance --lib test_prewarm_and_query_with_serializing_backend -- --nocapture` - `cargo check -p lance-examples --example hnsw` Closes lance-format#7904
Part 5/12 of lance-format#7877. Depends on lance-format#8021. This is an independently reviewable step toward the final layout demonstrated in lance-format#7979. This PR adds writer implementations owned by `versions::v2_0`, `versions::v2_1`, `versions::v2_2`, and `versions::v2_3`. Each exact version composes the mechanisms it supports; genuinely shared structural machinery remains under the version-neutral writer module. Some composition code is intentionally repeated across adjacent versions. That repetition is the isolation boundary: changing a newer version must not silently change an older version. Public writer dispatch is left unchanged in this step and is activated by the next PR. Validation: - exact-version writer compatibility fixtures - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all -- --check`
Unreleased version after creating v10.0.0-rc.1
Part 11/12 of lance-format#7877. Depends on lance-format#8027. This is an independently reviewable step toward the final layout demonstrated in lance-format#7979. This PR converts scalar and vector index readers, writers, shufflers, and distributed mergers to exact `ConcreteFileVersion` identities. Dataset-backed indexes inherit the dataset format explicitly; legacy dataset physical indexes map explicitly to V2.0. With the last transitional consumers migrated, this PR removes file-version ordering and implicit conversions between selectors and exact formats. Remaining version decisions are exhaustive matches at declared boundaries rather than `>=`, `max`, or selector round-trips. Validation: - index format inheritance tests - legacy dataset physical-index format test - V3 shuffler tests - `cargo clippy --all --tests --benches -- -D warnings` - Python and Java binding checks
## What changed? Update the origin-only guards in the nightly workflow from the former `lancedb/lance` repository name to `lance-format/lance`. ## Why is this needed? The stale repository checks skip every nightly job on the current repository. This restores the file-verification dispatch, jumbo tests, and the index maintenance-sequence compatibility test. Recent scheduled runs showing the workflow was skipped: https://github.com/lance-format/lance/actions/workflows/nightly_run.yml ## Validation - Parsed `.github/workflows/nightly_run.yml` with Ruby YAML - Confirmed all three origin guards use the current repository name - `git diff --check` - Pre-commit hooks passed
## What is the bug? Compound WAND score bounds sum non-negative clause maxima in f64 and round upward once. Exact scoring accumulates f32 values recursively, and a different clause order can produce a score one ULP above that bound. A competitive document can therefore be pruned incorrectly. ## How does this PR fix it? - Widen the f64 total with the existing clause-count upper-bound factor before converting to f32. - Round upward again when the f32 conversion rounds down. - Add deterministic bit-level regressions for query-order and reordered f32 accumulation. ## Stack This is 1/3 for OSS-1706: 1. lance-format#8473 - conservative score-sum bounds 2. lance-format#8474 - pure-SHOULD clause MAXSCORE 3. lance-format#8475 - metrics and end-to-end observability ## Validation - cargo fmt --all -- --check - cargo test -p lance-index scalar::inverted::wand::tests::conservative_score_sum_covers_query_order_f32_rounding --lib -- --exact - cargo clippy --all --tests --benches -- -D warnings
) Summary - Add Java APIs to select registered native cache backends using either backend URIs or structured CacheBackendConfig. - Allow index and metadata caches to switch backends independently while preserving existing size-based and default configurations. - Bridge backend configuration through JNI to Lance’s cache registry, with validation and tests for conflicting or invalid settings.
Each was sized far beyond what it asserts. Measured locally on 30 cores with --profile ci: binary_copy 118.9s -> 1.9s, sparse_large_string_list 174.0s -> 11.8s, ngram_index_with_spill 130.2s -> 15.7s. The lance + lance-index + lance-encoding lib suite goes 246.8s -> 128.3s wall, test CPU -14%, all tests still passing. binary_copy runs its four file versions as rstest cases rather than a serial loop, and compacts 100 input fragments instead of 1,000 (~14ms each, single compaction task either way). sparse_large_string_list derives its size from max_repdef_levels_per_chunk() instead of a literal 2.5M, so it keeps crossing the rep/def threshold it was added to cover (lance-format#6184). ngram_index_with_spill uses 512 rows instead of 4,096, which still forces ~57 spills and a many-way merge. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#8438) ## What `Operation::UpdateMemWalState` builds its manifest from scratch and never populates `final_fragments`, so the commit publishes a manifest with **no fragments**. Every row in the table disappears. Nothing errors. The operation touches indexes rather than data, so it is declared compatible with concurrent `Append` / `CreateIndex` and no conflict is raised — the commit succeeds and the data is gone. Compare `UpdateBases`, the arm immediately below: also index/metadata-only, and it does carry the fragment list forward. ## Fix ```rust final_fragments.extend(maybe_existing_fragments?.clone()); ``` ## Reachability No caller in this repo commits a standalone `UpdateMemWalState` today. MemWAL compaction progress is recorded by `MergeInsertBuilder::mark_sstables_as_compacted`, which rides an `Operation::Update` — a data operation that populates fragments normally. So the bug is latent here, not active. It becomes reachable the moment anything commits the operation on its own, which the MemWAL index catch-up work (lance-format#8263) does. ## Tests `test_update_mem_wal_state_preserves_fragments`: commit `UpdateMemWalState` on a dataset with rows, assert the fragment list and row count are unchanged. Verified it fails without the fix (`left: []`, `right: [0]`). The existing tests in `index/mem_wal.rs` already commit this operation against a dataset holding 10 rows — they pass because none of them read the rows afterwards, asserting only on conflict behavior and `compacted_sstables`. `cargo test -p lance --lib -- mem_wal transaction` — 636 passed.
## Summary - reject `retain_versions=0` in the shared cleanup policy builder before version lookup - return a descriptive invalid-input error instead of panicking - document the positive-value requirement and cover both Rust and Python callers ## Root cause `retain_n_versions` calculated the cutoff as `versions[versions.len() - n]`. When `n` was zero, this indexed one element past the end of the versions list and triggered a Rust panic through the Python binding. ## Validation - `cargo test -p lance cleanup_rejects_retain_zero_versions` - `uv run pytest python/tests/test_dataset.py::test_cleanup_with_retain_versions` - `cargo fmt --all` - `cargo clippy --all --tests --benches -- -D warnings` - `uv run make lint` Fixes lance-format#8464 <!-- lance-gatekeeper-fix:v1 agent=d202932b065b13e0c1c4f8077f166778 generation=1 --> Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
…ormat#8483) The dedicated job spent 13m41s compiling to run 49 tests in 6.57s, and cached no target dir so every run was cold. Those tests are 60-row fixtures and are not slow any more, so stop excluding slow_tests from ALL_FEATURES and drop the job. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…iteColumns (lance-format#6748) ## Summary * Follow-up to lance-format#6650 (fix: propagate update_columns offsets and partial last_updated for RewriteColumns). * `Update.java`: adds `Map<Long, byte[]> updatedFragmentOffsets` field, 7-arg constructor, accessor `updatedFragmentOffsets()`, and `Builder.updatedFragmentOffsets(...)` setter. Defaults to `Collections.emptyMap()`. Values are portable RoaringBitmap bytes. * `java/lance-jni/src/transaction.rs` — two JNI directions updated: FromJava deserializes each `byte[]` value into a `RoaringBitmap` and sets `updated_fragment_offsets` on the Rust operation; IntoJava serializes each bitmap to `byte[]` and populates a `HashMap<Long, byte[]>` passed to the 7-arg `Update` constructor (previously the field was ignored and the 6-arg form was used). * `Update.java` `equals` / `hashCode`: deep-compares `byte[]` values by content; `hashCode` added per the Java contract. * Required by [lance-spark#418](lance-format/lance-spark#528) ## Background PR lance-format#6650 added `updated_fragment_offsets` on the Rust `Operation::Update` (proto field 9), `build_manifest` partial refresh logic, and `FragmentUpdateResult.getUpdatedRowOffsets()`. Two gaps remained: 1. The Java `Update` class had no field for these offsets and `convert_to_rust_operation` always set `updated_fragment_offsets: None`, so the lance-spark commit path (UpdateColumnsBackfillBatchWrite) had no way to pass offsets to Rust and the partial refresh in `build_manifest` could never activate from a JVM caller. 2. `convert_to_java_operation_inner` still used the old 6-arg constructor signature for `new_object`. With the 6-arg constructor removed from `Update.java` (replaced by the 7-arg form), any Rust→Java materialization of `Operation::Update` (e.g. reading back a transaction) would fail at runtime with `NoSuchMethodError`. ## Implementation notes * Values are portable RoaringBitmap bytes (little-endian, spec-compliant). The JNI boundary stays O(bitmap size) rather than O(n matched rows). * `with_local_frame(4, ..)` per bitmap entry in IntoJava bounds local-ref growth on large offset maps. `JMap` was avoided inside the frame because it holds a `JObject` with the outer frame's lifetime, causing borrow-checker conflicts; `call_method` on the outer `java_map` reference is used instead. * The `Vec<u8>` buffer for each bitmap is allocated in Rust before entering the frame, so its lifetime is independent of JNI frame scope. * `with_local_frame(8, ..)` per iteration in FromJava bounds local-ref growth for large multi-fragment maps. * `build_manifest` validates bitmap cardinality and max offset against the fragment's `physical_rows` from `existing_fragments` before `.collect()`, preventing a compact RLE bitmap from expanding into an unbounded allocation. * `UpdatedFragmentOffsets` added to the `lance::dataset::transaction` import. ## Why the protobuf field alone is not enough lance-spark commits by calling `CommitBuilder.execute(transaction)`, which passes the Java `Transaction` object to `nativeCommitToDataset` via JNI. The JNI handler calls `convert_to_rust_transaction` → `convert_to_rust_operation`, which reflects on the Java `Update` object to build the Rust `Operation::Update` struct. The protobuf field (field 9) is only used when a Transaction is serialized as a proto blob; it has no effect on the reflection-based JNI path unless the Java `Update` class exposes the field and the JNI deserialization reads it. ## Additional change `FragmentUpdateResult` (from lance-format#6650) returned matched row offsets as an expanded `long[]` at the executor JNI boundary. This PR also passes those offsets as portable RoaringBitmap bytes so lance-spark can wire them through to `Update.updatedFragmentOffsets()` without an O(n rows) expansion on the executor→driver path. * `FragmentUpdateResult.getUpdatedRowOffsetBytes()` — primary accessor; values are the same portable RoaringBitmap byte format as `Update.updatedFragmentOffsets()`. * `java/lance-jni/src/fragment.rs` — `update_columns_with_offsets` serializes `matched_offsets` once with `RoaringBitmap::serialize_into`; JNI constructs results via the private `(FragmentMetadata, long[], byte[])` constructor (JNI can access private ctors). `FragmentUpdateResult.create(FragmentMetadata, long[], byte[])` — public static factory; primary construction path for callers using the bytes API. * `@Deprecated getUpdatedRowOffsets()` — retained for backward compatibility; expands bytes via `expandRowOffsetsFromBytes` only when called (lazy O(n rows)). * `@Deprecated` 3-arg `(FragmentMetadata, long[] fieldsModified, long[] updatedRowOffsets)` constructor — encodes offsets via `encodeRowOffsetsToBytes` for source compat. * `FragmentUpdateResultTest` — round-trip bytes, deprecated constructor encode, and `updateColumns()` integration asserting matched offsets `{0,1,2,3}` on the test fixture. ## Test plan * `UpdateTest#testUpdatedFragmentOffsetsRoundTrip` — commits an `Update` with a non-empty `updatedFragmentOffsets` map through `CommitBuilder.execute` (exercises the FromJava JNI path), reads the transaction back via `Dataset.readTransaction()` (exercises the IntoJava JNI path), and asserts the offsets match. Map value is hardcoded portable RoaringBitmap bytes encoding {1, 3, 5}; verified with `assertArrayEquals` after the round-trip. * `FragmentUpdateResultTest` — see [Additional change](#additional-change). ## Compatibility * Additive new API on `Update` — the `updatedFragmentOffsets` field did not exist in any prior release. The builder setter is optional and defaults to `Collections.emptyMap()`, so existing `Update.builder()...build()` call sites compile and behave identically. * Java — `equals` / `hashCode`: `equals` uses `offsetMapsEqual` to deep-compare `byte[]` values via `Arrays.equals`; `hashCode` is added per the Java contract. * JNI — constructor signature: the IntoJava `new_object` call is updated from the 6-arg to the 7-arg form in the same PR. Both files must ship together; within that atomic change there is no compatibility gap. * Rust / proto: no changes. The `updated_fragment_offsets` proto field and Rust struct field were already added in lance-format#6650. * `FragmentUpdateResult` — `@Deprecated` long[] getter and constructor retained; new bytes getter is the supported path for new callers (see [Additional change](#additional-change)). --------- Co-authored-by: Jing chen He <jingh@adobe.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…t#8220) ## Summary - reject dictionary pages when normalization widens indices beyond the declared Arrow key type - widen normalized indices when an appended or reused null dictionary value is outside the declared key range - reject mismatched dictionary buffers at the Arrow decode boundary, including when optional validation is disabled - cover appended and reused null positions plus the low-level corrupt-buffer case ## Root cause Nullable dictionaries store nulls as a dictionary value. When an `Int8` dictionary needed a null position outside its declared key range, normalization could either append or reuse that value. The append path widened the physical indices to `UInt32`, but the page remained declared as `Int8`; the reuse path cast an existing out-of-range null position back to `Int8` without widening. Both paths could therefore write data that the declared key type could not represent. The writer now rejects widened normalized indices, null normalization applies the declared range check to both appended and reused null values, and the decoder rejects mismatched widths as corrupt data. ## Validation - `cargo fmt --all -- --check` - `cargo test -p lance-encoding dictionary -- --nocapture` (41 passed, 1 existing ignored) - `cargo test -p lance write_rejects_dictionary_null_index_outside_declared_key_range -- --nocapture` (2 passed) - `cargo test -p lance append_dictionary -- --nocapture` (2 passed) - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#8217 <!-- lance-gatekeeper-fix:v1 agent=247db31c00f1ec1edade20019168c899 generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
## Summary - remap BloomFilter candidate row addresses through the loaded fragment reuse index - preserve exactness and nullable-row semantics while remapping search results - add a multi-fragment deferred-compaction regression test that verifies an indexed equality match remains visible ## Root cause BloomFilter zones retain the physical row addresses from index creation. Deferred-remap compaction moves those rows and supplies a fragment reuse index when loading the scalar index, but BloomFilter search returned the original zone ranges without applying that mapping. ## Validation - `cargo test -p lance test_read_bloom_filter_index_with_defer_index_remap -- --nocapture` - `cargo test -p lance-index scalar::bloomfilter::tests` - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` - `git diff --check` Fixes lance-format#8221 <!-- lance-gatekeeper-fix:v1 agent=b2f937c7aee8cfefab0c005735457c74 generation=1 --> Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
## Summary - read blob-v2 columns selected by the fragment updater through their descriptor representation - convert existing descriptors back to logical blob values before joining and rewriting the column - preserve untouched non-empty, empty, and null blob cells in a regression test ## Root cause `FileFragment::update_columns` opened the logical blob-v2 struct schema directly against a blob encoded as one atomic physical column. The decoder therefore received more logical fields than projected physical column metadata. Existing fallback rows also need conversion from stored descriptors to the logical writer representation before they can be interleaved with incoming blob values. ## Validation - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` - `cd python && make build` - `cd python && uv run make lint` - `cd python && uv run pytest python/tests/test_fragment.py -k 'fragment_update_columns' -q` (9 passed) Fixes lance-format#8336 <!-- lance-gatekeeper-fix:v1 agent=d987d544ac5c64f553ecc4e30dee0065 generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
…#8462) Hi team, I think `create_branch` initiates a new branch from the current version, instead of latest version. I used the following script to confirm. ```py def main() -> None: with tempfile.TemporaryDirectory(prefix="lance-reference-none-") as tmp: uri = Path(tmp) / "dataset.lance" v1 = pa.table({"value": ["v1"]}) v2 = pa.table({"value": ["v2"]}) ds = lance.write_dataset(v1, uri) ds = lance.write_dataset(v2, uri, mode="overwrite") historical = ds.checkout_version(1) default_branch = historical.create_branch("default-from-checkout") print(f"Default branch version: {default_branch.version}") # 1 ```
…e-format#8352) ## Problem A tombstone carries the primary key and null in every other column, so `ShardWriter::delete` required every non-PK column to be nullable in the base table. That pushes a storage-engine detail into the user's schema for no reason the user can see. ## Approach Split the shard's schema in two: - **logical** — the base table's schema, exactly as the caller declared it. The contract input is validated against, and the schema the scan path returns. - **storage** — every non-PK top-level field widened to nullable. What the memtable, WAL entries, and SSTables physically carry. This mirrors the logical/physical split `SchemaAdapter` already applies to JSON and view types in `dataset/utils.rs`, and rides the boundary that already exists — `_tombstone` is a physical column the SSTable schema carries and the base table does not. Widening is **top-level only**. Arrow validates nullability just at the top level of a `RecordBatch`, so a null `FixedSizeList`/`Struct` needs no change below the top; a vector column's item field is untouched and gains no validity layer. Primary keys are never widened (`Schema::unenforced_primary_key` requires them non-nullable), so `build_tombstone_batch` still rejects a null, mistyped, or missing key — the delete path needed no new validation. ## Where the contract is enforced **Ingress** — `put` validates against the logical schema before the WAL append: column names and order first, then count, types, and nullability. This is now the *only* gate. Both append (`write/insert.rs`) and `merge_insert` compare schemas with `NullabilityComparison::Ignore`, and the encoder derives validity from the array rather than the field, so a null that got past `put` would land in a non-nullable base column silently. Validating pre-append also matters for a second reason: a batch that is appended and only then rejected fails identically on every replay, leaving the shard unable to reopen — the same hazard that puts `validate_index_configs` ahead of `claim_epoch`. Names are checked separately from `RecordBatch::try_new`, which matches positionally against bare `ArrayRef`s that carry no names. `ensure_tombstone_column` then re-labels positionally too, so a caller batch with two same-typed columns in the wrong order would be written to the memtable, WAL, and SSTables under each other's names, and nothing downstream would notice — `MemTable::insert_batches_only`'s schema-equality check runs *after* the relabel, comparing the storage schema against itself. WAL-only mode is covered as well; it previously validated nothing at all. **Egress** — the scan narrows back to the logical schema after tombstones are filtered. `project_to_canonical` documented that it emits its `target_schema` but did not: DataFusion derives `ProjectionExec` nullability from its expressions, not from the requested schema. A new `SchemaRelabelExec` makes that real, applied through `force_schema`, which wraps only when the schemas actually disagree. The same node widens arms so they agree before `UnionExec`/`CoalesceFirstExec`, both of which require exact schema equality — `CoalesceFirstExec::new` asserts it and would otherwise panic on the base-arm/WAL-arm nullability difference. The narrowing doubles as a runtime assertion: if a tombstone ever escaped its filter, `RecordBatch` validation rejects the null instead of handing the caller a row of nulls. The row count is carried explicitly through the relabel, so a column-less batch keeps its rows and an empty batch is still checked against the target schema rather than waved through. Ordering falls out of this — `carry_schema` in the point-lookup path is built on the widened schema, because tombstones are still in flight until `filter_tombstones_after_coalesce`. `vector_search` and `fts_search` needed no changes; they already route every arm through `project_to_canonical`. `ensure_tombstone_column` now always re-labels instead of passing through a batch that already has the column, so an entry written under an older storage schema replays into the current one. ## Tests 18 new tests. - `test_delete_against_non_nullable_base_column_round_trip` — the headline: delete against a base table with a non-nullable non-PK column, survivors keep their values, and the scan reports the base table's own nullability. - `test_put_rejects_null_in_non_nullable_base_column` / `..._wal_only_...` — the ingress gate in both modes. - `test_put_rejects_swapped_same_typed_columns` — two `Utf8` columns handed over in the wrong order are rejected by name; before this gate they were stored transposed. - `test_build_tombstone_batch_nulls_non_nullable_base_column` / `..._rejects_null_primary_key` — widening works, PKs still strict. - `relax_*` — top-level-only widening, nested fields untouched, `_tombstone` stays non-nullable, idempotence, metadata preserved (the PK marker rides on field metadata). - `schema_relabel::tests` — widening, narrowing, narrowing rejects a surviving null, empty batches relabeled and still type-checked. - `projection::tests` — `force_schema` leaves a matching plan alone and wraps a nullability mismatch; `project_to_canonical` reports its target schema exactly. ## Verification - `cargo test -p lance --lib` — 2726 passed, 0 failed, 3 ignored (589 of them `mem_wal`). - `cargo clippy --all --tests --benches -- -D warnings` — clean. - `cargo fmt --all` — clean. ## Outstanding Draft because of this, not because the change is incomplete: - No end-to-end test yet confirming that Lance silently accepts a null into a non-nullable base column via `merge_insert`. The ingress gate is written as if it is load-bearing for base-table integrity, which is the safe assumption and what the code reading indicates, but it is unverified. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ormat#8241) ## What Two additive, read-only accessors on the MemWAL writer. **`MemTableStats.frozen_count` / `.frozen_bytes`** — `memtable_stats` reports the active memtable only, so the backpressure threshold (`max_unflushed_memtable_bytes`, which meters active + frozen) has no observable numerator. Both are read under the read lock `memtable_stats` already holds. Deliberately not `in_memory_memtable_refs`: that calls `check_poisoned()?` first, so it goes blank exactly when an operator most needs the numbers. `memtable_stats` is already documented as poison-tolerant for that reason. The two fields have different denominators on purpose, and the doc comments say so: `frozen_bytes` drains the moment a flush commits (that is what backpressure meters), while the handle lingers in the read view for `frozen_memtable_grace`, so `frozen_count` does not drop to zero with it. **`ShardWriter::backpressure_stats()`** — `BackpressureController::stats()` and `BackpressureStats::snapshot()` are already public, but the controller sits inside a private `WriterMode` variant with no accessor, so the counters are unreachable from outside the writer. Answers in both modes, so a caller never has to know which one it is in. ## Why LanceDB's WAL service polls these to publish four Prometheus metrics it cannot derive today: `wal_unflushed_bytes`, `wal_sealed_memtables`, `wal_backpressure_waits_total`, `wal_backpressure_wait_seconds_total`. Without them a WAL pod can be throttling, or sitting a hair under an OOM, with nothing on a dashboard to say so. ## Tests - `test_memtable_stats_frozen_count_outlives_frozen_bytes` — pins the differing-denominator semantics: after `wait_for_flush_drain` under a long grace, count is non-zero and bytes are zero. - `test_backpressure_stats_reachable_in_both_modes` — `rstest` over `enable_memtable` true/false, since the point of the accessor is that the private-variant match covers both. `cargo test -p lance --lib -- mem_wal` → 561 passed, 1 ignored. `cargo fmt --all` clean. Clippy is clean on the changed file; `-D warnings` currently fails elsewhere in the tree on pre-existing `single_range_in_vec_init` under Rust 1.97. ## Scope `rust/lance/src/dataset/mem_wal/write.rs` only. No format change, nothing in `lance-core`, no public signature altered — only new fields on a struct and one new method. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ance-format#8481) Replaces the `IndexCatchupAdvance` mechanism from lance-format#8263. A commit no longer carries a claim about index coverage; the coverage is derived from the version the transaction read. ## Why Under lance-format#8263, a worker that extended an index had to describe what it had done — the index name, the exact segment UUIDs it expected to publish, the fragments those segments covered when it looked, and every fragment live at that moment — and the commit re-validated all of it. Four fields and a validation pass to transmit a fact the commit can already see. It can see it because coverage has only one possible proof. Nothing maps a compaction generation to the fragments its rows landed in. The only way an index can show it holds those rows is to span the table as the transaction read it. So rather than accept a claim and check it, derive it: an index whose segments together cover every fragment live at `read_version` is caught up to that version's `compacted_sstables`. Three things follow that the advance model could not offer: - **A claim cannot go stale.** There is no window between inspecting and committing, because there is nothing to inspect. - **The answer survives rebase.** `read_version` is fixed for a transaction's life, so every commit attempt derives the same result. lance-format#8263 needed the advance carried through the rebase untouched and re-validated. - **Any operation that commits can earn coverage.** An ordinary index build that happens to cover the table records catch-up as a side effect. Under lance-format#8263 only a dedicated repair could, so a build that fully covered had to throw the fact away and wait for a repair to re-establish it. Because the position is only written by a commit, `optimize_indices` keeps committing on an activated table even when it has no new segment to publish. ## What it keeps from lance-format#8263 The parts that were not about transmission: - `FLAG_MEM_WAL_INDEX_CATCHUP`, both words, and the refusal of a half-set state - `index_catchup` on `MemWalIndexDetails`, and the reader rule that a missing entry means "not caught up" - Activation (`require_index_catchup`), including its refusal of a table that already carries beta-protocol compaction progress - Withdraw-on-change: an index this commit changes keeps no position it cannot re-earn Two rules bound what a commit may record. It never credits past its own `compacted_sstables`, so a read version since rolled back cannot retire SSTables no live commit copied in. And it never lowers a position an index already held, provided the index is unchanged. "Unchanged" compares whole segment metadata, not segment UUIDs. `Operation::Update` prunes a segment's fragment bitmap in place when it touches an indexed field, keeping the UUID — so a UUID-only comparison carries a position forward for an index that now covers less. That is not hypothetical; it is reachable from an ordinary merge-insert. `a_bitmap_pruned_in_place_does_not_keep_its_position` pins it. ## What it removes `IndexCatchupAdvance` and its proto message, the `mem_wal_index_catchup_advances` field on `CreateIndex`, `OptimizeOptions::mem_wal_index_catchup`, the advance-validation pass, and the rebase handling that carried an advance through. ## Tests 34 unit tests over the derivation, in `dataset/transaction.rs`, and 6 through a real commit, in `index/mem_wal.rs`. The derivation alone is not the feature — `commit_transaction` has to load the read version and hand it down, and only for tables carrying the bit — so the commit-path tests cover that an index earns coverage, a legacy table earns none, and a rebase past an append does not move what a commit earns. Twelve fences were regressed one at a time and the failing test confirmed. Four of the first attempts caught nothing, because the test asserted an outcome that both the correct and the broken path produce; each was replaced with one that discriminates. One guard is deliberately untested: skipping the read-version index load on legacy tables is a cost guard, not a correctness one, and regressing it changes no observable behaviour. `cargo test -p lance --lib`: 2914 passed. fmt and clippy clean. ## Follow-ups - A user index build cannot rebase past an `UpdateMemWalState` commit — only the system index may, anything else is rejected outright rather than retried (`conflict_resolver.rs`, unchanged since January). Now that an ordinary build can earn coverage, that race is worth revisiting: it costs a completed build. - `segments_before` still clones every index segment on each commit for tables on the protocol. The snapshot has to be owned because the operation rewrites the list, but a smaller snapshot would do.
…at#8482) During a distributed FTS index build, write canonical metadata even when no partitions are produced. This distinguishes a valid empty index segment from a failed or incomplete build.
## Summary - add a streaming Rust version-count API that enumerates retained manifest locations - avoid reading and deserializing every historical manifest, unlike `versions()` - expose the count through Java as `Dataset.getVersionCount()` - keep detached versions excluded, matching the normal version history ## Testing - `cd java && ./mvnw test -Dtest=DatasetTest` - `cargo clippy -p lance --tests --benches -- -D warnings` - `cd java && cargo clippy --tests --manifest-path lance-jni/Cargo.toml -- -D warnings` - `cd java && ./mvnw spotless:check` --------- Co-authored-by: wangzheyan <wangzheyan@bytedance.com>
## What changed? Extend the cross-version index maintenance-sequence search with a bounded `IVF_PQ` + `BTREE` prefilter scenario. - cover every single maintenance operation and ordered operation pair across valid writer/reader splits - add five curated deeper lifecycles, including the two-vector-delta plus scalar-unindexed-row state from lance-format#3769 - compare scalar-index results with a full scan - compare indexed ANN prefilter results with exact index-free KNN and require recall@10 >= 0.5 - cap vector search at four shards and avoid exponential sequence growth - include `IVF_PQ` in manual compat-pair `all` ## Why is this needed? The existing sequence search covers scalar and FTS indexes but not vector/scalar prefilter interactions. That is the main remaining unique signal in the legacy recurring test. This adds deterministic, isolated cross-version coverage with correctness oracles before that recurring matrix is removed. ## Validation - deterministic bounded-generator test - current runtime: all 103 max-length-5 cases passed across four shards in 10.51s - Pylance 9.0.1 writer to 10.0.0 reader: - all 103 cases passed in one shard - the default four-shard run passed in 9.07s after environment setup - existing `BTREE` sequence smoke passed - `uv run make lint` - workflow YAML parse - `git diff --check`
## Summary - intersect explicit scan fragment scopes with scalar-index coverage during count pushdown - restrict index-backed and fallback count branches to only the requested fragments - cover single-fragment counts, partial index coverage, and stable row IDs with deletions ## Test plan - [x] `cargo fmt --all` - [x] `cargo clippy --all --tests --benches -- -D warnings` - [x] `cargo test -p lance --lib io::exec::count_pushdown::tests -- --test-threads=1` - [x] `cargo test -p lance --lib io::exec::count_from_mask::tests -- --test-threads=1` - [x] `cargo test -p lance --lib dataset::fragment::tests::test_fragment_count -- --test-threads=1` Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>
## What changed? - remove the weekly Recurring Tests workflow - remove the legacy all-permutations Python test - remove its pytest marker and default test exclusion - clean up the stale recurring-test wording on a release-build-only skip - keep the shared failure-issue action used by cargo, PyPI, and Java publish workflows ## Why is this needed? The legacy test mutates one dataset across supposed permutations, uses unseeded random data, and analyzes query plans without checking query results. Making its full workload complete required 192 matrix jobs and about 92.7 GitHub-hosted runner hours in [the successful full run](https://github.com/lance-format/lance/actions/runs/31202432846). Its useful coverage is replaced by: - lance-format#8470, which restores the nightly cross-version compatibility sequence workflow after the repository rename - lance-format#8476, which adds deterministic `IVF_PQ` + `BTREE` prefilter maintenance sequences with exact-result and recall oracles This PR should merge after both replacement PRs. ## Validation - `make install` - `uv run make lint` - `uv run --no-sync pytest -q python/tests/test_filter.py` (`16 passed, 1 skipped`) - verified no remaining recurring-test references - verified the shared failure-issue action is still referenced by cargo, PyPI, and Java publish workflows - `git diff --check` Closes lance-format#4511
The Rust core's MergeInsertBuilder supports source_dedupe_behavior (Fail/FirstSeen) to control how duplicate source rows that match the same target row are handled, but the Java binding never wired it. Java callers were stuck on the default (Fail) with no way to opt into FirstSeen. Add SourceDedupeBehavior enum + withSourceDedupeBehavior() builder to the Java MergeInsertParams, pass it through the JNI layer, and re-export SourceDedupeBehavior from lance::dataset so the binding can import it alongside the other merge types. Tests cover both enum values across the JNI boundary: FirstSeen keeps the first duplicate source row, Fail errors on duplicate source keys and leaves the dataset unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tighten the Fail test to verify the thrown exception carries the "Ambiguous merge inserts are prohibited" cause, not just that some exception is raised. Asserts on the stable message substring only, excluding the volatile file:line suffix and JNI error-class prefix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The existing SourceDedupeBehavior tests only exercised the update path (withMatchedUpdateAll). Add whenMatched=Delete coverage for both plan shapes the Rust core routes through: a full-schema source (Delete + InsertAll) and a key-only delete-only source (Delete + DoNothing). For each, Fail rejects the duplicate source key with the "Ambiguous merge inserts are prohibited" error and leaves the target unchanged, while FirstSeen deletes the matched target row once and skips the duplicate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sezruby
force-pushed
the
feat/java-source-dedupe-behavior
branch
from
August 12, 2026 16:06
cfdfb95 to
d029326
Compare
…dary tests Fail rejects only multiple source rows matching the same target row (an ambiguous update or delete); it does not reject duplicate join keys among unmatched rows. FirstSeen dedupes matched rows and unmatched rows that would otherwise insert the same non-null key more than once. NULL join keys are never duplicates (SQL NULL != NULL). Add unmatched-key boundary tests: Fail inserts both unmatched id=10 rows, FirstSeen collapses them to one, and NULL keys insert twice even under FirstSeen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Expose the Rust core's
source_dedupe_behavior(Fail/FirstSeen) on the JavaMergeInsertParams. TheMergeInsertBuilderin lance-core already supports it, but the Java binding never wired it — Java callers were stuck on the default (Fail) with no way to opt intoFirstSeen.Changes
rust/lance/src/dataset.rs— re-exportSourceDedupeBehaviorfromlance::datasetso the JNI crate can import it alongside the other merge types.MergeInsertParams.java— addSourceDedupeBehavior { Fail, FirstSeen }enum,withSourceDedupeBehavior()builder, getters, andtoString()entry. Defaults toFail, matching the Rust default.merge_insert.rs(JNI) —extract_source_dedupe_behavior()helper (mirrorsextract_when_matched), passed through to the builder.MergeInsertTest.java— cover both enum values across the JNI boundary:FirstSeenkeeps the first duplicate source row;Failerrors on duplicate source keys and leaves the dataset unchanged.Why
A merge source with duplicate join keys (e.g. a CDC stream with multiple updates per key) currently fails on the Java path with no recourse.
FirstSeenis the documented escape hatch in lance-core; this makes it reachable from Java.This is also a prerequisite for wiring native
MergeInsertParamsinto the lance-spark connector, where Spark'sMERGE INTOsemantics require a defined behavior for duplicate source keys.Test
MergeInsertTest: 11/11 pass (9 existing + 2 new)cargo fmt/cargo clippy(lance-jni) clean;cargo check -p lanceclean./mvnw spotless:checkclean🤖 Generated with Claude Code