PiPNN 2/6: add numerical kernels

[weiyaoluo (SeliMeli)]

PiPNN (Pick-in-Partitions Nearest Neighbors) builds ANN graph candidates with overlapping partitions and dense matrix work instead of running beam search against a partially built graph for every inserted point. This numerical layer adds the kernels used by later PiPNN stages. It does not yet build or persist a graph.

PiPNN now lives under diskann::graph::pipnn; there is no separate implementation crate. This keeps graph construction beside DiskANN graph policy and lets later layers reuse crate-private graph internals without publishing them.

Concepts

• A leader is a sampled point that names a child partition.
• fanout is the number of nearest leaders retained for each point, creating overlapping child partitions.
• A leaf is a bounded partition processed with one lower-triangular all-pairs dot-product matrix.
• Leaf k is the number of local companions retained per point; it is construction policy, not final graph degree R.

Code map

1. diskann-linalg::sgemm_aat_lower computes A · Aᵀ and writes only the lower triangle. Callers may leave the upper triangle uninitialized.
2. diskann/src/graph/pipnn/kernel_metric.rs owns concrete metric marker types, formulas, scale units, and zero/NaN behavior shared by both kernels; runtime dispatch remains outside the numerical layer.
3. partition_kernel.rs converts point-by-leader dot products into sorted nearest leader IDs. Metric-specific scale handling happens before insertion into a caller-owned, runtime-sized tracker.
4. leaf_kernel.rs scans each strict-lower-triangle pair once and updates both endpoint top-k trackers. It accepts k <= 3 and uses fixed-size insertion; larger widths are rejected.
5. partition_kernel.rs and leaf_kernel.rs co-locate private seam tests with independent formula differentials; diskann-linalg/src/lib.rs similarly owns the lower-triangle GEMM tests.

End-to-end flow

Caller selects concrete architecture A and metric marker M → computes dense dot products → typed kernel input validates matrix/scales/output → scalar/SIMD chunks convert dots to metric distances → stable top-k insertion writes caller-owned IDs/neighbors. #1290 performs that selection once for the complete graph build. This layer exposes no runtime kernel convenience wrapper, visitor, trait object, or stored function pointer.

The kernels do not own providers, graph IDs, recursion, edge merging, thread pools, persistence, or search.

Invariants and boundaries

• MatrixView owns backing-shape validation; kernels validate cross-view row counts, scale layouts/lengths, fanout versus leader count, output width, and ID representation.
• Partition output contains leader-local u32 positions; leaf output contains leaf-local target positions.
• Leaf traversal reads only the diagonal and strict lower triangle, updating each unordered pair exactly once.
• Ties preserve encounter order. NaN candidates are non-rankable; finite f32::MAX remains rankable.
• Cosine zero/subnormal norms produce zero similarity without erasing unrelated NaN behavior.
• Scalar tails and dispatched chunks preserve the documented rounding/order contract.
• PiPNN names no ISA, target feature, or raw architecture intrinsic; dispatch remains owned by diskann-wide.

Review path

1. Start with kernel_metric.rs: metric formulas, scale kinds, zero thresholds, NaN handling, and scalar equivalents.
2. Review partition_kernel::nearest_leaders validation and tracker insertion, then compare scalar tails with SIMD chunks.
3. Review leaf_kernel::nearest_neighbors lower-triangle traversal, dual-endpoint updates, fixed-width top-k paths, and workspace reuse.
4. Verify sgemm_aat_lower never touches the upper triangle.
5. Finish with the co-located independent differentials around 4/8/16-lane and second-chunk boundaries.

Validation

• 33 co-located PiPNN kernel tests at this layer: 11 private seam tests plus 22 interface differentials; seven lower-triangle GEMM tests live beside sgemm_aat_lower.
• Differential coverage spans all four metrics, k/fanout paths, dimensions around lane boundaries, tails, ties, NaN, infinities, signed zero, zero/singleton/capacity inputs, and validation failures.
• x86-64 baseline and AVX-512 SDE jobs explicitly enable the workspace feature pipnn; existing feature-gated CI and nightly jobs include pipnn,testing.
• AArch64 and Windows cross-target checks compile the feature.

Stack relation

Stack 2/6. Depends on #1315, which extracts crate-private shared RobustPrune. This layer supplies numerical selection; #1290 adds partition/leaf orchestration using both lower layers.

Stack 2/6: #1315 → #1290

[Copilot]

Pull request overview

This PR adds the first set of PiPNN “kernel” building blocks to the DiskANN Rust workspace: SIMD-accelerated top‑k selection for partition assignment and leaf neighbor selection, along with supporting SIMD division and a new lower-triangular A·Aᵀ helper in diskann-linalg.

Changes:

• Add a new diskann-pipnn crate with partition_kernel and leaf_kernel implementations plus extensive correctness tests and Criterion benchmarks.
• Extend diskann-wide to support Div on relevant f32 SIMD types (native, doubled, and scalar/emulated) and add a corresponding division test macro.
• Add diskann_linalg::sgemm_aat_lower (lower-triangle-only AAT) and wire new crate/tests/CI/mutants exclusions into the workspace.

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
diskann-wide/src/test_utils/ops.rs	Adds test_div! macro to validate lane-wise SIMD division correctness.
diskann-wide/src/emulated.rs	Adds Div for scalar/emulated Emulated<f32, N, A> to support division in scalar dispatch.
diskann-wide/src/doubled.rs	Adds Div for Doubled<T> to support composite SIMD widths.
diskann-wide/src/arch/x86_64/v4/f32x8_.rs	Adds AVX Div op mapping + division tests.
diskann-wide/src/arch/x86_64/v4/f32x4_.rs	Adds SSE Div op mapping + division tests.
diskann-wide/src/arch/x86_64/v4/f32x16_.rs	Adds AVX-512 Div op mapping + division tests.
diskann-wide/src/arch/x86_64/v3/f32x8_.rs	Adds AVX Div op mapping + division tests for V3.
diskann-wide/src/arch/x86_64/v3/f32x4_.rs	Adds SSE Div op mapping + division tests for V3.
diskann-wide/src/arch/x86_64/v3/f32x16_.rs	Adds division tests for the f32x16 V3 path (likely via doubled composition).
diskann-wide/src/arch/aarch64/f32x4_.rs	Adds Neon Div op mapping + division tests.
diskann-wide/src/arch/aarch64/f32x2_.rs	Adds Neon Div op mapping + division tests.
diskann-pipnn/tests/partition_kernel.rs	New integration tests for partition top‑k dispatch correctness and edge cases.
diskann-pipnn/tests/leaf_kernel.rs	New integration tests for leaf neighbor top‑k dispatch correctness and edge cases.
diskann-pipnn/src/partition_kernel/tests.rs	New unit tests comparing scalar reference vs runtime dispatch and metric contracts.
diskann-pipnn/src/partition_kernel.rs	New partition-assignment distance + top‑k kernel with validation and SIMD dispatch.
diskann-pipnn/src/lib.rs	New crate root exporting PiPNN kernel modules.
diskann-pipnn/src/leaf_kernel/tests.rs	New unit tests for scalar reference parity and workspace behavior.
diskann-pipnn/src/leaf_kernel.rs	New fused lower-triangle leaf neighbor kernel with SIMD dispatch and workspace support.
diskann-pipnn/Cargo.toml	Defines new diskann-pipnn crate, dev-deps, and benches.
diskann-pipnn/benches/kernels.rs	Adds benchmarks for partition top‑k, lower AAT, leaf top‑k, and full leaf workflow.
diskann-linalg/tests/sgemm_aat_lower.rs	New tests for lower-triangle AAT behavior and validation errors.
diskann-linalg/src/lib.rs	Adds public sgemm_aat_lower API with dimension checks.
diskann-linalg/src/faer.rs	Implements sgemm_aat_lower_impl using Faer triangular matmul.
Cargo.toml	Adds diskann-pipnn to workspace members and workspace dependencies.
Cargo.lock	Records the new diskann-pipnn package entry.
.github/workflows/ci.yml	Adds diskann-pipnn to CI test package lists.
.cargo/mutants.toml	Adds mutation-test exclusions for kernel code paths and equivalent transformations.

Comments suppressed due to low confidence (2)

diskann-pipnn/src/leaf_kernel.rs:651

• Same issue as the L2 arm: using max_simd for lower clamping can erase NaNs on the Scalar/Emulated backend, making NaN distances rankable. Clamp with lt_simd + select to preserve NaNs consistently.

        Metric::CosineNormalized => {
            let distance = F::splat(arch, 1.0) - dot;
            zero.max_simd(distance)
        }

diskann-pipnn/src/leaf_kernel.rs:664

• The cosine path also uses zero.max_simd(distance) for clamping, which can collapse NaNs to zero on the Scalar/Emulated backend (via f32::max). That contradicts the comment about preserving non-rankable NaNs and can change output ordering. Prefer an lt_simd + select clamp here as well.

            let distance = one - cosine;
            // Comparisons with NaN are false, so this explicit lower clamp
            // preserves non-rankable NaNs while matching the existing PiPNN
            // distance formulas for finite values.
            zero.max_simd(distance)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated no new comments.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.28%. Comparing base (c2e6be6) to head (12c2c2c).

Additional details and impacted files

@@                     Coverage Diff                     @@
##           pipnn-stack/02-final-prune    #1287   +/-   ##
===========================================================
  Coverage                       91.27%   91.28%           
===========================================================
  Files                             517      517           
  Lines                           98541    98624   +83     
===========================================================
+ Hits                            89945    90027   +82     
- Misses                           8596     8597    +1     

Flag	Coverage Δ
miri	91.28% <100.00%> (+<0.01%)	⬆️
unittests	90.95% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-linalg/src/faer.rs	100.00% <100.00%> (ø)
diskann-linalg/src/lib.rs	99.72% <100.00%> (+0.03%)	⬆️
diskann-utils/src/views.rs	100.00% <100.00%> (ø)
diskann-wide/src/arch/x86_64/v3/f32x16_.rs	100.00% <ø> (ø)
diskann-wide/src/arch/x86_64/v3/f32x4_.rs	100.00% <ø> (ø)
diskann-wide/src/arch/x86_64/v3/f32x8_.rs	100.00% <ø> (ø)
diskann-wide/src/arch/x86_64/v4/f32x16_.rs	14.11% <ø> (ø)
diskann-wide/src/arch/x86_64/v4/f32x4_.rs	16.90% <ø> (ø)
diskann-wide/src/arch/x86_64/v4/f32x8_.rs	16.90% <ø> (ø)
diskann-wide/src/doubled.rs	86.89% <100.00%> (+0.17%)	⬆️
... and 2 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

diskann-pipnn/src/partition_kernel/tests.rs:20

• The PartitionTopK contract for Metric::L2 expects leader_scales to contain squared leader norms (see docs and distance(Metric::L2, ..) test). This helper currently populates unsquared norms, which makes the test data inconsistent with the public API contract and could hide contract-related bugs.

    let leader_scales = match metric {
        Metric::L2 => (0..leaders).map(|leader| (leader + 1) as f32).collect(),
        Metric::Cosine => (0..leaders)
            .map(|leader| {

diskann-pipnn/src/partition_kernel.rs:61

• InvalidFanout’s error message says the maximum is {maximum}, but validation also rejects fanout > leaders. When leaders < maximum this message is misleading (it implies the only limit is {maximum}). Consider spelling out both constraints in the message so callers immediately see why it failed.

    #[error("invalid fanout {fanout} for {leaders} leaders; maximum is {maximum}")]

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Suppressed comments (1)

diskann-pipnn/src/partition_kernel.rs:294

• For Metric::Cosine, NaN norms currently produce a finite distance (1.0) because denominator.gt_simd(0) is false for NaN, so the lane falls back to cosine = 0. That makes NaN-derived pairs/leaders “rankable”, which contradicts the module’s stated NaN-rejection behavior and differs from diskann-vector cosine semantics (NaN norms propagate to a NaN similarity/distance). Consider explicitly preserving NaN denominators so the resulting distance stays NaN and is ignored by insert_topk.

        let denominator = row_norm * leader_norm;
        let valid = denominator.gt_simd(zero);
        let safe_denominator = valid.select(denominator, one);
        let cosine = valid.select(dot / safe_denominator, zero);
        one - cosine

[juchen-ms (partychen)]

Nice work overall. I found one correctness issue in the cosine handling that should be resolved before merge. The remaining comments are mostly about reducing duplicated or unsafe code and tightening the API contracts.

[juchen-ms (partychen)]

checked_area, check_length, ShapeOverflow and InvalidBufferLength are duplicated character-for-character with leaf_kernel. Small enough to shrug at now, but with four more PRs coming it's probably worth a src/shape.rs with a shared ShapeError that each kernel error wraps via #[from].

[weiyaoluo (SeliMeli)]

I kept the two tiny checked-area/length adapters local because they construct different public kernel error types and sit immediately before each module's unsafe accesses. MatrixView adoption removed the other duplicated shape state; introducing a shared wrapped error would enlarge the public error interface for two call sites.

[Aditya Krishnan (arkrishn94)]

Thanks Weiyao, this is progress from the previous mega-PR. I still have some big-picture comments (we covered most of these offline) -

• Documentation: As I mentioned, we need thorough documentation in the diskann-pipnn crate. The main modules, partition_kernel and leaf_kernel need documentation up top, highlighting the main structures and how they are used - e.g. process_rows_binary/unary and nearest_leaders. Similarly with process_pairs_simd_* and nearest_leaf_neighbors
• Testing: I am concerned about the lack of testing for partition_kernel.rs and leaf_kernel.rs.
  • I notice some e2e integration tests but these kernels should be thoroughly tested, sweeping different input parameters, architectures and edge cases. This is especially needed given the amount of unsafe code.
  • That brings me to miri - there should be miri tests too.
  • I'm curious why are the tests in a separate submodule to the main files (for partition_kernel.rs and leaf_kernel.rs)? Let's try to keep tests along with the code being tested.
• Criterion: Since criterion is not a standard part of our library for benchmarking, let us not introduce it for this crate.
• Kernel dispatch: I left comments about you're disptaching the kernels, please take a look.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (2)

diskann-pipnn/src/partition_kernel/tests.rs:19

• PartitionTopK::leader_scales is documented as "squared leader norms for L2" (and cosine uses unsquared norms), but this test helper feeds unsquared values for the L2 case. That makes the test data inconsistent with the public contract and can mask mistakes in distance computation. Consider squaring the L2 norms here so the tests exercise the intended inputs.

    let leader_scales = match metric {
        Metric::L2 => (0..leaders).map(|leader| (leader + 1) as f32).collect(),
        Metric::Cosine => (0..leaders)

diskann-pipnn/src/partition_kernel.rs:252

• For the L2 path, the SIMD chunk uses mul_add_simd (fused multiply-add) but the scalar tail uses norm - 2.0 * dot (non-fused). This can introduce small rounding differences between SIMD and tail elements, which can change ordering/tie behavior right at SIMD-width boundaries. Use f32::mul_add for the scalar tail so both paths compute the same value shape.

                |dot, norm| F::splat(arch, -2.0).mul_add_simd(dot, norm),
                |dot, norm| norm - 2.0 * dot,

[weiyaoluo (SeliMeli)]

weiyaoluo (@SeliMeli) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

[weiyaoluo (SeliMeli)]

Simplification follow-up in acecad1d: removed the one-field LeafInput wrapper and pass MatrixView<f32> directly. MetricVisitor and prepared ISA/metric function-pointer dispatch are unchanged.

Validation: 33 kernel tests, all-target Clippy, and PiPNN doctests pass.

[weiyaoluo (SeliMeli)]

Follow-up simplification in 64c2a811:

• k is capped at 3; dynamic-width storage and insertion were deleted;
• invalid widths return LeafKernelError rather than reaching panic!/unreachable!;
• MatrixBase now rejects shape multiplication overflow, so kernels trust MatrixView/MutMatrixView backing-length invariants;
• duplicate matrix-area validation and associated partition error path were removed;
• kernel module docs now retain contracts, numerical invariants, safety gates, and complexity without repeating PR narrative.

A generic fixed-row bubble insertion was measured and rejected: local Callgrind k=3 estimated cycles were +0.51247% versus explicit 1/2/3 shifts. The submitted fixed shifts remain, with no dynamic-width path.