Skip to content

AArch64: one reduction BitMask, FCVTNS Convert - #19

Open
Auburn wants to merge 2 commits into
masterfrom
perf/aarch64-conversions
Open

Auburn wants to merge 2 commits into
masterfrom
perf/aarch64-conversions

Conversation

@Auburn

@Auburn Auburn commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Change

Two commits, both AArch64 only (ToolSet/ARM/128/m32x4.h and ToolSet/ARM/NEON.h). The NEON level is unchanged.

  1. BitMask weights each lane by its bit and adds across. 4 lanes: and with the weights + addv, instead of ushr + ushl + addv. 8 lanes (2 × native, which the 512 and 1024 bit composites also go through): uzp2 narrows both halves into one register of 16 bit lanes, then one and + addv instead of a reduction per half.
  2. Convert f32 → i32 with FCVTNS, for every compiler. vcvtq_s32_f32( Round( a ) ) is FRINTN + FCVTZS with GCC 13 and clang 15 and 18. vcvtnq_s32_f32 is a single FCVTNS that rounds to nearest even with the same results (ties to even, saturation, NaN → 0).

Like AnyMask (vmaxvq), the new BitMask expects each lane to be all ones or all zeros, which is what comparisons and mask operations produce. It reads one bit per lane, not the sign bit.

Changes since the first version of this PR:

  • The 8 lane AnyMask overload is gone. Since Composite AnyMask: test the OR of both halves #15 the generic composite AnyMask is the same AnyMask( a.v0 | a.v1 ), and the generated code is identical without it.
  • The NEON level no longer has its own 8 lane BitMask. That branch compiled to the same code as the generic composite.
  • Convert no longer keeps clang on Round + vcvtq. Only recent clang fuses that pair into FCVTNS (23 does, 15 and 18 don't), so the exception left older clang on two instructions.

Why

Per call on AARCH64, -O3 -ffast-math, after a compare. Not counting loads, the compares or the weights load:

FastSIMD master This PR
4 lane BitMask, GCC 13 ushr, ushl, addv, fmov and, addv, fmov
4 lane BitMask, clang 18 movi, and, ushl, addv, fmov and, addv, fmov
4 lane BitMask, clang 23 and, addv, fmov same
8 lane BitMask, GCC 13 9 (2 × ushr/ushl/addv/fmov, orr) 4 (uzp2, and, addv, umov)
8 lane BitMask, clang 18 10 (movi, 2 × and/ushl/addv/fmov, orr) 4 (uzp2, and, addv, fmov)
8 lane BitMask, clang 23 7 (2 × and/addv/fmov, orr) 4 (uzp2, and, addv, fmov)
Convert, GCC 13 and clang 18 frintn, fcvtzs fcvtns
Convert, clang 23 fcvtns fcvtns
Convert( Floor ), GCC 13 frintm, frintn, fcvtzs frintm, fcvtns
Convert( Floor ), clang 18 frintm, fcvtzs frintm, fcvtns
Convert( Floor ), clang 23 fcvtms frintm, fcvtns
Convert( Round ), clang 23 fcvtns frintn, fcvtns

The last two rows are the cost: clang 23 doesn't fuse a rounding into the intrinsic, so Convert( Floor( x ) ) or Convert( Round( x ) ) takes one more instruction when the rounded float isn't also used. A float → int floor function (vcvtmq_s32_f32) would fix that for every compiler.

FastNoise2 master (c47a7b9), FastSIMD_FastNoise AARCH64 object, instruction count:

FastSIMD master This PR
GCC 13 71,456 71,155: 240 FRINTN + FCVTZS become FCVTNS
clang 18 50,541 50,484: the same 240
clang 23 52,059 51,978: 120 FCVTMS become FCVTNS, no extra FRINTM (the floored values are also used as floats), the rest is register allocation

The NEON objects are identical for all three compilers. Nothing in FastNoise2 master calls BitMask on ARM.

Testing

  • FastSIMD tests, Release, cross compiled for AArch64 Linux and run under qemu-aarch64, with GCC 13, clang 18 and clang 23. Full suite at 65,536 inputs for seeds 24301, 1 and 2: 628 passed, 88 known failures, 0 failed, output identical to master in all nine runs. The mask and convert tests also ran at the full count (4,194,304) and pass on all three compilers.
  • As a check, swapping two weights fails m32 and m32i bit mask to int on AARCH64: at 128 bit for the 4 lane version, at 256, 512 and 1024 bit for the 8 lane version.
  • x86 is unaffected, since these headers are ARM only.
  • Not built with MSVC for ARM64. MSVC 14.51's arm64_neon.h defines every intrinsic used.
  • Not timed on hardware. The M1 timings in the first version measured the AnyMask change (now in Composite AnyMask: test the OR of both halves #15) together with BitMask.

Not in this PR

Convert( Floor( x ) ) is FRINTM + FCVTNS where FCVTMS would do. That needs a float → int floor function (vcvtmq_s32_f32). FastNoise2's performance branch currently does this on its own side.

4 lanes: AND with the lane weights + ADDV instead of USHR + USHL + ADDV.
8 lanes (2 x native, also used by the 512 and 1024 bit composites): UZP2
narrows both halves into one register of 16 bit lanes, then one AND +
ADDV instead of a reduction per half (GCC 13: 9 instructions to 4).

Both read one bit per lane rather than the sign bit, so like AnyMask
they rely on each mask lane being all ones or all zeros. The NEON level
keeps the existing versions. Covered by the m32/m32i bit mask to int
tests.
Round + vcvtq_s32_f32 is FRINTN + FCVTZS with GCC 13 and clang 15/18;
vcvtnq_s32_f32 rounds to nearest even in one instruction with the same
results (ties to even, saturation, NaN -> 0).

Clang 23 already fuses the pair into FCVTNS, but through the intrinsic
it no longer fuses a preceding Floor into FCVTMS (or Round into FCVTNS).
That only costs an instruction when the rounded float isn't used too:
in FastNoise2 master clang 23 swaps all 120 FCVTMS for FCVTNS with no
extra FRINTM. The NEON level keeps Round + vcvtq.
@Auburn
Auburn force-pushed the perf/aarch64-conversions branch from 2e06e5a to 0275c69 Compare October 17, 2026 16:53
@Auburn Auburn changed the title ARM: single reduction for 8 lane masks, FCVTNS Convert for GCC and MSVC AArch64: one reduction BitMask, FCVTNS Convert Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant