Repository navigation
cuda: sigmoidf_, softplusf_, silu and silu_grad in common/act.cuh, SASS unchanged - #1556
Merged
Merged
Conversation
This was referenced Oct 28, 2026
…rp_sum, SASS unchanged The three remaining float warp_sum copies (xor butterfly 16..1) give way to common/reduce.cuh through a using declaration, as in ashhart#1440. qwen4_gdn.cu is the checked-in text Gdn.cuda generates from flashnext_gdn.cu (source_test), so both change together. qwen4_gdn_cuda_host.zig, which builds that text for the CPU, now strips the include and the using declaration instead of the deleted function. flashnext_gdn.cu's gradient-serving pin takes the new bytes; flashnext_batch.cu's two pins already differ on main and are left as they are. cuobjdump -sass is identical for every function of the three images (65 functions, sm_80/86/89/120/121) with nvcc 13.0 and 13.2, base prims-1409.
Step 3 of ashhart#1409. The text-identical copies move to tf_common in one header: - fp4pair (E2M1 pair to bf16x2): qmm_frag.cuh, prompt16, nvfp4_experts and nvfp4_experts_tile; - fp8pair (two adjacent E4M3 bytes to bf16x2): drama3_fp8 and fp8_experts; - e4m3f (E4M3 scale to float): nvfp4_experts and nvfp4_experts_tile. Each file takes them by a using-declaration, so no call site changes. qmm_frag keeps qmm_frag::fp4pair the same way. Left out because their text differs: qmmf's fp8pair (bytes spread first) and e4m3f(uint8_t), and prompt16's fp8pair with a unit argument. cuda_sass_identity.py's fp4pair-constant control now plants in common/fp_pairs.cuh, where the constant lives.
…ytes fp_pairs.cuh leaves Both pins matched main before this branch. fp8_experts.cu's and nvfp4_experts.cu's pins already differ on main and are left as they are, as ashhart#1550 does for flashnext_batch.cu.
…SS unchanged Step 4 of ashhart#1409. The same-text activation copies move to tf_common in one header: - sigmoidf_: gdn_train, drama3_common.cuh, glm_kda, and the file-local sigm of attn_train, train_attn, flashnext_train, qwen27_train and train_q4 (renamed at their call sites); - softplusf_: gdn_train and drama3_common.cuh; - silu: flashnext_train, qwen27_train, train_q4 and train; - silu_grad: gdn_train, flashnext_train, qwen27_train, train_q4 and train (train's spelled the sigmoid inline). Each file takes them by a using-declaration. glm_kda.cu's gradient-serving pin takes the new bytes. Left out: flashnext_gdn, qwen4_gdn and flashnext_batch (ashhart#1550 edits them and qwen4_gdn's CPU twin), qwen4_gdn's stable sigmoid, act_l2's __fdiv_rn silu, drama3's siluf_ and train's softplus (other names used elsewhere or not at all), and deepseek_v41/pointwise's torch silu.
ashhart
force-pushed
the
prims-1409-step4
branch
from
October 29, 2026 12:21
f193363 to
c075441
Compare
This was referenced Oct 29, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Step 4 of #1409. It is independent of #1550 and #1555, with no shared files. It is a pure refactor:
cuobjdump -sassis identical for every function of every image that includes a touched file.The activation copies whose text is the same move into
tf_commonin a new header,common/act.cuh:sigmoidf_(1 / (1 + expf(-x))):gdn_train.cu,drama3_common.cuhandglm_kda.cu. It also replaces the file-localsigmofattn_train.cu,train_attn.cu,flashnext_train.cu,qwen27_train.cuandtrain_q4.cu, which is the same function under another name, renamed at its call sites in those files;softplusf_(x > 20 ? x : log1pf(expf(x))):gdn_train.cuanddrama3_common.cuh;silu(v / (1 + expf(-v))):flashnext_train.cu,qwen27_train.cu,train_q4.cuandtrain.cu;silu_grad:gdn_train.cu,flashnext_train.cu,qwen27_train.cu,train_q4.cuandtrain.cu.train.cuwrote the sigmoid inline.Each file takes them through a
usingdeclaration.glm_kda.cu's gradient-serving pin takes the new bytes.11 files (with the new header), +52 / −40.
Left alone on purpose
flashnext_gdn.cu,qwen4_gdn.cuandflashnext_batch.cualso holdsigmoidf_andsoftplusf_copies. cuda: flashnext_batch, flashnext_gdn and qwen4_gdn take tf_common::warp_sum, SASS unchanged #1550 edits these files andqwen4_gdn's CPU twin, so they come after it lands.qwen4_gdn.cu'ssigmoidf_is the overflow-safe form (expf(-fabsf(x))).act_l2.cuh'ssiludivides with__fdiv_rn.drama3'ssiluf_andtrain.cu'ssoftpluskeep their own names.siluf_is used across the drama3 files, andtrain_mixers.cuhcallssoftplus.deepseek_v41/pointwise.cu'ssiluis documented as torch's, in its own namespace.Verification
dd0a9cc5e, headf19336386, nvcc 13.0, sm_80/86/89/120/121. The 9 images are attn_train, drama3, flashnext_train, gdn_train, glm_kda, qwen27_train, train, train_attn and train_q4. All 635 functions: 0 changed, 0 added, 0 removed, 0 unstable (tools: cuda_sass_identity.py rebuilds a differing image and fails on an unstable build #1532's rebuild check).warp_sumbutterfly changes 250 functions in 9 of 9 images. Thefp4paircontrol cannot be planted, because none of these images includescommon/qmm_frag.cuh. The tool exits 1 on that line alone.zig build test -Dnvcc=/usr/local/cuda/bin/nvcc: passes (RC 0).Part of #1409.