Repository navigation
[WebGPU] Use Queue.WriteBuffer for small CPU-to-GPU uploads - #33551
Merged
Baiju Meswani (baijumeswani) merged 5 commits intoOct 28, 2026
Merged
Conversation
Use queue-ordered WriteBuffer uploads after flushing recorded readers. Preserve mapped, unaligned, and capture/replay fallbacks, including mapped zero-byte unmapping. Add native correctness, lifetime, ordering, validation, and concurrency coverage; leave the option off by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove environment gating and use Queue.WriteBuffer automatically for eligible uploads through 1 MiB. Preserve mapped and zero-byte handling, ordering/error checks, and staging for larger, unaligned, and capture/replay transfers. Simplify opt-in tests, verify queue/staging routing at size and state boundaries, and document the conservative A100-backed cap and outstanding Metal/D3D12 validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
aciddelgado Could #33491 help on your case? |
Contributor
Author
yes, this looks like a related optimization. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Main advanced during the A100 benchmarks only by changing a macOS workflow timeout. Runtime sources, dependencies and build inputs are identical to the measured main revision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover alternating batched and single uploads with recorded readers. Document controlled A100/Vulkan microbenchmarks and five paired Qwen baseline/width-3 generation runs after batched input uploads, including path/driver submission counters, correctness results, limitations and merge recommendation. Preserve the 1 MiB queue-write limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The default upload-path change needs maintainer confirmation of current-head Windows/D3D12 and macOS/Metal validation alongside the reported Vulkan results.
No open issues 🎉
What changed in this PR
Optimizes WebGPU single CPU-to-GPU uploads with Queue.WriteBuffer for eligible transfers up to 1 MiB, complementing the existing batched-upload optimization from #33491.
Changes:
- Flushes recorded work before queue writes while preserving mapped-buffer and staging fallbacks.
- Adds coverage for upload boundaries, source lifetime, ordering, validation, and mixed batch/single transfers.
- Documents transfer guarantees, the size threshold, and incremental A100/Vulkan benchmark results.
| File | Description |
|---|---|
| onnxruntime/test/providers/webgpu/webgpu_context_test.cc | Adds queue-upload correctness, fallback, and validation tests. |
| onnxruntime/test/providers/webgpu/data_transfer_test.cc | Tests ordering across alternating batched and single uploads. |
| onnxruntime/core/providers/webgpu/buffer_manager.cc | Adds the bounded queue-write path and zero-byte handling. |
| docs/WebGPU_Queue_Uploads.md | Documents behavior, performance evidence, and remaining platform validation. |
Review effort: Balanced
Baiju Meswani (baijumeswani)
approved these changes
Oct 28, 2026
Baiju Meswani (baijumeswani)
deleted the
aciddelgado/webgpu_queue_uploads
branch
October 28, 2026 06:06
Jiajia Qin (qjia7)
left a comment
Jiajia Qin (qjia7)
left a comment
Contributor
There was a problem hiding this comment.
Sorry, just find I have a pending comment and forgot to send.
| @@ -0,0 +1,327 @@ | |||
| # Native WebGPU queue uploads | |||
Contributor
There was a problem hiding this comment.
Could we remove this standalone document from the PR? Intermediate revisions, machine-local reproduction paths, and merge recommendations will become stale rather than serve as maintained WebGPU documentation. Others LGTM, thanks.
Contributor
Author
There was a problem hiding this comment.
Just seeing this sorry, will address in separate PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Automatically use
Queue.WriteBufferfor eligible CPU-to-GPU uploads up to and including 1 MiB.Motivation and Context
Profiling Qwen/DFlash2 on A100 identified substantial host overhead from allocating, mapping, copying, and submitting a fresh staging buffer for each upload. Queue writes let Dawn reuse staging allocations and batch pending transfers with subsequent queue work, without adding CPU completion waits.
The conservative 1 MiB cap is the largest consistently beneficial size measured on A100/Vulkan. Larger transfers can regress, so they retain the existing staging path rather than extrapolating the small-upload improvement.
Validation: On Linux/A100/Vulkan, 80 native tests passed with one unrelated build-mode skip, and 100 GenAI tests passed. Both 512-token baseline and DFlash2 generation runs preserved exact tokens and speculative counters. Formatting and lint checks passed.
Update:
Rebased onto latest main, including merged PR #33491, and reran A100/Vulkan benchmarks to measure the incremental benefit of
Queue.WriteBuffer.Results:
All generated tokens and speculative counters matched. Native and plugin tests passed. Existing batched uploads remain unaffected, and larger transfers retain the staging path.
This confirms that the optimization provides additional performance benefits beyond #33491.