[Jiajia Qin (qjia7)]

Description

• Batch eligible input uploads for built-in and plugin WebGPU EPs, including UMA devices.
• Use staging batches up to 16 MiB while preserving mapped-buffer, single-input, oversized-input, and ineligible-transfer fallbacks.
• Add regression tests for data integrity, batch boundaries, repeated uploads, and plugin submission counts.

Performance

Intel Arc 140V / Vulkan, previous A/B measurement: 44.84 MB of adapter data across 784 tensors: 95.3 → 10.8 ms upload host-call time (~8.8× faster).

Validation

• Built-in and plugin Vulkan builds passed.
• All 6 built-in data-transfer tests and 62 enabled plugin tests passed.
• All seven added regression tests passed; lint and whitespace checks passed.

[Copilot]

🟡 Changes recommended

Built-in batching accepts mapped destinations and bypasses their existing copy-and-unmap fallback, causing validation failures.

1 open issue

• Preflight accepts mapped destinations and causes UploadBatch validation failure · New

What changed in this PR

Extracts WebGPU input-upload batching into a shared implementation for built-in and plugin execution providers.

Changes:

• Groups eligible uploads into staging buffers capped at 16 MiB.
• Adds plugin batching on a shared owning session stream.
• Adds six regression tests for data integrity, boundary cases, and submission counts.

File	Description
onnxruntime/​test/​providers/​webgpu/​plugin/​test_webgpu_allocators.cc	Tests session uploads and submission counts.
onnxruntime/​test/​providers/​webgpu/​data_transfer_test.cc	Tests upload integrity across chunks and repeated writes.
onnxruntime/​core/​providers/​webgpu/​webgpu_provider_factory.cc	Checks plugin batch eligibility.
onnxruntime/​core/​providers/​webgpu/​ep/​sync_stream.h	Declares stream-based batch uploads.
onnxruntime/​core/​providers/​webgpu/​ep/​sync_stream.cc	Routes uploads through the owning provider.
onnxruntime/​core/​providers/​webgpu/​data_transfer.h	Declares built-in batch transfer support.
onnxruntime/​core/​providers/​webgpu/​data_transfer.cc	Routes eligible built-in transfers to batching.
onnxruntime/​core/​providers/​webgpu/​buffer_manager.h	Defines the shared batch-upload interface.
onnxruntime/​core/​providers/​webgpu/​buffer_manager.cc	Implements staging chunks and oversized-upload fallback.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No unresolved findings remain, and regression coverage addresses batching boundaries, data reuse, and preserved fallbacks.

No open issues 🎉

1 resolved since last review

• Preflight accepts mapped destinations and causes UploadBatch validation failure

_{Review effort: Balanced}

[Xiaofei Han (xiaofeihan1)]

Thanks for working on this improvement and sharing the performance numbers! The reduction in upload overhead looks very promising. I have a few questions to better understand the tradeoffs and scope:

1. Could we compare this approach with issuing multiple wgpu::Queue::WriteBuffer calls? My understanding is that Dawn has internal staging/batching optimizations, so these calls may not each require a separate queue submission. It would be helpful to benchmark both approaches on the same workload to see whether we can achieve similar performance with a simpler implementation. Please correct me if I am missing a limitation that makes WriteBuffer unsuitable here.

2. Is the main benefit for Session::Run calls with many CPU-resident inputs, including LoRA adapter parameters? How common is this workload in practice? For LoRA parameters that remain unchanged across runs, an application could potentially upload them once and reuse GPU-resident tensors as explicit inputs or through I/O binding, avoiding repeated copies altogether. I realize that is not a drop-in replacement for the current active-adapter API, but I would appreciate some context on the intended use cases and why the shared staging-buffer approach is worth the additional complexity compared with that option or WriteBuffer.

3. Could you also confirm that initializer/weight uploads during session creation are a separate issue and are not batched by this change? From the current loading path, it looks like weights are still uploaded through individual CopyTensor calls, so reducing the number of flushes there would require a separate change. Is that understanding correct?

Thanks again for the optimization and for helping clarify these points!

[Jiajia Qin (qjia7)]

I compared both approaches locally on Intel Arc 140V with Vulkan. Here are the results and scope clarifications:

1. WriteBuffer comparison. Dawn does manage staging internally, and WriteBuffer does not inherently require one submission per tensor. The explicit-submit variant in my benchmark calls WriteBuffer for each tensor, then submits once for the whole group and waits for completion.

   WriteBuffer is competitive for small uploads and sometimes faster. For fresh, unmapped destinations, upload-through-completion medians were:

   Buffer size	Current staging path	WriteBuffer + Submit
   1 MiB	0.40 ms	0.26 ms
   4 MiB	1.79 ms	1.42 ms
   64 MiB	17.07 ms	22.53 ms
   128 MiB	44.18 ms	57.89 ms

   These timings exclude destination creation. The large-buffer cases use the existing individual staging fallback, since one tensor exceeds the 16 MiB batching limit. Staging won for these fresh large buffers, but this is not a universal size rule: reused large buffers were comparable, and some pending-work cases favored WriteBuffer. For the actual 784-tensor adapter workload, end-to-end prefill performance was also close between the approaches.

   There is an integration difference: our explicit copies are encoded into the session’s command encoder for session-bound transfers. WriteBuffer operates directly on the queue, so the prototype first flushes older recorded session work to preserve ordering. Our current uploader also flushes after each staging chunk; it does not yet combine uploads with subsequent computation in one command buffer. Keeping this design preserves that future option, rather than demonstrating an existing benefit from such coalescing. Given the fresh-large-buffer results and recording flexibility, I prefer retaining the current implementation for this PR.

2. Intended workload and GPU-resident reuse. The target is ordinary Run execution with many CPU-resident inputs that need to reach the GPU. Our LoRA workload has 784 adapter tensors totaling 44.84 MB; I do not have broader workload-prevalence data.

   Uploading unchanged weights once and reusing GPU-resident inputs is preferable when the application can manage it. The current WebGPU active-adapter path retains CPU-resident parameters, so this is not a drop-in replacement. Persistent GPU adapter caching is a good optimization suggestion for a separate follow-up. It is complementary and does not affect this PR’s optimization of transfers that are still required for CPU-resident inputs.

   Captured replay can bypass ordinary input copies, so this should not be interpreted as an upload on every decode step.

3. Initializer uploads. Your understanding is correct: session-creation initializer uploads still use individual CopyTensor calls and are unchanged by this PR. A follow-up could reuse UploadBatch, but would need to collect transfers and preserve source lifetimes and ordering. It should also preserve the existing direct mapped-write path for read-only UMA initializers. I suggest addressing initializer batching in a separate follow-up.

[Xiaofei Han (xiaofeihan1)]

Thanks for verifying this! LGTM, approved.