fix: guard the 3D convolutions the pad-folding pass hides from the conv guard - #4644
Open
tp5uiuc wants to merge 1 commit into
Open
fix: guard the 3D convolutions the pad-folding pass hides from the conv guard#4644tp5uiuc wants to merge 1 commit into
tp5uiuc wants to merge 1 commit into
Conversation
github-actions
Bot
added
component: tests
This was referenced Aug 29, 2026
tp5uiuc
force-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 29, 2026 19:05
1262c9d to
a8de90f
Compare
tp5uiuc
force-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 29, 2026 23:01
a8de90f to
3dd0c1b
Compare
tp5uiuc
commented
Aug 30, 2026
tp5uiuc
force-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
2 times, most recently
from
August 30, 2026 00:52
749d1b9 to
3c19297
Compare
…nv guard
convolution_capability_validator rejects forward 3D convolutions when Turing
(SM 7.5) is a build target, because TensorRT-RTX builds an engine for them whose
createExecutionContext() then returns nullptr. It is registered on
aten.convolution.default only.
The fuse_pad_into_convolution lowering pass rewrites constant_pad_nd -> convolution
into tensorrt::conv_asym_pad, and that op's converter had no capability_validator at
all. By the time partitioning runs there is no aten.convolution node left to
validate, so a padded 3D convolution reaches TensorRT-RTX regardless of the guard
and surfaces at runtime as
Expected exec_ctx_.get() != nullptr to be true but got false
Unable to (re)create TensorRT execution context
This is a production gap, not just a test gap. The pass runs in the normal compile
path, and it fires for any non-transposed, zero-fill, non-negative
constant_pad_nd -> conv pair -- asymmetry is not required -- so any real model with a
padded conv3d hits it on Turing.
* aten_ops_converters.py: extract turing_rejects_forward_convolution() so the
"3D forward conv is unsupported on this target" rule lives in one place.
convolution_capability_validator delegates to it; its behaviour is unchanged,
including failing open when meta["val"] is absent.
* custom_ops_converters.py: add conv_asym_pad_capability_validator and register
it on the fused op. It reads the spatial rank off the argument list rather than
node.meta -- the fused op's args are (source, weight, bias, stride, pre_padding,
post_padding, dilation, groups), so len(stride) is the rank -- which makes the
guard hold whether or not the caller ran the dynamo tracer. The pass never fuses
a transposed convolution, so there is no deconvolution case to handle here. Only
rank 3 is rejected; 2D keeps running on TensorRT.
* test_fuse_pad_into_convolution.py: skip test_padded_conv3d on Turing.
DispatchTestCase has no PyTorch-fallback path -- run_test hands the graph
straight to TRTInterpreter, skipping the partitioner -- so a rejected node
raises UnsupportedOperatorException instead of falling back. Guarding without
skipping would only swap one failure for another.
* test_turing_capability_guards.py: add a padded-conv3d fallback case and a
padded-conv2d positive control. Both key on target_compute_capabilities=[(7, 5)]
rather than the live device, so they exercise the guard on any GPU and therefore
in CI. Reverting just the registration makes the conv3d case fail, so it covers
the gap rather than merely restating it.
Deliberately not fixed in the lowering pass. Declining to fuse when the underlying
convolution would be rejected keeps one rule in one place and would pick up any
future guard on aten.convolution for free, and the pass already receives the
settings it would need. But TestFusePadIntoConvolutionPass builds its graphs with a
default CompilationSettings(), and test_graph_contains_fused_op_after_lowering calls
post_lowering with one too. Default settings mean target_compute_capabilities=None,
which resolves against the current device -- so on a Turing GPU the pass would
decline to fuse and the five tests that assert the fusion *does* happen would fail.
Each would then need a pinned non-Turing target or a skip.
Testing (T4 / SM 7.5 and L40S / SM 8.9, driver 595.58.03, identical pinned stacks):
* All 8 test_padded_conv3d cases go fail -> skip on Turing and still run and pass
on the L40S control, confirming the guard is inert off Turing.
* Full lowering/ sweep on both arms, all 259 tests accounted for on each. On
Turing exactly 8 tests change status, all of them fail -> skip and all of them
test_padded_conv3d: 9 -> 1 failed, 4 -> 12 skipped, and 242 passing either way.
(The other 4 pre-existing skips, and 4 fp16 SDPA tests that abort the worker,
are unrelated and unchanged.) On the non-Turing control every one of the 259
keeps its previous status and the skip count stays at 0.
* The 11 other tests in that file stay green on both arms -- in particular
test_graph_contains_fused_op_after_lowering and
test_fuses_causal_3d_pad, both of which build 3D graphs and assert the fusion
still happens on the T4.
* test_turing_capability_guards.py passes on both arms: T4 17 -> 19 passed
/ 1 skipped, L40S 12 -> 14 passed / 6 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tp5uiuc
force-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 30, 2026 18:37
3c19297 to
2e84e29
Compare
Collaborator
Author
CI summary
Suite ran to completion: Why these are not from this PR: the same |
This was referenced Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What — Registers a capability validator on the
tensorrt::conv_asym_padcustom op so a padded 3Dconvolution falls back to PyTorch on Turing.
Why — The
fuse_pad_into_convolutionlowering pass rewritesconstant_pad_nd -> convolutioninto
tensorrt::conv_asym_pad, erasing theaten.convolutionnode the 3D-conv guard keys on.That converter was registered with no
capability_validator, so a padded conv3d walked straight pastthe guard and hit a null execution context (
TRTEngine.cpp:959). It fires for any non-transposed,zero-fill, non-negative pad, so symmetric cases fail too. This is a production path, not just tests.
How — Share the convolution capability predicate between the aten converter and the fused
custom op, reading rank from the fused node, so the guard survives the rewrite instead of being
erased with the node it keyed on.
Testing — Confirmation sweep, both arms: 8
lowering/failures closed, 0 Turing-specificremaining in that module. No status change on the L40S.
Cost / Gotchas — Accounts for 8 of the 105 tests the guards switch off on Turing.
🤖 Generated with Claude Code