Skip to content

test(cuda.core): isolate pending-call saturation cleanup#2406

Merged
Andy-Jost merged 3 commits into
NVIDIA:mainfrom
Andy-Jost:fix-pending-call-saturation-test
Jul 23, 2026
Merged

test(cuda.core): isolate pending-call saturation cleanup#2406
Andy-Jost merged 3 commits into
NVIDIA:mainfrom
Andy-Jost:fix-pending-call-saturation-test

Conversation

@Andy-Jost

@Andy-Jost Andy-Jost commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Isolate the CPython pending-call saturation probe so its process-global queue cannot contaminate other tests running concurrently under free-threaded Python. Explicitly drain the synthetic no-op callbacks before exercising cuda.core's existing retry path.

Changes

  • Run test_pending_call_queue_saturation_preserves_cleanup in a subprocess using the current Python executable.
  • Call Py_MakePendingCalls after saturating the queue and before creating the second graph attachment that retries cleanup scheduling.
  • Keep the cleanup-completion and main-thread-finalization assertions inside the isolated process.

Test Coverage

  • python -m py_compile cuda_core/tests/graph/test_graph_definition_lifetime.py
  • pre-commit run --all-files
  • Full CI passed all ten Python 3.14t rows across Linux x86-64, Linux arm64, and Windows.
  • Targeted Python 3.14t reruns passed on Linux x86-64 (T4), Linux arm64 (L4), and Windows (A100).

Related Work

Run the process-global queue probe in a subprocess and explicitly drain its synthetic callbacks so parallel free-threaded tests cannot contaminate one another.
@Andy-Jost Andy-Jost added this to the cuda.core 1.2.0 milestone Jul 22, 2026
@Andy-Jost Andy-Jost added bug Something isn't working P0 High priority - Must do! cuda.core Everything related to the cuda.core module labels Jul 22, 2026
@Andy-Jost Andy-Jost self-assigned this Jul 22, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@Andy-Jost

Copy link
Copy Markdown
Contributor Author

/ok to test

@github-actions

This comment has been minimized.

@mdboom

mdboom commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Driveby comment: We could add a dependency on pytest-forked and it could handle running the test in a separate process.

Pass the finalization timeout into the child so process isolation does not depend on importing test helpers from the temporary working directory.
@Andy-Jost

Copy link
Copy Markdown
Contributor Author

/ok to test

@Andy-Jost

Copy link
Copy Markdown
Contributor Author

Driveby comment: We could add a dependency on pytest-forked and it could handle running the test in a separate process.

Oh, it looks like pytest-forked uses os.fork (not supported by CUDA) and also doesn't work on Windows.

@Andy-Jost
Andy-Jost marked this pull request as ready for review July 22, 2026 21:49
@Andy-Jost
Andy-Jost requested a review from mdboom July 22, 2026 21:52
Preserve subprocess isolation while adopting the latest safe-close retry behavior for pending cleanup.

@mdboom mdboom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Understood that pytest-fork won't work. But we should probably (as a follow-on) refactor the ~4 or so tests that do this to use a fixture to run in a subprocess. See #2412.

@Andy-Jost
Andy-Jost merged commit 55cb344 into NVIDIA:main Jul 23, 2026
195 of 201 checks passed
@Andy-Jost
Andy-Jost deleted the fix-pending-call-saturation-test branch July 23, 2026 21:06
@github-actions

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: graph-lifetime tests flaky on free-threaded 3.14t jobs since #2371

2 participants