Skip to content

fix(docker): AUT-1080 persist cancellation status - #570

Merged
ko3n1g merged 4 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/docker-cancel-terminal-state
Jul 28, 2026
Merged

fix(docker): AUT-1080 persist cancellation status#570
ko3n1g merged 4 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/docker-cancel-terminal-state

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Background

Megatron-LM PR #6054 detects flaky failures live and cancels its NeMo Run Docker job. Merge-queue validation showed cancellation removed the container without a terminal status. Fork PRs also could not pass the required release rehearsal because GitHub withholds BOT_KEY.

What changed

  • Persist failed terminal status before deleting a cancelled Docker container.
  • Cover cancellation with and without a stored job request.
  • Skip release rehearsal and its summary only for fork-origin PRs.

Details

Trusted branches and same-repository PRs retain release validation. Fork PRs cannot access the required GitHub App private key, so their required summary no longer fails on unavailable secrets.

Companion downstream: NVIDIA/Megatron-LM#6054.

Tested

  • Workflow YAML parsed successfully
  • Ruff format clean
  • Both focused Docker cancellation tests passed
  • Prior full product CI and Codecov green

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@ko3n1g
ko3n1g enabled auto-merge (squash) July 28, 2026 22:16
@ko3n1g
ko3n1g disabled auto-merge July 28, 2026 22:16
@ko3n1g
ko3n1g enabled auto-merge (squash) July 28, 2026 22:16
…erminal-state

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@ko3n1g
ko3n1g disabled auto-merge July 28, 2026 22:28
@ko3n1g
ko3n1g merged commit e393539 into NVIDIA-NeMo:main Jul 28, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants