[Issue]: workloads are failing on scheduling pytorch or any other

[codetap-developer]

Problem Description

workloads are failing on scheduling pytorch or any other

after login into session workloads are failing when we schedule it one simple message failed

Operating System

ubuntu

CPU

AMD

GPU

355

ROCm Version

7.2.1

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[powderluv]

Thanks for reporting this. To help us investigate, could you provide:

1. Which container image are you using? (e.g., rocm/pytorch:rocm6.3_ubuntu22.04_py3.10_pytorch_release_2.4.0)
2. What command/workload fails? (e.g., the exact Python script or command)
3. Error output — what does the terminal show when the workload fails?
4. Session state — does the session reach "running" state, or does it fail during scheduling?

If the session starts successfully but PyTorch fails inside, it may be a container image issue (missing libraries, wrong ROCm version, etc.). If the session itself fails to start, please share the error message from the session detail page.

We've also made several improvements in this release that may help:

• rocm-smi Not Consistently Available Inside Container #38: ROCm tools (rocm-smi, etc.) are now always in PATH inside sessions
• GPU Utilization Count Inaccurate — Scheduler Miscounts Available GPUs #41: GPU capacity counting is now more accurate (excludes DOWN/DRAINED nodes)
• [Issue]: No GPU Access Control to limit use #36: Per-user GPU quotas are now available to prevent resource exhaustion

[codetap-developer]

Container Image:
Not specific to a single image (observed across sessions)

Workload / Command:
N/A — issue occurs at session access level (SSH/websocket), not tied to a specific workload

Error Output:
Users encounter a "connection lost" error. Refreshing the page does not restore the connection.

Session State:
The session successfully reaches the "running" state, and the workload continues to run in the backend.

Additional Notes / Workaround:
This appears to be an intermittent SSH/websocket connectivity issue. Currently, users are advised to:

Terminate the affected session
Reschedule a new workload
Retry until SSH access is successfully established

The issue does not impact job execution but affects user access to the running session for that workload