Repository navigation
Vulkan (cupcake) CI: xe copy-engine reset of a kernel-submitted job kills parallel tests on the Arc B570 #2192
Description
The Vulkan (cupcake) CI job fails nondeterministically on the Arc B570. The xe driver resets the copy engine after one of its own internal jobs times out, and the B570 watchdog then kills whatever chipStar tests hold the GPU. It happened twice on 2026-10-29, in runs for two different PRs:
https://github.com/CHIP-SPV/chipStar/actions/runs/40807226190/job/123228954032 (#2171)
https://github.com/CHIP-SPV/chipStar/actions/runs/40820096821/job/123270296920 (#2187)
05:24:07 xe 0000:03:00.0: [drm] GT0: Check job timeout: seqno=4294967175, lrc_seqno=4294967175, guc_id=2, not started
05:24:08 xe 0000:03:00.0: [drm] GT0: Engine reset: engine_class=bcs, logical_mask: 0x2, guc_id=0
05:24:08 xe 0000:03:00.0: [drm] GT0: Timedout job: seqno=130092, lrc_seqno=130092, guc_id=0, flags=0x13 in no process [-1]
05:24:08 xe 0000:03:00.0: [drm] GT0: Kernel-submitted job timed out
05:24:08 WARNING: ... at drivers/gpu/drm/xe/xe_guc_submit.c:1291 guc_exec_queue_timedout_job+0x87d/0xbf0 [xe]
05:24:08 xe 0000:03:00.0: [drm] GT0: reset done
06:59:29 xe 0000:03:00.0: [drm] GT0: Engine reset: engine_class=bcs, logical_mask: 0x2, guc_id=0
06:59:29 xe 0000:03:00.0: [drm] GT0: Timedout job: seqno=152156, lrc_seqno=152156, guc_id=0, flags=0x13 in no process [-1]
flags=0x13 marks a kernel-owned, permanent, high-priority exec queue (xe_exec_queue_types.h in Linux 6.17), which matches xe's internal migration queue on the copy engine. The timed-out job was therefore submitted by the driver itself, not by a test process. Which operation stalled (clear, copy or page table update) is not known. The GPU recovered both times and was not declared wedged.
The tests holding the GPU at the time were hip-tests memory cases run in parallel by check.py. They may be victims rather than triggers:
05:24 MemoryTest2 Unit_hipHostMalloc_Negative, Unit_hipMemcpyPeerAsync_Basic, Unit_hipMemPoolApi_Opportunistic
MemoryTest1 Unit_hipMemcpy2DFromArray_multiDevicePinnedMemPeerGpu
06:59 MemoryTest2 Unit_hipMalloc3DArray_Negative_ZeroWidth, _ZeroHeight, _InvalidFlags, _InvalidFormat
Environment: kernel 6.17.0-14-generic (Ubuntu), Mesa ANV, native Vulkan backend. The GuC firmware is hand-installed 70.65.0 rather than the distro one, which is an uncontrolled variable.
The suspected cause is in xe or the GuC rather than in chipStar, but that is not proven. No upstream drm/xe report for this signature has been identified yet. Related but not shown to be the same failure: #2150 (also logs "Kernel-submitted job timed out", but after a user submission and with a wedge) and #1734 (a user kernel on rcs reset by preemption timeout).
Next steps: repeat the memory tests in parallel with the stock GuC firmware, capture the devcoredump (root only, deleted after about an hour), and report upstream to drm/xe if it reproduces.
This issue was generated using AI
Activity
pvelesko commentedon Oct 29, 2026
Earlier incidents on the same lane, 2026-10-28, from
~/b570-watchdog.logon cupcake. They are not shown to share this cause. The 22:19 one is worse: the GuC refused the engine reset, xe declared the device wedged, and only a reboot recovered it.The kernel log from the start of the 22:19 wedge was lost: the reset messages repeated millions of times and rotated it out, so the engine and the submitting process are unknown. GraphsTest1 cases were running in both of these incidents and in neither of the copy engine ones. Next time it wedges, save
journalctl -k -bat once and copy/sys/class/drm/card*/device/devcoredump/dataas root before rebooting.This issue was generated using AI
pvelesko commentedon Oct 29, 2026
Another full wedge on 2026-10-29 at 13:22, during the Vulkan (cupcake) job for #2171. It has the same signature as the 10-28 22:19 one:
*ERROR* GT0: GuC engine reset request failed on 3:8 because 0x00000000, thentrying reset from guc_exec_queue_timedout_jobrepeating, with ttm kworkers stuck in D state for 3 hours. The processes the watchdog killed wereChannelDescriptorTest Unit_ChannelDescriptor_Positive_Basic_4D(char4, ushort4, short4, uint4). Those are host-only and launch no kernels, so they held the device but did not cause the wedge. The log from the start of the wedge was lost again because the repeating line pushed it out. The watchdog should savejournalctl -kas soon as it first sees the reset.This issue was generated using AI
pvelesko commentedon Oct 30, 2026
This boot's log survived intact, and it shows that the copy engine timeout is how the full wedge begins. Kernel 6.17.0-14-generic, GuC 70.65.0:
The processes holding the GPU were different again: ChannelDescriptorTest cases at 01:45; at 02:07, MemoryTest2
Unit_hipHostMalloc*cases from CI and an OCCAfpMathtest from a separate local chipStar build, sharing the B570. No GraphsTest1 case was involved. After the wedge, Mesa ANV reportsUnknown kernel mode driverfor the B570, and only a reboot recovers it.The PCODE "Illegal Command" and power limit errors at every boot point at a firmware mismatch, which fits the hand-installed GuC. Next: reboot, then run GraphsTest1 alone and in parallel to see whether it triggers this, and repeat with the stock GuC firmware.
This issue was generated using AI
pvelesko commentedon Oct 30, 2026
GraphsTest1 does not trigger this on its own. On a freshly booted cupcake (2026-10-30), using the CI container's Vulkan build on the B570, all 245 GraphsTest1 cases ran once serially and then three times with
ctest -j4. The kernel log, captured from the start, has no Engine reset, Timedout job or wedge line. 177 cases passed; the other 68 failed the same way every time, and all 68 are listed in the B570 known failures. The GraphsTest1 cases in the 10-28 incidents were therefore bystanders, like the ChannelDescriptor and MemoryTest cases. The common factor is the xe driver's own copy engine job timing out under mixed load (the full CI suite, plus other processes on the GPU). The next thing to try is swapping in the stock GuC firmware, since the PCODE errors at boot point there.This issue was generated using AI
pvelesko commentedon Oct 30, 2026
Root cause identified, pending an A/B confirmation: this is the upstream xe bug fixed by torvalds/linux@9f83c94 ("drm/xe: Order ring writes before ring tail updates", cherry picked from 136360290f31, in mainline v7.2,
Cc: stable,Fixes: dd08ebf6c352). It closes https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/8651, which has the same BMG signature (bcs, guc_id=0, "no process", "Kernel-submitted job timed out"), and https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/7810.From the commit message:
The ring and the context image are both write-combined, so the new ring tail can become visible to the GPU before the commands behind it. The copy engine then runs stale commands and hangs. A timeout on a kernel queue always escalates to a full GT reset, and when that reset fails (
failed to clear GRDOM_FULL) the device is wedged. Ubuntu's 6.17.0-14 is based on 6.17.9 and predates the fix:xe_lrc_write_ring()at tagUbuntu-hwe-6.17-6.17.0-14.14_24.04.1has noxe_device_wmb(). Upstream 6.17.y is EOL.It reproduces without chipStar. The test is a Vulkan program that only calls
vkAllocateMemoryandvkFreeMemoryon device-local memory, up to 64 MiB, from 8 processes of 16 threads each. Every new VRAM buffer makes the kernel submit a clear job to the migration queue. On cupcake it gave the identical signature after 7.5 minutes:Why CI hits it: check.py runs many test processes that allocate and free device memory at the same time, and the B570 watchdog SIGKILLs every process on the card at the first reset, which frees all their buffers at once. The chipStar tests named in the earlier comments were bystanders.
The
PCODE Mailbox failed: 1 Illegal Commanderrors at boot come from outdated BMG card firmware (https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/6269) and are unrelated.The fix for cupcake is a kernel that carries 9f83c94469ff: either Ubuntu's own xe module rebuilt with the one-line
xe_device_wmb(xe)at the end ofxe_lrc_write_ring(), or a v7.2+ kernel. Next step: rerun the same stress with the patched module to confirm.This issue was generated using AI
pvelesko commentedon Oct 30, 2026
Confirmed by an A/B on cupcake. I rebuilt Ubuntu's own xe module from tag
Ubuntu-hwe-6.17-6.17.0-14.14_24.04.1with only the upstream barrier added, i.e. torvalds/linux@9f83c94. A stock rebuild of the same tree matches the shipped module (same 5174 global functions):Both runs used the same stress: 8 processes x 16 threads allocating and freeing device-local VRAM.
The patched run also had a host load average near 300, which should make the submit-side race more likely, not less. The patched module is now installed on cupcake under
/lib/modules/6.17.0-14-generic/updates/. It has to be rebuilt for any kernel update until cupcake runs v7.2 or later, which carries the fix. Nothing changes in chipStar for this; the B570 resets in Vulkan CI should stop.This issue was generated using AI