Repository navigation
tests/ui occasionally wedges instead of failing, with no output and no crash #39
Description
Context
While diagnosing the intermittent access violation in tests/ui on 2026-09-04, a second, different
failure mode showed up three times: a pytest run wedged indefinitely — no output, no crash, no
progress — rather than terminating. Runs were bounded at 120 s during that work, so each was
recorded as a hang and abandoned rather than waited out.
It happened once on an otherwise unmodified tree, so it is not specific to any branch.
This is filed separately, and deliberately, because the access violation next to it spent a long
time mislabelled as "a known PyAV crash in roughly 3 of 25 runs" and was therefore never
investigated. It turned out to have nothing to do with PyAV: RenderController.shutdown() destroyed
every widget it owns except the job dialog, which outlives its job to show the completion summary
and, having no dialog parent, was left to Python's garbage collector — so a QWidget could be
destroyed from inside a running event loop. That one is fixed (ebf51b2 on
feat/issue-25-transform-stage, measured 1 crash / 9 runs before, 0 / 6 after, 2 / 4 with the
teardown removed again as a control).
The hang has no such explanation, and an unexplained failure mode without an issue tends to become
the next piece of folklore.
What is known
- Three occurrences during roughly two hours of repeated runs of
tests/ui, at least one of them on
a tree with no source changes. - It did not recur in the final eight consecutive runs after
ebf51b2. That is not evidence it is
gone — the crash it sits next to also went from 11% to 100% and back purely on allocation
timing, and eight runs cannot distinguish "fixed" from "rare again". - Plausibly the other face of the same shutdown-ordering area, since that is where the widget and
thread teardown of a job lives: a hang is what aQThread.wait()or an event loop that never
returns looks like, where a use-after-free looks like the access violation. This is a hypothesis,
not a finding — nobody has captured a stack from a wedged run.
What would settle it
The cheap first step is to capture the state of a wedged process rather than killing it: a thread
dump (faulthandler.dump_traceback_later, or attaching to the stuck process) would say immediately
whether it is parked in a wait(), in app.exec, or somewhere unrelated. logs.py already enables
faulthandler for the application; the test runs do not.
Running tests/ui in a loop under a timeout that dumps instead of killing would likely collect that
within an hour, given the observed rate.
Why it matters
An intermittent hang is worse than an intermittent crash in CI: a crashed run fails fast and is
retried, a wedged one burns the job's whole time budget before anything is reported, and locally it
costs a person the time before they notice.
Activity
sensitron commentedon Sep 5, 2026
Measured, not fixed, per this issue's own framing. Added an opt-in
faulthandler.dump_traceback_later()switch totests/ui/conftest.py(
MATTELOOP_TEST_HANG_DUMP_SECONDS, off by default, normal suiteunaffected) and ran
tests/uiin a loop, so a hang gets a stack dumpinstead of being killed blind the way the three prior sightings were.
122 runs, Linux (devbox),
QT_QPA_PLATFORM=offscreen,uv run pytest -q tests/ui, 2026-09-04T22:30:33Z – 2026-09-05T00:43:05Z.Result: reproduced, 7/7 identical. Every one of 7 hangs (runs 4, 19, 43,
105, 110, 114, 122 in the loop) produced the exact same stack, parked in the
exact same test:
tests/ui/test_preview_controller.py::test_cancel_keeps_modal_dialog_open_until_the_safe_checkpoint,line 289:
That call's own 5s timeout never fired: the main thread was still parked
inside
qt_compat.py:160 exec()(the Qt event loop pytest-qt pumps whilewaiting) 45s later (faulthandler dump) and 900s later (still there when the
loop's outer watchdog killed the process). While parked in that pump, a
queued notification was being delivered synchronously:
PreviewController._notification->_terminal_notification->store.dispatch->MainWindow.render_state->_render->ResultPlayerCanvas.set_presented_frame->PreviewCanvas.set_presented_frame->PreviewCanvas.set_frame, where thevisible Python frame sits at
self.clear()/self.setText()(adjacentlines 161/162) every single time. faulthandler only shows Python frames, so
the actual block is one level deeper, inside that Qt/PySide call. A second
thread is present with
<no Python frame>-- i.e. not executing Python atthe moment of the dump.
In this issue's own vocabulary: this parks in the "event loop that never
returns" bucket (
app.exec()-equivalent, via pytest-qt's pump) -- not aQThread.wait()visible in Python, and not somewhere unrelated to thehypothesis.
Bonus finding -- the known crash, same shape, different test: 2 of the
122 runs (1, 111) hit the already-known
Fatal Python error: Segmentation faultinstead (caught by pytest's own always-onfaulthandler.enable(),independent of the switch above), both in
test_render_controller.py::test_cancelled_render_closes_without_showing_completion_summary-- also parked inside a
qtbot.waitUntilpumpingqt_compat.py exec()atthe moment of the fault, also a "cancel this operation" test. The crash's
visible frame is shallower than the hang's (right at
exec(), no deeperdispatch frames), so this supports the issue's "same shutdown-ordering bug,
two faces" hypothesis structurally -- both are a cancel test caught
mid-pump, paired with a second thread with no Python frame -- without
proving it's the identical fault.
Platform caveat, stated plainly: this was measured on Linux; the
original sighting was on Windows. A Linux reproduction does not prove the
Windows hang is the same bug -- different Qt platform plugin, different
thread/signal timing. What it does show: the hang is real, present, and
reproducible on at least one platform, and it clusters tightly on one
specific "cancel while an event-loop pump is in flight" test rather than
being a diffuse, evenly-distributed flake. The observed rate here (7/122,
~6%) should not be read as a stable base rate either way -- this issue's
own AV crash swung from 11% to 100% and back between otherwise-identical
runs purely from allocation timing, so treat "how often" as noise and "where"
as the finding.
Second bonus finding -- a clean, one-off shutdown-ordering exception:
run 54 neither hung nor crashed -- it failed cleanly, with a full Python
traceback, in a third test
(
test_preview_controller.py::test_preview_request_uses_the_selected_oriented_crop):SourceController.shutdown()iteratesself._threadsand calls.quit()on each; here one thread's underlying C++
QThreadhad already beendeleted by the time this run got there. Happened exactly once in 122 runs,
not investigated further (out of scope), but it's the same "who tears down
what, in what order" shape as the hang and the crash above, this time on a
stale reference surviving an async deletion rather than a stuck event-loop
pump or a native fault.
Side note found along the way: the CI
windowsjob never runstests/uiat all (excluded becausetest_render_controllercrashes itwith the access violation this issue's diagnosis grew out of) -- so a green
Windows CI has never exercised this code path, in either direction.
The curated stacks and the per-run protocol follow in the next comment. The raw
5.5 MB faulthandler dump and the per-run event logs lived on a throwaway
measurement branch and are not preserved; the switch that produces them is in
the pull request referencing this issue, so they can be regenerated with
MATTELOOP_TEST_HANG_DUMP_SECONDS=45.sensitron commentedon Sep 5, 2026
Issue #39 — curated hang evidence
The measurement produced a 5.3 MB faulthandler dump and a per-run protocol on a
throwaway measurement branch that has since been deleted. This is the distilled
version: the 9 anomalous runs out of 122, and the actual stack content. The raw
material can be regenerated with
MATTELOOP_TEST_HANG_DUMP_SECONDS=45using theswitch added in the pull request that references this issue.
The hang: 7/7 identical, always the same test
Every one of the 7 runs that hit the outer 900s timeout produced a
faulthandler dump, and all seven land on the exact same test and the exact
same call chain — only the thread IDs and one call-site line (161 vs 162,
two adjacent statements in the same method) differ:
Test:
tests/ui/test_preview_controller.py::test_cancel_keeps_modal_dialog_open_until_the_safe_checkpoint(both
[True]and[False]parametrizations occur across the 7).Representative stack (run 4, first occurrence):
Reading this
test_cancel_keeps_modal_dialog_open_until_the_safe_checkpointline 289 is:That call carries its own 5000ms timeout. The process was still parked here
45 seconds later (dump) and 900 seconds later (outer kill), which means
waitUntil's own timeout enforcement never got a chance to run: the callchain shows the main thread inside
qt_compat.py:160 exec()— the Qt eventloop pytest-qt pumps while waiting — synchronously delivering a queued
notification (
PreviewController._notification->_terminal_notification)that dispatches through the reducer/store into
MainWindow.render_state,down to
PreviewCanvas.set_frame, where the Python-visible frame sits atself.clear()/self.setText()(adjacent lines 161/162, both nativePySide6/Qt calls) every single time.
Faulthandler only shows Python frames, so the actual block is one level
below what's visible here, inside the Qt/PySide call. The second thread
shows
<no Python frame>-- consistent with a native (Qt-internal) threadholding whatever the main thread's Qt call is waiting on, but that's an
inference from the shape of the dump, not something the dump proves by
itself.
In the issue's own vocabulary: this is squarely the "event loop that never
returns" bucket (
app.exec()-equivalent via pytest-qt's pump), not aQThread.wait()call visible in Python, and not somewhere unrelated to thehypothesis.
Bonus finding: the known crash, same shape, different test
Two runs (1 and 111) crashed instead of hanging --
Fatal Python error: Segmentation fault, pytest's own built-in faulthandler catching it (this isindependent of the MATTELOOP_TEST_HANG_DUMP_SECONDS switch; pytest enables
faulthandler.enable()unconditionally). Both land intests/ui/test_render_controller.py::test_cancelled_render_closes_without_showing_completion_summary,also while pumping the event loop inside a
qtbot.waitUntil:Note the crash's Python-visible frame is shallower than the hang's --
it stops right at
qt_compat.py:160 exec(), with none of the deeperdispatch/render frames the hang shows. So this is not proof of an identical
root cause, only a structural family resemblance: both anomalies are a
"cancel this operation" test, both caught mid-pump inside the same
pytest-qt event-loop-exec call, both paired with a second thread carrying
no Python frame. That resemblance is exactly what Issue #39 predicted --
crash and hang as two faces of the same shutdown-ordering problem -- but it
is evidence for the hypothesis, not proof of it.
(Runs 1 and 111 in the loop; the raw captures were on the deleted measurement
branch.)
Second bonus finding: a clean, one-off shutdown-ordering exception
Run 54 (2026-09-04T23:27:43Z) neither hung nor crashed -- it failed cleanly,
with a full Python traceback, in
tests/ui/test_preview_controller.py::test_preview_request_uses_the_selected_oriented_crop:This happened exactly once across all 122 runs. It's a third, independent
data point for the same theme:
SourceController.shutdown()(
src/matteloop/ui/controller.py:320-340) iteratesself._threadsandcalls
.quit()on each, but by the time this particular run got there, onethread's underlying C++
QThreadhad already been deleted -- consistentwith the thread having already been torn down and removed elsewhere (e.g.
via a finished-signal callback) while
self._threadsstill held areference to it. Not investigated further -- out of scope for a measurement
task -- but it's the same "who tears down what, and in what order" shape as
the hang and the crash, on a third mechanism (a stale reference used after
async deletion, rather than a stuck event-loop pump or a native fault).
smb commentedon Sep 5, 2026
This reproduced in CI today, not just locally, and the numbers are unusually
clean — so recording them here as evidence. I am not picking the issue up; it is
yours.
Occurrence: run
33952768746, jobtest (ui), onfix/english-provider-labels.Wedged in step 6,
uv run pytest -q tests/ui, for 1 h 47 min before Icancelled it. Every other job in the same run finished normally (lint 18 s,
test (core) 41 s, test (jobs) 81 s, test (packaging) 36 s, swscale 148 s,
windows 228 s).
Baseline: across the twelve most recent
ci.ymlruns,test (ui)takesbetween 41 s and 63 s. There is no slow-but-finishing case in that sample — it
either takes under a minute or it never returns.
It is not the branch and it is not the code. Two data points, both from
today:
run, after one more commit.
docs/issue-43-screenshotscontains both of that branch's commits verbatim,plus more, and its
test (ui)passed in 50 s an hour and a half later.So the same tree passes and wedges depending on nothing observable, which
matches your note that it happened once on an otherwise unmodified tree.
No log is retrievable from a wedged job.
GET /actions/jobs/<id>/logsreturns
BlobNotFoundwhile the job is in progress, so there is nothing to readunless the job is allowed to run into the 6-hour default timeout — at which
point GitHub kills it and the log still shows only the last line before the
wedge.
That is worth fixing regardless of the root cause. A job-level
timeout-minuteswell under six hours would stop a wedge from occupying arunner for a working day and blocking a required check. And
pytest-timeoutwith
--timeout-method=threadwould turn the wedge into a failure carrying atraceback of every thread, which is exactly the evidence this issue currently
cannot obtain.
Separately, and possibly unrelated: a local
pytest -qrun on macOS failed oncetoday, once only, and four immediate re-runs were clean. That was a failure,
not a wedge, and I could not capture which test — noting it only so it is not
silently folded into this issue if it turns out to be a third mode.
smb commentedon Sep 5, 2026
The same job crashed instead of wedging, and left a stack behind. First
concrete evidence in this issue, so recording it here.
Run: https://github.com/smb/MatteLoop/actions/runs/33990641196/job/101372128654
(the
test (ui)job of a pull request whose change touches neither the UI northe render path). Exit code 139,
Fatal Python error: Segmentation fault,after roughly 41 seconds.
The crashing thread is not the main thread and carries no Python frame, so the
fault is inside Qt or a native extension while the main thread sits in the
event loop of
qtbot.waitUntil. Line 398 is the wait that follows the click onthe cancel button:
This narrows the issue in two ways. It is the cancel path, not the test
suite at large — which lines up with #64, where
close()blocks on_operation_lockwhile a worker is still inside an operation. And the failuremode is not only a hang: the same site can take the process down. The issue
title says "with no output and no crash"; that is now known to be one of two
outcomes.
Reproduction remains stubborn. The test passed eight consecutive local runs
(
tests/ui/test_render_controller.py, 15 passed each time), the fulltests/uisuite passes locally (339 passed), and an unchanged rerun of the same commit
went green. So it stays a race, but it is a race with an address now.
The job also ran for its full length before dying rather than being killed, and
no CI job in this repository declares
timeout-minutes— a wedged run inheritsGitHub's six-hour default. A separate change adds per-job limits so the hang
form of this defect fails in minutes instead.
smb commentedon Sep 5, 2026
Reproduced 27 minutes later, same test, same line, same signal — on a branch
whose entire diff is four
timeout-minutes:lines in.github/workflows/ci.yml.https://github.com/smb/MatteLoop/actions/runs/33991848683/job/101375363196
That branch changes no Python at all, so nothing about the product code can be
the cause. Together with the earlier crash on an unrelated source-probe branch,
the failure is now twice in roughly four
test (ui)runs on this runner imagewithin one hour, against a test that had been passing for weeks. Either the
race has always been this likely and simply landed on a quiet stretch of green
runs, or something in the environment changed recently.
Both occurrences are the same waiting site: after the click on cancel, while
qtbot.waitUntilruns the event loop and a worker thread is still shuttingdown.
smb commentedon Sep 5, 2026
Reopened. This was closed by the merge of #71, which only added job timeouts and explicitly does not address the defect — a commit body there contained the phrase "does not fix #39", and GitHub's keyword parser matched "fix #39" without reading the negation. No decision was made to close it.