Skip to content

Model download crashes the interpreter with an access violation, blocking every user after the rembg 2.0.75 bump #32

Open

Description

@sensitron

Summary

Downloading a model weight file kills the interpreter with a Windows access violation, in ui/download_transport.py. It reproduced 2 out of 2 attempts on a fresh cache namespace and left the app unusable: no model, so no Preview and no Render.

This is reachable for every user upgrading past #24. That PR moved the pin from rembg 2.0.72 to 2.0.75, and the model cache is namespaced by the pin (Cache/models/<version>/…), so the whole 13-model set has to be fetched again. The download path is what fails.

Impact

A user who upgrades and presses Preview or Render gets a hard process death — no dialog, no log line, no partial degradation. matteloop.log ends at the ONNX Runtime banner; the only trace is Logs/matteloop.fault. Retrying resumes the transfer (the .part grew 937 MB → 966 MB across two attempts), so it can eventually get through by chance, but each attempt is a crash.

This is the same failure shape as #22: a rembg bump lands, and the consequence surfaces nowhere the user can see it.

Reproduce

  1. Have a cache from a pre-fix: move the model namespace to rembg 2.0.75 and handle outdated weights #24 build (Cache/models/2.0.72/… populated).
  2. Update to a tree with rembg_version 2.0.75, so Cache/models/2.0.75/ is empty.
  3. Launch, load a video, press Preview (or Render) with birefnet-portrait.
  4. The download starts and the process dies part-way through.

Evidence

Two crashes, two different lines in the same path.

First:

Windows fatal exception: access violation

Thread 0x0000b170 (most recent call first):
  ui/download_transport.py, line 113 in _wait_for_event
  ui/download_transport.py, line 75 in read
  jobs/models/download.py, line 307 in _read_response
  jobs/models/download.py, line 200 in _download_locked
  jobs/models/download.py, line 127 in download
  jobs/models/session.py, line 140 in prepare
  ui/preview_controller/runtime.py, line 185 in prepare
  ui/preview_controller/worker.py, line 61 in run

Thread 0x00009da8 (most recent call first):
  app.py, line 79 in _run_gui
  app.py, line 308 in main

Second:

Windows fatal exception: access violation

Thread 0x0000bc08 (most recent call first):
  jobs/models/download.py, line 221 in _download_locked
  jobs/models/download.py, line 127 in download
  jobs/models/session.py, line 140 in prepare
  ui/preview_controller/runtime.py, line 185 in prepare
  ui/preview_controller/worker.py, line 61 in run

Thread 0x0000c7ac (most recent call first):
  ui/preview_controller/dialog.py, line 266 in set_progress
  ui/preview_controller/controller.py, line 254 in _notification
  app.py, line 79 in _run_gui
  app.py, line 308 in main

Hypothesis — not verified, offered only as a starting point

_QtNetworkDownloadResponse.__init__ (ui/download_transport.py:38-48) constructs a QNetworkAccessManager and its QNetworkReply, and _wait_for_event drives them with a nested QEventLoop. That whole object graph is built and driven on the preview worker thread (preview_controller/worker.py:61 → runtime.py:185 → session.py:140), while the main thread concurrently updates the progress dialog (controller.py:254 → dialog.py:266), as the second dump shows.

I have deliberately not gone further than that. I spent a long stretch on PR #27 today chasing a superficially similar Qt threading crash, and three plausible-sounding hypotheses each turned out wrong or actively harmful until a core dump settled it. The stacks above are what is actually known; the mechanism deserves the same treatment rather than a guess.

If it helps: on Linux, gdb on a core dump (ulimit -c unlimited, core_pattern already core) gives the C++ frames that faulthandler cannot, and that was what finally identified the equivalent bug in #27.

Workaround, if someone is blocked

Fetch the artifact outside the app and verify it against the manifest before use — no directory is migrated or renamed, so the namespace stays the integrity boundary:

cd "<cache>/models/2.0.75/birefnet-portrait"
curl -L -C - -o birefnet-portrait.onnx.part \
  "https://github.com/danielgatis/rembg/releases/download/v0.0.0/BiRefNet-portrait-epoch_150.onnx"
sha256sum birefnet-portrait.onnx.part   # must equal the manifest's sha256
mv birefnet-portrait.onnx.part birefnet-portrait.onnx

For birefnet-portrait the expected values from resources/model-manifest.json are 972666916 bytes and sha256 1ba1c8ff5a7bbfadc8d8d13fb11d7be793f91f23d9d466549e37a854f6668f99. Confirmed working here: after this the app starts and prepares the model normally. Copying the identically-sized 2.0.72 file into the 2.0.75 directory would have been faster, but CLAUDE.md forbids migrating weight directories, and re-fetching sidesteps the question entirely.

Environment

Activity

  1. smb commented on Sep 4, 2026

    @smb
    Collaborator

    This is #4, and the window it reappeared in is ours

    The first stack here is line-for-line the stack in #4 (_wait_for_event → read → _read_response → _download_locked → download). #4 was closed yesterday at 17:10 UTC as not reproducible, after you exercised the transport in isolation at the same shape and it stayed clean. It now reproduces 2 of 2. So the interesting question is not what the transport does, which you already tested, but what changed in that window.

    Two of my merges sit in it, and they are not equally suspect.

    #24 explains why a download runs at all. It moved the pin to rembg 2.0.75, and the cache is namespaced by the pin, so the whole set is refetched. That part of your analysis is right, and it is why this now blocks everyone rather than only a first-run user.

    #20 changed what is in the process while that download runs, and that is the part worth testing. I checked the history rather than assuming: the GUI process imported ONNX Runtime at startup before #20 as well, through provider_options_from_runtime (core/execution_providers.py), so "does the GUI import onnxruntime" is not the delta. What changed is which build. On Windows the dependency is now onnxruntime-directml, so DirectML.dll and the Direct3D 12 stack it pulls in are loaded into exactly the process that drives the Qt network download. Before #20 that process held the plain CPU wheel.

    Between "isolated transport does not fault" (#4) and "faults 2 of 2" (#32), that is the concrete difference I can point at.

    The experiment that settles it, one line

    Re-run the isolated transport harness from #4 — the one that downloaded a real weight to completion without faulting — with import onnxruntime executed before the harness starts. Same machine, same URL, same shape; the only delta is the DirectML runtime resident in the process.

    • If it faults: the download path is a victim, not the culprit, and the fix is where the runtime gets loaded (the GUI process does not need it; --providers and the provider list could come from the child, which already loads a runtime anyway).
    • If it does not fault: feat: ship DirectML on Windows and report the ONNX Runtime providers #20 is cleared and the difference is elsewhere, which is worth knowing before anyone touches download_transport.py.

    What I could contribute from here

    macOS, so no DirectML and no verdict on the crash itself, but a second platform data point on the transport under the same shape as the app: worker QObject moved onto a QThread, the download inside a slot, progress delivered by a queued signal, the main thread repainting a widget on a 16 ms timer. Four rounds of 300 MB, twice: once with nothing else loaded, once with onnxruntime imported first. All eight rounds completed with the exact byte count, no fault, no timeout. That is 2.4 GB through _QtNetworkDownloadResponse including several thousand nested QEventLoop cycles. It matches your #4 result: the transport in isolation does not fault, on either platform I can reach.

    One inconsistency worth a second look

    You observed the .part growing 937 MB → 966 MB across two attempts and read it as a resume. _open_part unlinks the file before creating it (jobs/models/download.py:349) and cache_fs opens it O_WRONLY|O_CREAT|O_EXCL (cache_fs.py:280), so a resume should be impossible and the second attempt should have restarted from zero. Either the unlink did not take effect on that machine, or the two numbers came from different files. If the unlink is genuinely not happening on Windows, that is a separate defect with its own consequences: a partial body would be appended to, and the SHA-256 check would then fail with a checksum error rather than a size error.

    I have not touched anything. Say the word if you want me to take the DirectML-in-process question — the change would be on my side of the fence anyway, since #20 introduced it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions