remoteproc: adsp/cdsp stay permanently offline when qcom_q6v5_pas loads from the initramfs

[lool]

A number of teams are chasing issues where remoteproc DSPs stop working after installing OOT kernel drivers. This class of issues is triggered because the initramfs is being regenerated on a booted device and starts including the qcom_q6v5_pas driver, while the default initramfs is generated in a clean build environment (e.g. qemu) and does not include that driver.

If qcom_q6v5_pas is loaded from the initramfs, i.e. before the real root
filesystem is mounted, the affected remote processors fail their firmware load
with -ENOENT and remain offline for the entire lifetime of the boot. The
initramfs contains no /lib/firmware, so the firmware genuinely is not
reachable at that point. There is no retry once the rootfs appears.

This is not a firmware packaging problem: the firmware is present and correct on
the root filesystem. It is purely an ordering problem plus a missing retry.

Other drivers on the same SoC hitting the exact same condition do recover (for instance geni),
which makes the remoteproc behaviour an inconsistency rather than an inherent
limitation. See "The same SoC already demonstrates the retry model working"
below.

Environment

• Board: Qualcomm RB3Gen2 (QCS6490)
• Kernel: qcom-next 7.2.0-rc3-g8d5dbc1b17ad
• Distro: Debian 13 (trixie), initramfs-tools 0.148.4, dracut-install 106-6
• CONFIG_REMOTEPROC=y (core built in); PAS / ADSP / MSS / SYSMON / PIL_INFO
  built as modules. CONFIG_QCOM_Q6V5_WCSS is not set.
• deferred_probe_timeout=30 on the kernel cmdline;
  CONFIG_DRIVER_DEFERRED_PROBE_TIMEOUT=60

Observed behaviour

Good boot (qcom_q6v5_pas loaded after switch_root):

remoteproc0 modem offline
remoteproc1 adsp running
remoteproc2 wpss running
remoteproc3 cdsp running

Bad boot (qcom_q6v5_pas present in the initramfs):

remoteproc0 modem offline
remoteproc1 adsp offline
remoteproc2 wpss running
remoteproc3 cdsp offline

adsp and cdsp regress. wpss is unaffected. modem is offline in both
cases and is a separate, pre-existing issue; see "Note on modem" below.

The cores can be recovered by hand after boot, which confirms the firmware and
the hardware are fine and that only the timing was wrong:

echo start > /sys/class/remoteproc/remoteproc1/state

Evidence

Relevant dmesg extract from a bad boot, with the rootfs mount left in place
for timing reference:

[    5.226122] Run /init as init process
[    5.888547] remoteproc remoteproc1: adsp is available
[    5.913181] remoteproc remoteproc1: Direct firmware load for qcom/qcs6490/adsp.mbn failed with error -2
[    5.929130] remoteproc remoteproc1: powering up adsp
[    5.940219] remoteproc remoteproc1: request_firmware failed: -2
[    5.956409] remoteproc remoteproc3: cdsp is available
[    5.966939] remoteproc remoteproc3: Direct firmware load for qcom/qcs6490/cdsp.mbn failed with error -2
[    6.015479] remoteproc remoteproc3: request_firmware failed: -2
[    7.602464] EXT4-fs (sda2): mounted filesystem 8fd73327-... ro with ordered data mode.
[   19.669528] ath11k 17a10040.wifi: wcn6750 hw1.0
[   19.671172] remoteproc remoteproc2: powering up wpss
[   19.684077] remoteproc remoteproc2: Booting fw image qcom/qcs6490/wpss.mbn, size 7734064
[   19.803446] remoteproc remoteproc2: remote processor wpss is now up

• adsp and cdsp attempt their firmware load at ~5.9s — 1.7s before the
  root filesystem is mounted at 7.60s. Nothing retries afterwards.
• wpss escapes only by accident of ordering: ath11k boots it at 19.6s, long
  after switch_root, when the firmware is reachable. It works first try.

Note there are two failure messages per core from a single auto-boot attempt.
That is explained under "Why remoteproc gives up permanently".

What triggers it

The trigger is anything that causes update-initramfs to run on the board
rather than in the image build environment.

In the case that surfaced this, installing kgsl-dkms was the trigger. It is
worth being explicit that this is not a module dependency:

• modinfo msm_kgsl shows depends: llcc-qcom,mdt_loader,ubwc_config and no
  remoteproc edge.
• modules.dep has no path from msm_kgsl to qcom_q6v5_pas.
• dracut-install -o -m msm_kgsl in isolation pulls only gpucc-sc7280,
  llcc-qcom, mdt_loader, socinfo, ubwc_config.

The actual mechanism is initramfs-tools' live-sysfs walk. With MODULES=most,
auto_add_modules reaches add_loaded_modules (hook-functions:617-620):

# clk, USB-PHY, pinctrl and reset devices are outside the device
# model (!) so match loaded modules by name
add_loaded_modules 'clk[-_]*'
add_loaded_modules 'phy[-_]*'
add_loaded_modules 'pinctrl[-_]*'
add_loaded_modules 'reset[-_]*'

add_loaded_modules iterates /sys/module/$pattern/drivers/*/* and calls
sys_walk_mod_add on each bound device. That walks supplier:* devlinks and
parent devices, and for every hop adds both the modalias and the currently
bound driver/module. On a generic arm64 build host none of that sysfs exists
and it contributes nothing; on the board it enumerates the running platform, and
qcom_q6v5_pas is reachable from the clk/pinctrl/regulator devices the walk
visits.

Consequences worth noting for anyone reproducing this:

• Removing the kgsl-dkms initramfs hook does not help. The walk adds
  qcom_q6v5_pas whether or not msm_kgsl is involved.
• Confirmed empirically: rebuilding the initramfs on the board with the kgsl
  hook moved aside, with kgsl-dkms.conf moved aside, with msm_kgsl.ko
  removed from the module tree and depmod -a re-run, and with MODULES=dep
  and MODULES=list, produced the five remoteproc modules every single time.
• With MODULES=list and the hook disabled, msm_kgsl.ko itself still
  appeared — nothing static referenced it, the live walk found it.

So kgsl-dkms is only the event that caused the first on-board regeneration.
Any package with an initramfs hook, or any manual update-initramfs -u on a
running board, will do the same. This will bite anyone doing on-target
development.

Also relevant: the resulting initramfs contains 366 .ko files and zero
lib/firmware entries.

Why remoteproc gives up permanently

adsp_probe() → devm_rproc_add() → rproc_add() → rproc_trigger_auto_boot():

ret = request_firmware_nowait(THIS_MODULE, FW_ACTION_UEVENT, rproc->firmware,
                              &rproc->dev, GFP_KERNEL, rproc,
                              rproc_auto_boot_callback);

The async load fails (Direct firmware load ... failed with error -2), but
rproc_auto_boot_callback() runs unconditionally with fw == NULL and calls
rproc_boot() regardless, which performs a second, synchronous
request_firmware() — that is the request_firmware failed: -2 line. Hence two
messages per core from one auto-boot attempt.

rproc_boot() then bails, rproc->state stays OFFLINE, and the remoteproc
core has no retry path. Nothing revisits it.

The important detail: adsp_probe() itself succeeded. The failure is
post-probe and asynchronous, so the deferred-probe machinery is never consulted
and the usual "wait for the resource to appear" behaviour never engages.

The same SoC already demonstrates the retry model working

geni_spi / geni_i2c hit the identical missing-firmware condition on the same
boot and recover:

[    7.031281] geni_spi 98c000.spi: spi master firmware load failed ret: -517
[    7.031818] geni_i2c 980000.i2c: Direct firmware load for qcom/qcs6490/qupv3fw.elf failed with error -2
[    7.186147] geni_spi 98c000.spi: spi master firmware load failed ret: -517
[    7.235307] geni_spi 98c000.spi: spi master firmware load failed ret: -517
[    7.439343] geni_spi 98c000.spi: spi master firmware load failed ret: -517
[    7.453343] geni_i2c 980000.i2c: Direct firmware load for qcom/qcs6490/qupv3fw.elf failed with error -2
[    7.536333] geni_i2c 984000.i2c: Direct firmware load for qcom/qcs6490/qupv3fw.elf failed with error -2
[    7.602464] EXT4-fs (sda2): mounted filesystem ... ro with ordered data mode.
        <- no further geni failures for the rest of the boot

Six retries between 7.03s and 7.54s, rootfs at 7.60s, then silence — and
980000.i2c, 984000.i2c and 98c000.spi are all bound afterwards.
request_firmware() is silent on success, so the retry across the pivot
succeeded.

These drivers return -EPROBE_DEFER (-517) and are retried; qcom_q6v5_pas
returns -ENOENT (-2) and is not. The asymmetry is entirely in how the two
drivers translate a missing firmware file. Retry-after-pivot is therefore not a
speculative design for this SoC — it is already load-bearing on it.

Proposed fixes

A. Return -EPROBE_DEFER from adsp_probe() when the firmware is unreachable

Smallest change, and it reuses machinery already proven on this board.

	/* Don't let rproc_add() auto-boot against a rootfs that isn't up yet */
	ret = request_firmware_direct(&fw, fw_name, &pdev->dev);
	if (ret < 0)
		return dev_err_probe(&pdev->dev, -EPROBE_DEFER,
				     "firmware %s not available yet\n", fw_name);
	release_firmware(fw);

request_firmware_direct() skips the userspace fallback and does not warn.

Objections to expect:

1. It reads the whole multi-MB blob only to discard it, and rproc_boot() then
   reads it again.
2. -EPROBE_DEFER is conventionally for absent devices, not absent files.

The counter to (2) is the in-tree geni behaviour above, on the same SoC, in the
same boot.

Caveat: with deferred_probe_timeout=30, 30s is ample against a 7.6s mount, but
a slow rootfs (NFS, LUKS, a full fsck) would exceed it and then fail permanently
with -ETIMEDOUT. That is arguably worse than today, because the failure becomes
timing-dependent rather than deterministic.

B. Retry inside the remoteproc core (preferred)

This is the architecturally correct place, because the defect is that
rproc_auto_boot_callback() discards a failed async load and then calls
rproc_boot() into a second guaranteed failure.

Have the callback distinguish fw == NULL and schedule a bounded delayed retry
instead. This benefits every rproc driver rather than only PAS, and avoids the
double read and the deferred-probe-timeout cliff of option A. Larger discussion
on linux-remoteproc, but it matches the actual bug.

C. Add MODULE_FIRMWARE() declarations to qcom_q6v5_pas (do this regardless)

modinfo -F firmware qcom_q6v5_pas is expected to be empty: PAS firmware names
come from driver match data and DT firmware-name, so nothing is declared.

That means dracut-install -m — and every other tool that bundles firmware via
modinfo -F firmware — cannot know to pull adsp.mbn / cdsp.mbn in
alongside the module. This is exactly why the initramfs here ends up with 366
.ko files and zero lib/firmware entries.

Declaring the static per-SoC names lets existing distro tooling do the right
thing automatically. It is an easier sell upstream than A or B, it fixes the
class of bug rather than this instance, and it is complementary to whichever of
A/B lands.

D. Do not use the CONFIG_FW_LOADER_USER_HELPER fallback

It looks tailor-made — FW_ACTION_UEVENT is already passed, so the sysfs
fallback loader would engage and block up to firmware_loading_timeout
(default 60s) — but it is deprecated, requires a helper in the initramfs, and
would be actively harmful here: modem.mbn genuinely does not exist for
qcs6490, so every boot would stall a full 60s waiting for a file that is never
coming.

E. Userspace stopgap (independent of the above)

state is writable, so a systemd unit ordered after local-fs.target can start
the cores post-boot:

echo start > /sys/class/remoteproc/remoteprocN/state

A legitimate hold-the-line measure for distro images while a kernel fix is in
flight.

Recommendation

• Land C now — low risk, uncontroversial, and it addresses the actual reason
  the firmware was never bundled.
• Propose B on linux-remoteproc as the real fix, with A as the fallback
  if the core change stalls.
• Keep E in distro images until a kernel fix ships.

The dmesg above is the strongest supporting argument for either A or B, since it
shows both behaviours — defer-and-recover versus fail-and-give-up — side by side
in a single boot on one SoC.

Note on modem

modem is not part of this regression and should not be conflated with it:

• It is already offline before the triggering change.
• There is no modem.mbn for qcs6490 in firmware-qcom-dsp. It exists only
  under apq8096, qrb4210, qcm2290 and sdm845.

Only adsp and cdsp regress.

This has a direct design implication for the fixes above: a genuinely absent
firmware blob must be treated as a permanent, quiet failure rather than
something to defer or retry indefinitely. Otherwise modem turns into a
boot-delay bug — which is precisely the trap in option D.

Reproducer

1. Boot a qcom-next Debian trixie image on RB3Gen2. Confirm adsp, cdsp and
   wpss are running.
2. On the board, trigger an initramfs regeneration — e.g. update-initramfs -u,
   or install any package with an initramfs hook (kgsl-dkms was the original
   trigger).
3. Confirm qcom_q6v5_pas.ko is now present:
   lsinitramfs /boot/initrd.img-$(uname -r) | grep remoteproc
4. Confirm the image ships no firmware:
   lsinitramfs /boot/initrd.img-$(uname -r) | grep -c lib/firmware → 0
5. Reboot. adsp and cdsp are now offline and stay that way.
6. dmesg | grep remoteproc shows the -2 failures occurring before the
   EXT4-fs ... mounted filesystem line for the root device.

[lool]

(Sorry for the slightly verbose report; this information was collected by AI and I wanted to start a public issue to start merging the 5 internal threads on this)

[lool]

With this patch:
0001-remoteproc-qcom_q6v5_pas-defer-probe-until-firmware-.patch

I'm back to all DSP cores showing up after reboot.

I'm not a kernel engineer and I'd like to hand this over to someone else to send a proper fix upstream.

[lool]

This LAVA job demonstrates:

• flashing a qcom-next Debian image
• all DSPs working
• run update-initramfs -u
• reboot
• some DSPs now failing

(Search for "remoteproc1 adsp")

[shoudil]

Loïc Minier (@lool) , thanks for listing down the issue details and proposals, discussed similar in an old mail thread, added you.
Mail: "[Hamoa Ubuntu] Discuss on adding firmware dependency with MODULE_FIRMWARE in remoteproc"

[lool]

There are at least 3 types of issues get discussed across threads (thanks Dmitry for splitting 1a/1b):

1. getting remoteproc devices to work when the driver was loaded when firmware was not yet available
   1a. reprobing IPA, Venus or other drivers which require firmware at the probe time
   1b. handling the drivers which probed, but then failed somewhere without the firmware
2. including boot critical drivers and their firmware, e.g. the USB TCPM example; we need a clear consensus on how to do this properly, kernel + initramfs generators

I believe our priority issue is 1b which affects DSPs and I'd like to focus this ticket on it, but there's the need for a wider solution for 1) (Bjorn can guide) and some strategy is also needed for 2).

[lool]

I've been researching workarounds while a proper kernel solution is devised. There are a lot of dimensions:

• dracut vs initramfs-tools as initramfs generator
• whether the initramfs is meant to be generic (typically generated from a clean virtual build environment) or device specific
• whether we're consider IoT platforms (can usually defer loading to post-initramfs) or laptop platforms (makena/hamoa must have ADSP reset with new firmware before udev to avoid an USB reset later)

For IoT boards, I've proposed an udev rule in fastrpc to start the expected remoteprocs after rootfs is available if they aren't started already.

Alternatively, we could include qcom_q6v5_pas in the initramfs unconditionally, but only include firmware for laptop platforms; a module blacklist would prevent autoloading, and loading would be forced only on laptop platforms.

[lool]

NB: support for laptops was contributed by Hans to dracut upstream (here, and here) and is being contributed by Agathe to initramfs-tools upstream

[lool]

For initramfs-tools, I got https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/201 working fine on T14s and RB3Gen2; needs upstream review and would need similar changes in dracut.