[PW_SID:1151269] [BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter - #2521
[PW_SID:1151269] [BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter#2521linux-riscv-bot wants to merge 8 commits into
Conversation
…_RWX When CONFIG_STRICT_MODULE_RWX is not set, execmem cannot create temporary writable mappings for read-only executable pages. In this case, the execmem ranges must already have writable permissions. Currently EXECMEM_KPROBES unconditionally uses PAGE_KERNEL_READ_EXEC, which causes kprobe instruction slot writes to trigger page faults on systems where CONFIG_STRICT_MODULE_RWX is not enabled. Fix this by using PAGE_KERNEL_EXEC when CONFIG_STRICT_MODULE_RWX is not available. Signed-off-by: Xiaofeng Yuan <xiaofengmian@163.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Reviewed-by: Nam Cao <namcao@linutronix.de> Link: https://patch.msgid.link/20260814082742.148403-2-xiaofengmian@163.com Signed-off-by: Paul Walmsley <pjw@kernel.org>
patch_map() always creates a temporary writable mapping via fixmap for kernel text addresses, even when CONFIG_STRICT_KERNEL_RWX is disabled and the kernel text is already mapped with _PAGE_WRITE. This is unnecessary overhead at best, and on minimal configurations it can cause page faults. Skip the fixmap path for kernel text when CONFIG_STRICT_KERNEL_RWX is not enabled, since the text pages are already writable in that case. The module text path is already gated on CONFIG_STRICT_MODULE_RWX and is kept unchanged. Reported-by: Klara Modin <klara@kasm.eu> Closes: https://lore.kernel.org/all/ant_8TaBbov_GS4i@soda.int.kasm.eu/ Reported-by: Lad Prabhakar <prabhakar.csengg@gmail.com> Closes: https://lore.kernel.org/all/CA+V-a8tQK8rih9SGGTyqrEBGpNkx4H0eX2YccCRrgkVAPr+EBg@mail.gmail.com/ Tested-by: Klara Modin <klarasmodin@gmail.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://patch.msgid.link/20260814082742.148403-3-xiaofengmian@163.com Signed-off-by: Paul Walmsley <pjw@kernel.org>
This reverts commit 3033b2b. The reverted patch is userspace-visible behavior change, not a bug fix. The two variables here (pmm and pmlen) control two independent features: pmm is the _hardware_ pointer masking mode that applies while executing in userspace. pmlen is the shift amount that the _kernel_ uses when untagging addresses; PMLEN_0 means no untagging occurs, so the kernel does not accept tagged addresses in syscall arguments. It is valid (as documented and tested by the self test) to enable pointer masking without enabling the tagged address ABI. This separation is necessary to allow userspace to create an execution environment similar to what the kernel supports on arm64 by default, where TBI is enabled but the tagged address ABI is not. (On arm64, there is no equivalent to PR_PMLEN_MASK because TBI is always enabled.) Signed-off-by: Samuel Holland <samuel.holland@sifive.com> Link: https://patch.msgid.link/20260820014551.1979772-1-samuel.holland@sifive.com Cc: stable@vger.kernel.org Fixes: 3033b2b ("riscv: Reset pmm when PR_TAGGED_ADDR_ENABLE is not set") Signed-off-by: Paul Walmsley <pjw@kernel.org>
Prefer the convenient string choice 'str_disabled_enabled()' helper over hardcoded strings in 'setup_global_riscv_enable()'. Signed-off-by: Dmitry Antipov <dmantipov@yandex.ru> Link: https://patch.msgid.link/20260819160546.3219942-1-dmantipov@yandex.ru Signed-off-by: Paul Walmsley <pjw@kernel.org>
The vendor-extension handlers reached from hwprobe_one_pair() (hwprobe_isa_vendor_ext_thead_0() and friends) only OR the present bits into pair->value via VENDOR_EXTENSION_SUPPORTED() and clear their own missing bits; they assume the caller has already zeroed pair->value. That holds for hwprobe_get_values() (it zeroes each pair) and hwprobe_get_cpus() (it re-initializes its scratch pair per key), but not for complete_hwprobe_vdso_data(), which reuses a single pair across all keys without re-zeroing. A vendor key therefore inherits stale bits from the previously probed key, and the wrong value is cached in the vDSO all_cpu_hwprobe_values[] and handed to userspace on the fast patih. Zero pair->value once at the top of hwprobe_one_pair() so every handler starts from a clean value regardless of the caller, and drop the now redundant zeroing in the *_BLOCK_SIZE cases. hwprobe_isa_ext0() keeps its own zeroing because hwprobe_ext0_has() calls it directly, bypassing hwprobe_one_pair(). Fixes: a5ea53d ("riscv: hwprobe: Add thead vendor extension probing") Signed-off-by: Andy Chiu <tchiu@tenstorrent.com> Reviewed-by: Jesse Taube <jtaubepe@redhat.com> Link: https://patch.msgid.link/20260725001614.2578617-2-tchiu@tenstorrent.com Cc: stable@vger.kernel.org Signed-off-by: Paul Walmsley <pjw@kernel.org>
x86 did this in commit b0a848f ("x86/bugs: Make i386 use GENERIC_BUG_RELATIVE_POINTERS") powerpc did this in commit 1baa1f7 ("powerpc: Allow relative pointers in bug table entries") Similar as x86 and powerpc does, make RV32 use GENERIC_BUG_RELATIVE_POINTERS for "there is only one code path." and "less #ifdef is more better". Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Link: https://patch.msgid.link/20260221024255.3552-1-jszhang@kernel.org Signed-off-by: Paul Walmsley <pjw@kernel.org>
…Milk-V Jupiter
Hi,
On a Milk-V Jupiter, probing the PCIe controller at ca800000 (port C, the
card slot) permanently wedges the CPU that runs the probe when one
particular add-in card is installed. The CPU stops responding to NMI, and
because the probe is asynchronous, kernel_init() then blocks forever in
async_synchronize_full() and the machine never finishes booting.
The same card, in the same slot, on the same board, does *not* hang the
vendor 6.6 kernel -- it reports "Phy link never came up" and boots normally.
Six other cards do not hang mainline either. So whatever the electrical
cause, this looks like a robustness problem in pcie-spacemit-k1: an endpoint
should not be able to hang a host-side DBI register access.
HARDWARE
--------
Milk-V Jupiter v1.1, SpacemiT M1 (socinfo: CPU[M1-8571] REV[C] DRO[130]),
16 GiB LPDDR4X.
Firmware: stock vendor U-Boot 2022.10 (k1-bl-v2.2.9), unmodified.
Port B (ca400000, M.2) has a Samsung PM9B1 NVMe and works throughout.
Port C (ca800000) is the card slot -- an x8-length connector, silkscreened
PCIE_X2, wired x2.
REPRODUCED ON
-------------
v7.1 and v7.2, riscv defconfig (plus PHY_SPACEMIT_K1_USB2, USB_DWC3,
SPACEMIT_K1_TSENSOR, IGB, IGC, NVMe/ext4 built in).
gcc 13.3.0 (cross) and gcc 16.2.0 (native, Debian sid).
Identical failure in all combinations. Not a regression -- port C has
never worked with this card on mainline.
Command line:
console=ttyS0,115200 earlycon root=/dev/nvme0n1p2 rootwait rw
swiotlb=65536 clk_ignore_unused pd_ignore_unused
SYMPTOM
-------
Port C prints its address ranges and then never speaks again. (Log below is
from a run with port B disabled in DT, so nothing is interleaved.)
[ 1.290074] spacemit-k1-pcie ca800000.pcie: host bridge /soc/pcie-bus/pcie@ca800000 ranges:
[ 1.297283] spacemit-k1-pcie ca800000.pcie: IO 0x00b7002000..0x00b7101fff -> 0x0000000000
[ 1.312783] spacemit-k1-pcie ca800000.pcie: MEM 0x00a0000000..0x00afffffff -> 0x00a0000000
[ 1.326753] spacemit-k1-pcie ca800000.pcie: MEM 0x00b0000000..0x00b6ffffff -> 0x00b0000000
[22.348490] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[22.351764] rcu: 4-...0: (12 GPs behind) idle=051c/1/0x4000000000000000 softirq=43/43 fqs=1908
[22.367040] Sending NMI from CPU 2 to CPUs 4:
[32.367049] After 10 seconds, these CPUS still haven't responded to the NMI: 4
The CPU ignoring an NMI for ten seconds is why I read this as an MMIO access
that never receives a completion rather than a spin or a deadlock.
LOCALISATION
------------
I added a dev_info() before each step of k1_pcie_init() (patch at the end of
this mail). The last marker port C prints is the one immediately before the
first DBI access:
[1.347050] spacemit-k1-pcie ca800000.pcie: K1DBG 1 toggle_soft_reset
[1.362635] spacemit-k1-pcie ca800000.pcie: K1DBG 2 enable_resources
[1.370918] spacemit-k1-pcie ca800000.pcie: K1DBG 3 first DBI write (vendor/device ID)
<nothing further>
i.e. it dies in
dw_pcie_dbi_ro_wr_en(pci);
dw_pcie_writew_dbi(pci, PCI_VENDOR_ID, PCI_VENDOR_ID_SPACEMIT);
which is the first register access to the controller after
k1_pcie_enable_resources() has enabled the clocks and deasserted the resets.
I also tried moving phy_init() ahead of that block, in case the DBI domain
needed the PHY running (port B is masked here, because vendor U-Boot
initialises port B's PHY and never touches port C's). phy_init() returned
success and the DBI access still hung:
K1DBG 1 toggle_soft_reset
K1DBG 2 enable_resources
K1DBG 4 assert PERST# + 100ms
K1DBG 5 RC mode + AUX_PWR_DET
K1DBG 6 phy_init
K1DBG 3 first DBI write (vendor/device ID) <- still the last line
CARD MATRIX (port C, mainline 7.2, otherwise identical boots)
-------------------------------------------------------------
empty slot boots
Intel I210 [8086:1533] Gen1 x1 link up, enumerates
HP NC360T [8086:105e] Gen1 x2 link up, enumerates
NVIDIA Quadro P400[10de:1cb3] Gen1 x2 link up, enumerates
NVIDIA Quadro T400[10de:1fb2] Gen1 x1 link up, enumerates
AMD Radeon RX 550 [1002:699f] Gen1 x2 link up, enumerates
Sun ATLS1QGE "Device found, but not active"
HP NC375T "Device found, but not active"
Intel I225-V rev 01 *** CPU HANG ***
Note the two cards that do not train fail *politely* -- the DWC core logs
"Device found, but not active", the probe completes, an empty bus 0002:00 is
created and the machine boots. So port C is perfectly capable of handling a
link that never comes up. The I225-V is different, and it fails long before
link training is reached.
The I225-V card itself is good: it works on a MACCHIATObin (Armada 8040) in
the same office, and it works on this same Jupiter under the vendor kernel.
THE VENDOR DRIVER SURVIVES THE SAME CARD
----------------------------------------
Bianbu 2.3.5 / Linux 6.6.63, vendor k1x-dwc-pcie driver, same board, same
slot, same I225-V:
[ 1.657482] k1x-dwc-pcie ca400000.pcie: PCIe Gen.2 x2 link up
[ 2.949762] k1x-dwc-pcie ca800000.pcie: Phy link never came up
[ 2.952808] k1x-dwc-pcie ca800000.pcie: PCI host bridge to bus 0002:00
... boots to a login prompt
The vendor driver does not do the "set the PCI vendor and device ID" DBI
write at that point in its sequence.
HYPOTHESES ELIMINATED BY EXPERIMENT
-----------------------------------
nvme driver initcall_blacklist=nvme_init -- still hangs
power domains pd_ignore_unused -- still hangs
link training dies before dw_pcie_iatu_detect()
kernel version identical on v7.1 and v7.2
compiler identical with gcc 13.3 and gcc 16.2
PHY init ordering phy_init() first -- returns 0, still hangs
CLKREQ# pinmux CLKREQ# removed from pcie2_4_cfg -- still hangs
port B interference port B disabled in DT -- still hangs
power supply 12 V / 12.5 A bench supply, not USB-C PD
A POSSIBLY RELATED OBSERVATION
------------------------------
k1_pcie_init() writes PCI_VENDOR_ID_SPACEMIT / PCI_DEVICE_ID_SPACEMIT_K1 to
both ports. On this board the write takes effect on port B but not on port C
-- including on the boots where port C works fine:
pci 0001:00:00.0: [201f:0001] type 01 class 0x060400 PCIe Root Port
pci 0002:00:00.0: [1e5d:3003] type 01 class 0x060400 PCIe Root Port
1e5d:3003 is the hardware default (ASR Microelectronics). So that same DBI
read-only write is being silently dropped on port C even when it does not
hang. I do not know whether this is the same underlying issue, but it is in
the same function and on the same port, so it seemed worth mentioning.
WORKAROUND
----------
&pcie2 { status = "disabled"; };
or, from U-Boot, before booti:
fdt set /soc/pcie-bus/pcie@ca800000 status disabled
With that, mainline 7.2 boots Debian happily on this board with root on the
M.2 NVMe. Nobody's board is stuck; the slot is just unusable with this card.
Note this is not a problem with the DTS enablement of &pcie2 (added in 7.1) --
an empty slot, and six of seven cards, work fine.
WHAT I AM ASKING
----------------
I do not have the K1 documentation, so I have gone as far as I can from
outside. I would appreciate a pointer to what could gate that first DBI
access on port C -- CLK_PCIE2_DBI/MASTER/SLAVE, RESET_PCIE2_*, or something
in the PMU/APMU block -- and I am happy to run any test you like on this
board.
Separately, and regardless of the cause: a hard CPU hang on an unanswered
DBI read is an unpleasant failure mode, since there is no completion timeout
and no machine check to abort it on RISC-V. If there is a sane way to bound
it, that seems worth having.
Full logs for every boot referenced above are available on request.
Thanks,
Bruno Banelli
--------------------------------------------------------------------------------
Instrumentation used for the localisation above (not for merging):
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
|
Patch 1: "[BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter" |
495bff0 to
c27f74f
Compare
PR for series 1151269 applied to workflow__riscv__fixes
Name: [BUG] PCI: spacemit-k1: port C probe hard-hangs a CPU with one endpoint on Milk-V Jupiter
URL: https://patchwork.kernel.org/project/linux-riscv/list/?series=1151269
Version: 1