RK3576 NPU (RKNN) support + enable on Flipper One - #23
Draft
Notnurb wants to merge 18 commits into
Draft
Conversation
rocket_ioctl_submit_job() releases rjob through rocket_job_put() on allocation error paths. rocket_job_cleanup() unconditionally calls rocket_iommu_domain_put(job->domain), but job->domain is assigned only after task copying and BO lookups. A failure before that assignment can therefore clean up a job with a NULL domain pointer. Take the per-file domain reference before the first error path can release rjob. Also clear rjob->tasks after freeing it in rocket_copy_tasks(), so the common cleanup path cannot free the task array again after a task-copy error. Fixes: 0810d5a ("accel/rocket: Add job submission IOCTL") Cc: stable@vger.kernel.org Signed-off-by: Shuvam Pandey <shuvampandey1@gmail.com> Link: https://lore.kernel.org/r/6a454b48.6a8fa39a.27019b.984b@mx.google.com Signed-off-by: Tomeu Vizoso <tomeu@tomeuvizoso.net>
…_push()
rocket_job_push() allocates a temporary array to hold all input and
output GEM object pointers:
bos = kvmalloc_array(job->in_bo_count + job->out_bo_count,
sizeof(void *), GFP_KERNEL);
memcpy(bos, job->in_bos, job->in_bo_count * sizeof(void *));
memcpy(&bos[job->in_bo_count], job->out_bos, ...);
Two bugs exist:
1. Missing NULL check: if kvmalloc_array() fails, bos is NULL and
the subsequent memcpy() dereferences it, causing a kernel NULL
pointer dereference.
2. Integer overflow: in_bo_count and out_bo_count are both u32, set
directly from userspace-supplied in_bo_handle_count and
out_bo_handle_count with no prior validation. Their sum is computed
in u32 arithmetic and can wrap to a smaller value, causing the
allocation count passed to kvmalloc_array() to be smaller than
intended. Subsequent uses still operate on the original counts when
copying and locking objects, which may lead to out-of-bounds accesses
on the temporary array.
Fix by using check_add_overflow() to detect count overflow before the
allocation, and adding a NULL check on the allocation result.
Fixes: 0810d5a ("accel/rocket: Add job submission IOCTL")
Cc: stable@vger.kernel.org
Signed-off-by: Muhammad Bilal <meatuni001@gmail.com>
Link: https://lore.kernel.org/r/20260524155716.90955-1-meatuni001@gmail.com
Signed-off-by: Tomeu Vizoso <tomeu@tomeuvizoso.net>
In rocket_job_run(), after taking an extra fence reference for
job->done_fence via dma_fence_get(), the error paths have three bugs:
- The dma_fence reference held by job->done_fence is never released,
causing a reference leak.
- pm_runtime_get_sync() increments the usage counter even on failure,
but the error path does not decrement it, leaking the runtime PM
reference and preventing the NPU from suspending.
- A valid but unsignaled fence is returned to the DRM scheduler,
which triggers WARN("Fence ... released with pending signals!")
when the scheduler drops its reference.
Fix by replacing pm_runtime_get_sync() with pm_runtime_resume_and_get()
which auto-balances the usage counter on failure, releasing both fence
references on error, and returning ERR_PTR(ret) instead of the
unsignaled fence.
Cc: stable@vger.kernel.org
Fixes: 0810d5a ("accel/rocket: Add job submission IOCTL")
Signed-off-by: ZhaoJinming <zhaojinming@uniontech.com>
Link: https://lore.kernel.org/r/20260610071045.3414828-1-zhaojinming@uniontech.com
[tomeu: Refactored error paths to use consolidated goto labels]
Signed-off-by: Tomeu Vizoso <tomeu@tomeuvizoso.net>
The RK3568 NPU rail (vdd_npu) needs to be enabled before the domain is powered on and disabled after it is powered off. Give DOMAIN_RK3568 a regulator parameter (like DOMAIN_RK3588 already has) so the NPU domain can set need_regulator, letting genpd manage the rail wired up as the domain's domain-supply instead of marking it always-on in DT. Suggested-by: Chaoyi Chen <chaoyi.chen@rock-chips.com> Signed-off-by: Midgy BALON <midgy971@gmail.com> Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.com> Reviewed-by: Heiko Stuebner <heiko@sntech.de> Signed-off-by: Ulf Hansson <ulfh@kernel.org>
rocket_core_init() hands core->clks to devm_clk_bulk_get() without ever
setting the .id members. The rocket_core array is allocated with
devm_kcalloc() in rocket_device_init(), and rocket_probe() only fills in
.rdev, .dev and .index, so all four clk_bulk_data entries are requested
with a NULL con_id (unlike core->resets, whose ids are set a few lines
above).
clk_get(dev, NULL) ends up in of_clk_get_hw(np, 0, NULL), and
of_parse_clkspec() only consults "clock-names" when a name was passed,
so the index stays 0 for all four entries. Every entry therefore ends up
holding a handle to the *first* clock of the DT "clocks" property, i.e.
ACLK_NPUn. Nothing fails: probe succeeds and the driver believes it owns
four different clocks.
The consequence is that rocket_device_runtime_resume() prepares and
enables the AXI clock four times, while hclk, pclk and - most
importantly - the NPU compute clock ("npu", SCMI_CLK_NPU on RK3588) are
never prepared or enabled by this driver at all. The NPU still works
only because the Rockchip power-domain driver sets GENPD_FLAG_PM_CLK and
its attach_dev() callback walks the device node with of_clk_get() and
adds every clock to the pm_clk list, so genpd happens to keep the
remaining clocks running. The bug is therefore latent today, but it
means the driver holds no reference to the clock that actually feeds the
NPU, which stands in the way of any future frequency scaling
(OPP/devfreq) work.
Found on an Orange Pi 5 Plus (RK3588) by reading the live clock tree:
/sys/kernel/debug/clk/clk_summary shows four "fdab0000.npu" consumer
handles on aclk_npu0 (and likewise on aclk_npu1/aclk_npu2 for the other
two cores), while hclk_npu0, pclk_npu_root and scmi_clk_npu have no
"fdab0000.npu" consumer at all - their only consumers are the
"npu@fdab0000" handles created by the power-domain driver via
of_clk_get().
Set the ids explicitly, in the order mandated by the binding
(Documentation/devicetree/bindings/npu/rockchip,rk3588-rknn-core.yaml):
aclk, hclk, npu, pclk. After the change the driver holds one handle per
distinct clock and clk_bulk_prepare_enable() covers all four.
Note that this is a user-visible tightening for out-of-tree DTs: the
old NULL-id requests resolved by index and succeeded no matter what
"clock-names" contained, while the named requests fail probe with
-ENOENT when one of the four names is missing. That is the right
outcome for in-tree users - the binding requires exactly these four
clock-names and rk3588-base.dtsi carries them on all three cores - but
a DT that relied on the permissive lookup goes from silently running on
the wrong clock handles to not probing at all, so record the change
here where git log will find it.
Fixes: ed98261 ("accel/rocket: Add a new driver for Rockchip's NPU")
Signed-off-by: Igor Paunovic <royalnet026@gmail.com>
Reviewed-by: Jiaxing Hu <gahing@gahingwoo.com>
rocket_job_handle_irq() writes OPERATION_ENABLE and INTERRUPT_CLEAR before taking job_lock, while rocket_job_hw_submit() writes OPERATION_ENABLE from inside it. The two can therefore race: a completion being handled on one core can write its zero after a submit on the same core has written its one, and stop a task that has only just started. Nothing in tree hits this often, because the interrupt is the only completion path and it does not overlap its own submit, but the ordering is wrong on its own terms. Move both writes inside the existing scoped_guard() rather than adding a second critical section, so stopping the block and deciding what to start next are one atomic step. Fixes: 0810d5a ("accel/rocket: Add job submission IOCTL") Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com> Tested-by: Igor Paunovic <royalnet026@gmail.com> # RK3588, three cores
rocket_reset() calls drm_sched_stop(), which stops the scheduler and returns. It does not wait for a threaded handler that is already running, so the comment that follows, "Remaining interrupts have been handled", states an assumption rather than something the code arranges. Call synchronize_irq(core->irq) after drm_sched_stop() and reword the comment to say what holds afterwards. It has to go before the scoped_guard(mutex, &core->job_lock) rather than inside it. rocket_job_handle_irq() takes job_lock, so waiting for the handler while holding that lock would be waiting for a handler that is waiting for us. Nothing is held at that point, and both callers, rocket_job_timedout() and rocket_reset_work(), run in process context, so sleeping there is allowed. This does not stop a handler that has already read in_flight_job from finishing its work on the job the reset is about to drop. That window needs the check and the register writes to be one step under the lock, which is what the previous patch does; the two are complementary. Mask the block before the sync as well. INTERRUPT_MASK is armed by hw_submit() on every submit and cleared only by the hardirq, so on an ordinary timeout it is still live and a completion can arrive after synchronize_irq() returns. Nothing is lost by clearing it, since the next submit arms it again. That write is the first register access this function has ever made, and it is guarded, because the function holds no runtime PM reference of its own. The only reference in the window belongs to in_flight_job, and the completion path can have put it and cleared the pointer before the timeout worker arrives: drm_sched_stop() sits in between and can block on cancel_work_sync() and on a dma_fence_wait(), and it subtracts every pending job's credits, so rocket_job_is_idle() is true and rocket_device_runtime_suspend() will not refuse. With the autosuspend delay elapsed the clocks are off and both NPU domains are down. A register access in that state takes an async SError on this hardware, which is the failure two later patches in this series describe from the power-on side. pm_runtime_get_if_active() resumes nothing and allocates nothing; if the core is already down there is no live interrupt to mask and the following synchronize_irq() is all that is needed. Igor Paunovic asked the general form of this on v8 -- whether rocket_reset() should hold a reference -- and it was deferred then because nothing in the path touched a register. This patch is what makes it matter. The deadlock this placement avoids would not have been reported. The wait is on desc->wait_for_threads rather than on a lock, so lockdep does not model it and it would have hung silently. Suggested-by: Igor Paunovic <royalnet026@gmail.com> Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
rocket_reset() drops the in-flight job's runtime PM reference with pm_runtime_put_noidle(), a bare decrement that requests nothing. The core is left at usage_count 0 but still runtime-active with no idle request pending, so it does not suspend until something else asks, and on a platform whose power domain does work on power-on that work never happens. On RK3576 that work is a bus interface reset the domain cycles when it comes up. Without it the NPU's IOMMU stops answering, and the job after a timeout returns a surface of the output zero point with rk_iommu reporting that MMU_DTE_ADDR is not functioning. Measured on a ROCK 4D in one boot, three runs, one variable between them. With the bare put the core reads runtime-active with its rail still up after the reset, the IOMMU reports the failure on the next attach and the inference returns 0 of 128 channels. With the reference put back through pm_runtime_put_autosuspend() the core reads suspended with the rail down, there is no IOMMU message, and the same inference returns 128 of 128. A third run repeating the first failed the same way. It also matches the put in the completion path a few lines away, so the reset path no longer leaves the device in a state the rest of the driver never produces. The remaining put, on the error path in rocket_job_run(), is a plain pm_runtime_put() and is left alone here: it unwinds a get_sync() that never reached the hardware, and changing it belongs in its own patch. Igor Paunovic ran the differential on RK3588: 45 induced resets across three cores, with and without the two preceding patches, and the domain dropped every single time with no MMU message on either kernel. So this is not rocket-wide. His conditions cross a healthy block with a lowered timeout rather than a hung one, which he was careful to say his protocol cannot settle, but it is what scopes the change to RK3576. Link: https://lore.kernel.org/all/20260819073530.6087-1-royalnet026@gmail.com/ Fixes: 0810d5a ("accel/rocket: Add job submission IOCTL") Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
rocket_job_handle_irq() stops the block and then either starts the job's next task or retires the job. The second half is a step of its own and reads better with a name, now that taking the register writes under job_lock has moved it a level deeper inside the scoped guard. Move it to rocket_job_next_locked(). The early return that used to leave the handler now leaves the helper, which is the same thing here: the scoped guard drops job_lock either way and nothing follows it. Doing it as its own patch keeps the locking fix at the head of the series minimal, so a bisect that stops before this one gets that fix and nothing else. There is one caller, and no functional change. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com> Reviewed-by: Igor Paunovic <royalnet026@gmail.com>
The RK3576 NPU has two cores of the same RKNN block the RK3588 binding already describes, but it wires them up differently: two extra CBUF clocks, two power domains per core, and a single reset instead of two. It also has no NPU SRAM supply. Widen the property ranges to cover both, then pin each SoC back to its own shape in allOf so nothing loosens for RK3588, and keep sram-supply required for rockchip,rk3588-rknn-core only. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Some domains do not come up in a usable state on their own and need their resets cycled once power is on. The RK3576 NPU domains are one case: without it the first access after power-on takes an async SError. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
The RK3576 NPU MMUs are rk3568-iommu compatible but take five clocks where every other Rockchip MMU takes two, the extra three being the compute clock and the two convolution buffer clocks. Give them a compatible of their own and pin both sides with an allOf, so that an rk3568-iommu cannot carry five clocks and an NPU MMU cannot carry two. Describing the extra clocks as belonging to one SoC without saying so in the schema, which is what a comment on a description does, leaves both of those spellings valid. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
The RK3576 NPU domains need a short settle time after the idle request is released before the registers behind the domain answer. Without it the QoS writes that rockchip_pmu_restore_qos() issues land while the domain is still coming up, and the NPU throws an async SError on the first cold power-on. Give rockchip_domain_info an optional delay_us and wait for it between releasing idle and restoring QoS. Rename DOMAIN_M_O_R_G to DOMAIN_M_O_R_G_W, since the suffixes name the fields the macro sets and this one now also carries a wakeup delay; RK3576 is its only user, so the old spelling is not kept around. While the macro is being rewritten, give it the regulator argument that DOMAIN_M_O_R and DOMAIN_M_R already take. Without .need_regulator set, rockchip_pd_regulator_enable() returns early for every RK3576 domain, so a domain-supply in the device tree is never looked up and never enabled. Add a DOMAIN_RK3576_R spelling that passes true and use it for RK3576_PD_NPU, which is the one RK3576 domain with a rail of its own; every other domain passes false and is unchanged. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
Some Rockchip domains come out of power-on with their bus interface in an undefined state. On the RK3576 NPU this shows up as a hang on the first register access after the domain is switched on, and pulsing the domain's resets at this point clears it. Take the domain node's resets if it has any, and pulse them between releasing idle and restoring QoS. The resets are optional, so domains that do not list any are unaffected. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
The RK3576 carries the same RKNN block with a different set of clocks and resets, so the counts cannot stay compile-time constants. Add a soc_data struct to the of_device_id match data and take the bulk counts from it. RK3588 keeps four clocks and two resets, so nothing changes for it, and the arrays keep their present sizes: the SoC that needs a longer one grows it in the patch that adds the names. rocket_core_reset() is switched over as well. It is the same array, and leaving it on ARRAY_SIZE() would walk entries that were never acquired once a SoC asks for fewer. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
The RK3576 has two cores of the same RKNN block and a few platform differences: - the CBUF (convolution buffer) has its own clock domain, so the core needs six clocks rather than four; - there is no per-core hclk reset. The CRU has SRST_A_RKNN0 and SRST_A_RKNN1 but no SRST_H_RKNN0 or SRST_H_RKNN1, so a core takes one reset where RK3588 takes two; - the NPU spans two power domains, and a device with more than one is skipped by the driver-core single-domain auto-attach, so the list has to be attached explicitly; - PC_TASK_CON packs the task number with sixteen bits rather than twelve, moving the three controls above it up by four. That last one is the reason this series has been reporting, since v3, that the block accepts exactly one task per reset. rocket_registers.h is generated from the RK3588 description, so writing it unchanged to an RK3576 asks for task_number 0x7001, which is 28673 tasks, and puts TASK_COUNT_CLEAR on a bit that does nothing. The counter is then only ever cleared by a reset. The layout was confirmed by Chaoyi Chen of Rockchip, including a fourth control at BIT(18), task_last_layer_clear, which belongs on every submit alongside the count clear: https://lore.kernel.org/all/4f300b78-d96d-4d98-8819-dc292b0c9b97@rock-chips.com/ With that written correctly a job of several tasks runs to completion, the completion interrupt arrives, and /proc/interrupts counts up. A convolution submitted three times with three different inputs is byte exact against the CPU reference each time, with no reset in between and with nothing retiring the job but the interrupt. Counting the cores now walks the driver's own match table instead of a second, hand-kept list of compatibles. The array sized from that count is indexed by every core that goes on to probe, so the two lists cannot be allowed to disagree. All of it hangs off the soc_data added earlier, so the RK3588 path keeps its existing counts and behaviour. The match table moves to rocket_drv.h so rocket_device.c can walk it with for_each_matching_node() rather than repeating a for_each_compatible_node() loop per SoC, which also keeps num_cores in step with the table that sizes the array it counts into. The declaration needs struct of_device_id, taken from <linux/device-id/of.h> rather than <linux/mod_devicetable.h>, which carries every subsystem's tables with it. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
Add the two RKNN cores and their IOMMUs. Both cores are disabled by default; boards enable what they wire up. PD_NPU0 and PD_NPU1 are siblings under PD_NPUTOP and hold one core each, but the convolution buffer and the DSU sit above them: ACLK_RKNN_CBUF, HCLK_RKNN_CBUF and CLK_RKNN_DSU0 belong to the block rather than to either core, and PD_NPUTOP already lists all three. Add them to both core domains as well, so a core domain switching state has the clocks of the path it shares running, and give each core domain the BIU reset that the pmdomain driver now cycles once power is on. Each core lists both core domains, its own first, so that a core in use has the whole block powered. Whether a single core can reach the shared path with the sibling domain off is not something this series establishes; listing both is the description that has been tested here. The IOMMU in front of each core lists that core's domain only. Label the outer PD_NPU node so a board can attach the NPU rail to the domain that gates the block. Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
Wire vdd_npu_s0 (PMIC dcdc-reg2) into the NPU power domain and enable the first RKNN core with its MMU. Without domain-supply on pd_npu the domain cannot be powered, and the core's npu-supply is what keeps the regulator from being switched off as unused during late init. Only core 0 is enabled. The RK3576 has two, but single-core is the supported bring-up configuration and core 1 needs its own validation before it is turned on. Both board revisions inherit this from the shared dtsi.
Collaborator
|
I’m very skeptical of untested contributions. Asking an LLM to patch stuff is not the hard part: understanding what it does and what it missed is. And statements like “I don’t have a cross-compiler so I didn’t even try building what the LLM produced” are outright disrespectful of maintainers’ time |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Not built, not booted. No cross toolchain or container was available on the machine this was prepared on, so nothing here has been compiled or run on hardware. Opening it so the shape of the change is reviewable, not because it is ready to merge.
What this is
18 commits enabling the RKNN NPU on RK3576. Only the last one is ours; the other 17 are other people's work, applied with
git amso authorship and Signed-off-by are intact.gahingwoo/linux-rk3576-npu)The risk to weigh: the 12-commit v9 series is unmerged and may change before it lands. If we take this, we are carrying an in-review series in a product kernel and will need to rebase as it churns. The alternative is waiting for it to merge upstream. That is a call for whoever owns the kernel branch, which is why this is a draft.
v9-0013from the original series is deliberately dropped. It enables the NPU on ROCK 4D, which is not our board. Our commit replaces it.Our commit
arm64: dts: rockchip: flipper-one: Enable the NPUdomain-supply = <&vdd_npu_s0>on&pd_npu, without which the domain cannot be powerednpu-supply = <&vdd_npu_s0>plusstatus = "okay"on&rknn_core_0, and&rknn_mmu_0enabledrk3576-flipper-one.dtsi, so bothrev-f0b0c1andrev-f0b1c2inherit it.What was actually verified
Static only, no compiler involved:
cpp+dtc): f0b0c1 95014 bytes, f0b1c2 95972 bytesnpu@27700000isrockchip,rk3576-rknn-core,status = "okay", and itsnpu-supplyphandle resolves to thedcdc-reg2node namedvdd_npu_s0pd_npu'sdomain-supplyresolves to that same phandlenpu@27708000andiommu@2770a000arestatus = "disabled", as intendednum_clks = 6against 6clock-names,num_resets = 1againstsrst_arocket_device.cusesfor_each_matching_node(rocket_dt_match)filtered onof_device_is_available(), so the disabled core 1 is not countedWhat was NOT verified
/dev/accel/accel0, no inferencedtbs_checkagainst dt-schemavdd_npu_s0survives past the late_initcall that drops unused regulators, which is the failure this whole series exists to avoidrocketGallium driver, but it has no RK3576 path, so even with this merged nothing will run models yetCompanion PR
flipperdevices/flipperone-linux-build-scripts#159 makes
CONFIG_DRM_ACCEL_ROCKETbuilt-in, which is required for any of this to work: as a module it probes after the regulator cleanup and the device never appears.