[PW_SID:1155270] mm: make userland page table freeing RCU-safe - #2563
[PW_SID:1155270] mm: make userland page table freeing RCU-safe#2563linux-riscv-bot wants to merge 15 commits into
Conversation
Overflow status and restart masks are u64, but bits were built with BIT(). On RV32 that is an unsigned long shift, so indices >= 32 truncate or wrap and corrupt the mask. Use BIT_ULL() for those u64 bitops. Fixes: a862521 ("drivers/perf: riscv: Implement SBI PMU snapshot function") Assisted-by: DeepSeek:deepseek-v3 Signed-off-by: Xixin Liu <liuxixin@kylinos.cn> Link: https://patch.msgid.link/prpmask01bitul.v2.1786434000.git.liuxixin@kylinos.cn [pjw@kernel.org: updated to apply] Signed-off-by: Paul Walmsley <pjw@kernel.org>
The available-counter mask was a single unsigned long, but iteration uses RISCV_MAX_COUNTERS, which is 64. On RV32 that reads past the object. Filling with an unsigned-long bit at index 32 and above is also wrong. Use DECLARE_BITMAP and set_bit/bitmap helpers. Walk each bitmap word into CFG_MATCH when checking events, when allocating an index, and when stopping all counters. Set the counter base to i times BITS_PER_LONG. Share the CFG_MATCH ecall through a small helper so the 32-bit argument split is not duplicated. On qemu-system-riscv32 the probe bitmap has bits above XLEN set, so the first word alone is not enough. Fixes: e999143 ("RISC-V: Add perf platform driver based on SBI PMU extension") Assisted-by: DeepSeek:deepseek-v3 Signed-off-by: Xixin Liu <liuxixin@kylinos.cn> Link: https://patch.msgid.link/prpmask02cmap.v2.1786434000.git.liuxixin@kylinos.cn [pjw@kernel.org: updated to apply] Signed-off-by: Paul Walmsley <pjw@kernel.org>
When an anonymous mapping is collapsed for THP, a PTE page table is 'deposited' with the installed PMD entry. This is done in order that a split can be performed without needing to allocate additional memory. The freeing occurs in zap_deposited_table() and is done directly without any delay via pte_free(). This is currently not a problem as existing page table walks are protected by the mmap or anon rmap lock. However this becomes problematic in a future where RCU-only page table walkers exist, as there is nothing to prevent a page table walker that started the walk prior to collapse having its PTE table freed underneath it. Commit 13cf577 ("mm/pgtable: add pte_free_defer() for pgtable as page") already provides us the mechanism by which to solve this - pte_free_defer(). Therefore, as a prerequisite to a future commit which will permit fully RCU page table walks, update zap_deposited_table() to use pte_free_defer() rather than pte_free(). Note that the IPI sync in collapse_huge_page() is still required to ensure refcount correctness against a GUP-fast operation. This is because GUP-fast might increment refcount, but __collapse_huge_page_isolate() determines whether it is safe to proceed by checking folio_ref_count() against folio_expected_ref_count(), so the two must be mutually excluded. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Commit e3ecf7c ("mm: pgtable: convert some architectures to use tlb_remove_ptdesc()") updated a number of architectures from using pagetable_dtor() + tlb_remove_page_ptdesc() to using tlb_remove_ptdesc() in __pte_free_tlb(). This is meaningful as tlb_remove_ptdesc() allows for RCU page table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE is specified. The csky, hexagon, nios2, openrisc, sh (except X2) and m68k-sun3 architectures all have 2 levels of page tables, so the only page tables ever freed by mmu_gather are PTEs, so this update suffices to ensure that every page table freed by the mmu_gather mechanism is freed under RCU. Therefore, update all of these architectures to select CONFIG_MMU_GATHER_RCU_TABLE_FREE. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Currently riscv gates MMU_GATHER_RCU_TABLE_FREE on CONFIG_SMP and CONFIG_MMU. Commit 69be3fb ("riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU") enabled CONFIG_MMU_GATHER_RCU_TABLE_FREE for CONFIG_SMP, CONFIG_MMU riscv builds. This is expressly for the safety of GUP-fast walkers (CONFIG_HAVE_GUP_FAST is enabled if CONFIG_MMU is enabled). Naturally a single core system does not encounter issues with software page table walkers being correctly synchronised across cores, as there is only a single core. However, CONFIG_PREEMPT_RCU is still available on a riscv UP system, so for a future RCU-only page table walker, this guarantee is required to prevent concurrent page table teardown. All page table freeing is already done via tlb_remove_ptdesc() so the conditions of CONFIG_MMU_GATHER_RCU_TABLE_FREE are already met. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Commit a0ad549 ("arm: mm: enable HAVE_RCU_TABLE_FREE logic") enabled CONFIG_MMU_GATHER_RCU_TABLE_FREE (then named HAVE_RCU_TABLE_FREE) for SMP arm architectures with LPAE enabled. Regardless of whether CONFIG_ARM_LPAE is enabled or not, the same page table freeing functions __pte_free_tlb() and __pmd_free_tlb() are used. Non-LPAE PMD page tables are folded into the PGD and freed by pgd_free() (PGD freeing is not part of mmu_gather page table freeing in any case), so this is a noop in this case. Since commit 358d1c3 ("arm: convert various functions to use ptdescs") both LPAE and non-LPAE PTE page table freeing uses tlb_remove_ptdesc(). Thus all page table freeing is performed under RCU with CONFIG_MMU_GATHER_RCU_TABLE_FREE enabled for LPAE and non-LPAE and thus it need not be gated on LPAE. A UP arm system can set CONFIG_PREEMPT_RCU, so a future pure RCU page table walker requires MMU_GATHER_RCU_TABLE_FREE to be enabled on UP as well, even if concurrent GUP fast is not possible there. Therefore, it is both safe and desirable to set CONFIG_MMU_GATHER_RCU_TABLE_FREE for all MMU arm architectures (nommu does not perform mmu_gather operations). This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Each of these architectures directly free page tables without routing these changes through tlb_remove_ptdesc(). The use of tlb_remove_ptdesc() is required for CONFIG_MMU_GATHER_RCU_TABLE_FREE to correctly free page tables under RCU, so simply update these architectures to use these functions. Since none of the architectures share page tables or do anything unusual, nothing complicated is required here. Therefore this is simply a mechanical change - convert __pud_free_tlb(), __pmd_free_tlb() and __pte_free_tlb() to use tlb_remove_ptdesc() as required. At the point this is in place, all mmu_gather page table freeing is performed under RCU, and thus MMU_GATHER_RCU_TABLE_FREE is selected for each architecture. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Commit 4a0100f ("sparc64: use RCU page table freeing") enabled CONFIG_MMU_GATHER_RCU_TABLE_FREE for SMP sparc64 architectures, expressly for GUP-fast page table walkers. Naturally, UP systems do not have to worry about concurrent GUP fast operations. However, CONFIG_PREEMPT_RCU is also available even on a UP system, so a future pure-RCU page table walker requires MMU_GATHER_RCU_TABLE_FREE to be enabled on UP, even if concurrent GUP fast is not possible there. To enable future pure-RCU page table walkers, enable MMU_GATHER_RCU_TABLE_FREE unconditionally. With this change, it is no longer necessary to have !CONFIG_SMP pgtable_free_tlb(), so also remove this now dead code. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Similar to sun3, the coldfire variant of m68k uses 2-level page tables. Update its __pte_free_tlb() function to use tlb_remove_ptdesc() in order that, with CONFIG_MMU_GATHER_RCU_TABLE_FREE, page tables are freed under RCU. The page tables occupy a page each and have no odd semantics, so this change suffices to allow enabling of CONFIG_MMU_GATHER_RCU_TABLE_FREE for m68k-coldfire, so do so. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Currently, non-x2 sh specifies CONFIG_MMU_GATHER_RCU_TABLE_FREE allowing RCU page table freeing. sh-X2 is problematic because it utilises slab-allocated PMD page tables, and thus tlb_remove_ptdesc() cannot be used in these cases. All other sh variants are fine as commit e3ecf7c ("mm: pgtable: convert some architectures to use tlb_remove_ptdesc()") already converted page table freeing to use tlb_remove_ptdesc(), which does so after an RCU grace period when CONFIG_MMU_GATHER_RCU_TABLE_FREE is specified. Resolve this issue by firstly specifying CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE for sh-X2, so the arch can provide its own __tlb_remove_table() implementation (called after the RCU grace period). Then, convert __pmd_free_tlb() to tag the pointer to the PMD, and have __tlb_remove_table() check this tag to determine whether to free via the slab or to use pagetable_dtor_free(). This follows the pattern used by sparc64 as implemented in commit 4a0100f ("sparc64: use RCU page table freeing"). Previously __pmd_free_tlb() freed PMD page tables immediately, before any TLB flush IPI. This seems to be a pre-existing bug, which this change also resolves. CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE is only specified for sh-X2, as setting it disables CONFIG_PT_RECLAIM and causes __tlb_remove_table_one() to call tlb_remove_table_sync_rcu() and synchronize_rcu() in turn, and this is not necessary for other sh variants. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
sun3 and coldfire are already supported, however motorola requires a little more care. Here, custom table removal logic is required, so CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE is enabled for m68k-motorola. Firstly as part of this change, the page table level must be communicated to the underlying __tlb_remove_table() implementation. Take advantage of the fact that page tables are aligned by more than enough to permit setting TABLE_PTE or TABLE_PMD in the low bits of the pointer, and store this there. Then update __pte_free_tlb() and __pmd_free_tlb() to pass this through, then have __tlb_remove_table() decode this and pass it to free_pointer_table(). The page table freeing is performed via call_rcu(), so free_pointer_table() now will be invoked from softirq context, and as such may be re-entrant. Introduce an irq save/restore spinlock to handle this, and hold it over the time a given ptable entry is being referenced in both get_pointer_table() and free_pointer_table(). In order to make things a little easier in this respect, separate out the logic for adding a new ptable entry into add_pointer_table() and only hold the lock during ptable entry insertion in this case. Note that original list_add_tail(new, dp) added new prior to dp, which is ptable_list[type].next, i.e. after ptable_list[type]. The equivalent therefore is list_add(new, &ptable_list[type]), which adds new after ptable_list[type], only without needing to make reference to dp. Note that, as m68k-motorola specifies CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE, it does not enable CONFIG_PT_RECLAIM. This isn't meaningfully impactful. With this applied, all of m68k implements CONFIG_MMU_GATHER_RCU_TABLE_FREE. This forms part of an overall effort to switch every architecture to this mode. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Careful handling is required for sparc32 which implements page tables as part of a shared backing page. To support this, a custom __tlb_remove_table() function is required, as specified by CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE. This allows __pte_free_tlb() and __pmd_free_tlb() to specify which page table level is being freed, which is transmitted to __tlb_remove_table() through setting the lowest bit of the page table to 1 for a PMD and 0 for a PTE (the page tables are 256-byte aligned so this is safe to do). Next, since the page table freeing is done via RCU callback, and thus might be executed in softirq context, update the spin locks to IRQ save/restore. Then, in __tlb_remove_table(), figure out whether to free a PMD page table via free_pmd_fast() or a PTE via the newly introduced __pte_free() function, using the lower bit encoded in __pte_free_tlb() or __pmd_free_tlb() to determine which to call. __pte_free() is identical to preexisting pte_free(), except that it optionally allows a NULL mm pointer to be provided, in which case there is no mm whose mm->page_table_lock can be taken. This lock doesn't appear to have been doing quite as much as it intended, as backing pages can contain page tables for multiple mm's, which are not serialised by it. But more importantly - the reference count increment in pte_alloc_one() and decrement in __pte_free() are atomic with full ordering, so it simply isn't possible for there to be a meaningful race here. Note that the specification of CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE disables CONFIG_PT_RECLAIM for sparc32, which mirrors sparc64. This forms part of an overall effort to switch every architecture to this mode, and with it complete, means every architecture now supports CONFIG_MMU_GATHER_RCU_TABLE_FREE. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Now every architecture has been converted to support CONFIG_MMU_GATHER_RCU_TABLE_FREE, this configuration option no longer makes any sense to keep around. Therefore remove it, and remove all the dead code that existed for !CONFIG_MMU_GATHER_RCU_TABLE_FREE architectures previously. Additionally, CONFIG_MMU_GATHER_TABLE_FREE is no longer necessary, as all architectures instead use CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE when a custom __tlb_remove_table() is required, so remove this too. A number of architectures only enabled CONFIG_MMU_GATHER_RCU_TABLE_FREE if CONFIG_MMU was set, however the mmu_gather logic only actually does something meaningful if CONFIG_MMU is set (mmu_gather.c is only compiled in this case, for instance). As a result, there's no need to gate any of this logic on CONFIG_MMU explicitly. CONFIG_PT_RECLAIM however does have a strict dependency on CONFIG_MMU, so make this dependency explicit. Additionally, correct comments to remove references to non-RCU page table gathering and make it clear that this is not 'semi-RCU', nor has it been since commit 1fb3d8c ("mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails"). With this change in place the kernel policy is now that all page tables are freed after an RCU grace period, and thus it is now safe to unconditionally perform page table walks under RCU, safe in the knowledge that page tables will not be freed underneath the walker. This is all that is guaranteed, however, so naturally it is still incumbent upon page table walkers to ensure that the page table entries are as expected. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Now that page tables are freed after an RCU grace period, it is safe for page table walkers to walk page table ranges that are being concurrently torn down, provided the mm is kept alive via mmgrab(). The comment block before pte_offset_map_lock() established a contract that this was unsafe, which was correct prior to these changes. Update it to reflect the change. Similarly update the process addresses documentation. Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 1: "[01/12] mm/huge_memory: zap deposited page tables after an RCU grace period" |
|
Patch 2: "[02/12] mm: enable MMU_GATHER_RCU_TABLE_FREE for most 2-level architectures" |
|
Patch 2: "[02/12] mm: enable MMU_GATHER_RCU_TABLE_FREE for most 2-level architectures" |
|
Patch 2: "[02/12] mm: enable MMU_GATHER_RCU_TABLE_FREE for most 2-level architectures" |
|
Patch 10: "[10/12] mm: enable MMU_GATHER_RCU_TABLE_FREE for sparc32" |
|
Patch 10: "[10/12] mm: enable MMU_GATHER_RCU_TABLE_FREE for sparc32" |
|
Patch 10: "[10/12] mm: enable MMU_GATHER_RCU_TABLE_FREE for sparc32" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 11: "[11/12] mm: make userland page table freeing RCU-safe" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
495bff0 to
c27f74f
Compare
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
|
Patch 12: "[12/12] mm: change the contract for free_pgtables(), update docs" |
9ed823f to
a54e736
Compare
1 participant
PR for series 1155270 applied to workflow__riscv__fixes
Name: mm: make userland page table freeing RCU-safe
URL: https://patchwork.kernel.org/project/linux-riscv/list/?series=1155270
Version: 1