From bf5c3a210a98463b60ad426bac78c16f4b7b62db Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Thu, 2 Jul 2026 10:07:24 -0400 Subject: [PATCH 1/5] Storage pool capacity TSG: capacity-pressure guidance and correct VM-offline handling Follow-up to #300 addressing the post-merge change request (add capacity escalation guidance; steer away from "pause VM" under capacity pressure). - Clarify pool allocation vs volume fill: the 80/90 figures are the default *volume* CapacityThreshold.Warning/.Critical health settings, not pool auto-actions. The pool raises only the configurable 70% thin alert (PoolCapacityThresholdExceeded) and the reserve-capacity fault, and goes read-only only on quorum loss or admin policy -- there is no 80/90/95 pool ladder and no capacity-based read-only "block." Verified against Microsoft Learn, Azure Local Health Service settings, and the product fault model. - Add "what happens if the pool fills" escalation plus an ordered admin response (audit/prune, restrict provisioning, freeze auto-expansion, expand). - Add a CAUTION against saving VM state under capacity pressure: Save writes a memory-sized saved-state file to the volume, and a Save automatic stop action can flood an already-full CSV on a cluster-wide stop. Set the stop action to Shut Down; a node drain is space-safe (live-migrates). - Path B: prefer a clean Stop-VM shutdown (releases VHDX handles, no saved-state file), Stop-VM -TurnOff as fallback. Correct that Suspend-VM only pauses (memory in RAM, handles stay open, so consolidation cannot run) and that Save is what writes the memory-sized file. - References: add Health Service settings and Set-VM. --- ...ot-Storage-StoragePoolCapacityThreshold.md | 133 +++++++++++++++--- 1 file changed, 114 insertions(+), 19 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 7201d9ed..80af4f33 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -107,6 +107,87 @@ effect if the Health Service alert has already been disabled (Option A4), and disabling the alert silences it regardless of the threshold value. Decide which layer you intend to act on before changing anything. +### Pool capacity is not the same as volume capacity (do not confuse the signals) + +Two different capacity signals exist, and they are frequently conflated: + +- **Pool allocation** — how much of the storage *pool* is committed to virtual + disks. This is what the **thin provisioning alert threshold (default 70%)** and + the **pool reserve-capacity** check (`System.Storage.StoragePool.CheckPoolReserveCapacity`) + evaluate. Pool allocation is the subject of this guide. +- **Volume fill** — how full an individual *volume's* file system is. The Health + Service evaluates this against separate, **volume-level** settings whose defaults + are `System.Storage.Volume.CapacityThreshold.Warning = 80` and + `.Critical = 90` + ([Health service settings](https://learn.microsoft.com/azure/azure-local/manage/health-service-settings)). + An **80%** or **90%** figure quoted for "capacity" almost always refers to these + **volume** thresholds, **not** to an automatic pool action. + +The pool has **no automatic 80/90/95 capacity ladder and no capacity-based +read-only "block."** At the pool level the platform raises only two advisory, +Warning-class health faults — `StoragePool.PoolCapacityThresholdExceeded` (the +configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacity` +— and neither takes automatic action. A Storage Spaces pool is set **read-only** +only on **quorum loss** (too many drives offline — operational state `Incomplete`) +or by **administrator policy** (`Policy`), never from a capacity percentage. See +[Storage Spaces and Storage Spaces Direct health and operational states](https://learn.microsoft.com/windows-server/storage/storage-spaces/storage-spaces-states). +What actually happens as a pool approaches full is described next. + +### What happens if the pool is allowed to fill + +Treat the 70% alert as an **early warning**, not the danger line — the real safety +floor is the reserve (the equivalent of one capacity drive per server, up to four). +As pool allocation climbs past the reserve toward full, risk escalates: + +- **Repair headroom shrinks, then disappears.** After a drive or node loss, the + auto-repair jobs that rebuild resiliency need free pool capacity to write into. + With no reserve, those jobs stay **suspended** until the failed hardware is + replaced, leaving volumes running with reduced or no redundancy. +- **At true exhaustion, writes fail.** When a **thin** volume's backing pool + capacity is exhausted, new allocations fail (on Azure Local 23H2+ with Arc VMs + this surfaces as an out-of-capacity error from the underlying virtualization + layer). The volume can be taken offline and the affected VMs can stop or enter a + paused state as the platform reacts to the write failure — an unplanned outage, + not a graceful, admin-scheduled action. + +Act while the alert is still an early warning — do the cheapest, most reversible +things first, and escalate only as needed: + +1. **Audit and prune.** Merge or remove stale Hyper-V checkpoints, and find and + remove orphaned or stale `.vhdx` files. On thin volumes the reclaimed space + returns to the pool gradually (about 15 minutes; see + [Path B](#path-b--thin-provisioned-volumes-reclaim-unused-capacity)). +2. **Restrict new provisioning.** Stop creating new virtual disks or volumes on + the pressured pool. +3. **Freeze automated thin-disk or volume expansion** so background growth cannot + consume the remaining headroom. +4. **Prepare to expand the pool.** Add OEM-supported physical disks or a node + ([Option A1](#option-a1--add-capacity-recommended-when-growth-is-expected--low-risk)). + Adding capacity is the durable fix. + +> [!CAUTION] +> **Do not respond to a pool or CSV capacity warning by saving VM state.** Saving a +> VM — `Save-VM`, `Stop-VM -Save`, or the **Save the virtual machine state** +> automatic stop action — writes a saved-state file roughly the size of the VM's +> assigned memory onto its volume (similar to hibernating), consuming the very +> capacity you are short of and potentially pushing a nearly-full CSV or pool over +> the edge. Pausing a VM with `Suspend-VM` writes no file, but it frees no capacity +> and is not a remediation either. +> +> - A VM whose **automatic stop action** is **Save the virtual machine state** +> (historically the default) writes a saved-state file the size of its memory +> onto its volume whenever it is stopped **without a live-migration target** — +> for example during a full-cluster `Stop-Cluster`, or a host OS shutdown of a +> non-HA VM. (A node *drain* is space-safe: it live-migrates VMs, copying memory +> over the network and leaving the VHDX on the CSV.) A cluster-wide stop can +> therefore trigger a wave of save-state writes into an already-constrained CSV. +> For VMs on capacity-constrained volumes, set the automatic stop action to +> **Shut down the guest operating system** instead. For Arc VMs, stop the VM +> **from Azure**, not with host tools. +> - Focus remediation on **pool and physical-disk utilization**, not just CSV or +> volume free space. Extending a thin volume or CSV to create file-system free +> space does **not** add pool capacity and can make pool pressure worse. + ## Step 1 — Determine the provisioning type (required first) Run this on any cluster node before choosing a remediation: @@ -232,36 +313,48 @@ data into fewer slabs and releases the emptied slabs back to the pool. > close to `Size × resiliency`). If footprint matches the data actually written, > there is nothing to reclaim. -**Procedure (requires a VM suspend window on the affected volume):** — [MEDIUM RISK] +**Procedure (requires a brief offline window for VMs on the affected volume):** — [MEDIUM RISK] 1. *(Optional, no downtime)* Merge Hyper-V checkpoints that are no longer needed (`Get-VM | Get-VMSnapshot`, then `Remove-VMSnapshot`). Checkpoint files pin extra slabs and reduce what consolidation can recover. -2. **Suspend the VMs running on the affected volume** so their virtual disk file +2. **Take the VMs on the affected volume offline** so their virtual disk file handles are released (required for consolidation). First find where each VM is - running, because a VM must be suspended on its owner node: + running: ```powershell Get-ClusterGroup | Where-Object GroupType -eq 'VirtualMachine' | Select-Object Name, OwnerNode, State ``` + **Prefer a clean guest shutdown**, which releases the file handles **without** + writing a saved-state file: + ```powershell - Suspend-VM -Name "" # run on (or target) the VM's owner node + Stop-VM -Name "" # graceful guest shutdown; run on/target the owner node ``` - Each suspended VM writes a saved-state file roughly the size of its assigned - memory, so confirm there is enough free space on the volume first. Putting the - cluster resource into redirected access is **not** sufficient — the VMs must - actually be suspended (or stopped). + If a guest will not shut down cleanly (hung, or no integration services), use a + forced turn-off — `Stop-VM -Name "" -TurnOff` — which also releases the + file handles **without** writing a saved-state file. + + > [!CAUTION] + > Do **not** substitute `Save-VM` (or the **Save** automatic stop action) or + > `Suspend-VM` here. **Saving** releases the handles but writes a saved-state + > file the size of the VM's memory onto the very volume you are trying to free. + > **Suspending** only *pauses* the VM — its memory stays in host RAM and its + > virtual disk handles stay **open**, so slab consolidation cannot proceed. + > Putting the cluster resource into redirected access is likewise **not** + > sufficient. The VM's file handles must actually be released, which means a + > shutdown or turn-off. > [!IMPORTANT] - > For **Arc-managed VMs** (Azure Local 23H2+), stop the VM from Azure - > (portal or CLI) rather than using `Suspend-VM` on the host. Suspending an - > Arc VM directly on the host can desynchronize the Arc agent / Arc Resource - > Bridge view of the VM state. Once workloads on the volume are stopped or - > suspended cluster-wide, proceed with consolidation. + > For **Arc-managed VMs** (Azure Local 23H2+), stop the VM from Azure (portal + > or CLI) rather than using host tools such as `Stop-VM` or `Suspend-VM`. + > Driving an Arc VM's power state directly on the host can desynchronize the Arc + > agent / Arc Resource Bridge view of the VM state. Once workloads on the volume + > are stopped cluster-wide, proceed with consolidation. 3. **Consolidate slabs** on the volume. For a cluster shared volume (CSV), address it by path and run on the CSV owner node — `-FileSystemLabel` resolves @@ -290,9 +383,9 @@ data into fewer slabs and releases the emptied slabs back to the pool. step, is what releases the emptied slabs. > [!NOTE] - > VMs only need to stay suspended through the consolidation in Step 3. Once - > Step 3 reports complete, you can resume the VMs (Step 6) and run the - > remaining steps with workloads online, shortening the maintenance window. + > VMs only need to stay offline through the consolidation in Step 3. Once + > Step 3 reports complete, you can bring the VMs back online (Step 6) and run + > the remaining steps with workloads online, shortening the maintenance window. 5. **(Optional) Rebalance the pool allocation:** @@ -309,14 +402,14 @@ data into fewer slabs and releases the emptied slabs back to the pool. reclamation failed; confirm the result with the pool fill query in [Verify](#verify). -6. **Resume the VMs:** +6. **Bring the VMs back online:** ```powershell - Resume-VM -Name "" + Start-VM -Name "" ``` > [!NOTE] -> In some cases, even after a correct consolidation pass with workloads suspended, +> In some cases, even after a correct consolidation pass with workloads offline, > a final batch of slabs may remain committed and the pool does not drop as far as > expected. If the pool stays above the threshold after a clean consolidation > pass, open a Microsoft support case rather than repeating the procedure. @@ -366,5 +459,7 @@ capacity finding is resolved or accepted. - [Optimize-Volume](https://learn.microsoft.com/powershell/module/storage/optimize-volume) - [Optimize-StoragePool](https://learn.microsoft.com/powershell/module/storage/optimize-storagepool) - [Troubleshoot Storage Spaces Direct health and operational states](https://learn.microsoft.com/windows-server/storage/storage-spaces/storage-spaces-states) +- [Azure Local Health Service settings (volume and pool capacity thresholds)](https://learn.microsoft.com/azure/azure-local/manage/health-service-settings) +- [Set-VM (automatic stop action)](https://learn.microsoft.com/powershell/module/hyper-v/set-vm) --- From a1811d44813bd2695fe0c8a9480f6a9045770171 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Thu, 2 Jul 2026 10:13:46 -0400 Subject: [PATCH 2/5] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .../Troubleshoot-Storage-StoragePoolCapacityThreshold.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 80af4f33..eae8e326 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -313,7 +313,7 @@ data into fewer slabs and releases the emptied slabs back to the pool. > close to `Size × resiliency`). If footprint matches the data actually written, > there is nothing to reclaim. -**Procedure (requires a brief offline window for VMs on the affected volume):** — [MEDIUM RISK] +**Procedure (requires an offline window for VMs on the affected volume; the window lasts through slab consolidation, which can take hours on large volumes):** — [MEDIUM RISK] 1. *(Optional, no downtime)* Merge Hyper-V checkpoints that are no longer needed (`Get-VM | Get-VMSnapshot`, then `Remove-VMSnapshot`). Checkpoint files pin From d112bbc3ad0edced8d63be3e4123a9cbc2838600 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Fri, 3 Jul 2026 20:31:31 -0400 Subject: [PATCH 3/5] Fix Optimize-Volume CSV targeting in slab-consolidation step Step 3 used `Optimize-Volume -Path "C:\ClusterStorage\"`, which fails on a CSV with "No MSFT_Volume objects found" because -Path binds to the volume's own path (MSFT_Volume.Path), not the CSV mount/access path. Resolve the CSV with `Get-Volume -FilePath` and pipe to Optimize-Volume instead, with a Format-List confirm step to guard against wrong-node/wrong-volume runs. Also enumerate the legitimate low/zero-recovery causes in the end-of-Path-B note (footprint already matches live data, or slabs pinned by in-use VMs/checkpoints) so operators check preconditions before escalating. Addresses review feedback on #312. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...ot-Storage-StoragePoolCapacityThreshold.md | 38 +++++++++++++------ 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index eae8e326..964c69a5 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -356,19 +356,31 @@ data into fewer slabs and releases the emptied slabs back to the pool. > agent / Arc Resource Bridge view of the VM state. Once workloads on the volume > are stopped cluster-wide, proceed with consolidation. -3. **Consolidate slabs** on the volume. For a cluster shared volume (CSV), - address it by path and run on the CSV owner node — `-FileSystemLabel` resolves - against the local node's volume cache and can miss or mismatch a CSV owned by - another node: +3. **Consolidate slabs** on the volume. Run this on the CSV **owner node**. + Resolve the CSV's `C:\ClusterStorage\` path to its volume object with + `Get-Volume -FilePath`, confirm it is the volume you intend, then pipe it to + `Optimize-Volume`: ```powershell - # Identify the CSV owner node + # Identify the CSV owner node, and run the rest on that node Get-ClusterSharedVolume | Select-Object Name, OwnerNode - # On the owner node, consolidate by path - Optimize-Volume -Path "C:\ClusterStorage\" -SlabConsolidate -Verbose + $csv = "C:\ClusterStorage\" + $vol = Get-Volume -FilePath $csv + $vol | Format-List FileSystemLabel, Size, SizeRemaining, Path # confirm this is the intended CSV + $vol | Optimize-Volume -SlabConsolidate -Verbose ``` + > [!IMPORTANT] + > Resolve the CSV with `Get-Volume -FilePath`. Do **not** pass the CSV mount to + > `Optimize-Volume -Path "C:\ClusterStorage\"`: `-Path` matches a + > volume's own device path, not a CSV mount/access path, so it fails with + > `No MSFT_Volume objects found with property 'Path' equal to + > 'C:\ClusterStorage\'`. Other selectors (`-DriveLetter`, + > `-FileSystemLabel`) are also unreliable for a CSV mount. `Get-Volume + > -FilePath` returns the correct volume object and pipes it straight into + > `Optimize-Volume`. + > [!IMPORTANT] > Do **not** add `-ReTrim`. On thin-provisioned ReFS, `-ReTrim` does nothing > useful — ReFS does not use the NTFS retrim mechanism; it has its own @@ -409,10 +421,14 @@ data into fewer slabs and releases the emptied slabs back to the pool. ``` > [!NOTE] -> In some cases, even after a correct consolidation pass with workloads offline, -> a final batch of slabs may remain committed and the pool does not drop as far as -> expected. If the pool stays above the threshold after a clean consolidation -> pass, open a Microsoft support case rather than repeating the procedure. +> A consolidation pass can legitimately return little or no capacity — most often +> because the volume's footprint already matches the data actually written (there +> is nothing to reclaim; see the note at the start of Path B), or because slabs +> are still pinned by data in use (confirm every VM on the volume is stopped in +> Step 2 and that stale checkpoints were merged in Step 1). If real interior free +> space exists, all workloads were offline, and checkpoints were merged, but the +> pool still does not drop after the unmap wait (Step 4), open a Microsoft support +> case rather than repeating the procedure. ## Choose the right option From 3137ff5ebb77db380c59e4de52fbcf23ec210900 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Thu, 9 Jul 2026 10:49:53 -0400 Subject: [PATCH 4/5] Storage pool capacity TSG: add reader-usability sections Implements the usability panel from the PR #312 approval review (1008covingtonlane): a top-of-doc at-a-glance box (impact / owner / time-to-resolve / downtime), a copy-paste quick-triage block, a terminology glossary with S2D/CSV/ReFS expansion, a plain-language orientation line, and a "Data to Collect Before Opening a Support Case" + "When to escalate" (OEM vs Microsoft) section for CSS. The CSS fault names and event IDs were verified against source (Windows Storage Health Service constants; AEO alert catalog) and a live Azure Local cluster's Storage Spaces provider manifest, and every collection command was run on the cluster to confirm it executes read-only: - Get-HealthFault: Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacityFault (on-box, per the Health Service faults reference) and ...PoolCapacityThresholdExceeded (Azure portal / Insights; on-box correlate EventID 103). - Microsoft-Windows-StorageSpaces-Driver/Operational EventIDs 103/104/310 (read from the provider manifest; noted as build-observed, distinct from the documented 311). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...ot-Storage-StoragePoolCapacityThreshold.md | 199 +++++++++++++++++- 1 file changed, 198 insertions(+), 1 deletion(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 964c69a5..6164bf1d 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -19,6 +19,78 @@ +> **In plain terms:** the storage *pool* — the shared disk capacity behind every +> volume on the cluster — is filling up. Deciding what to do is a capacity task for +> the **cluster / storage administrator**; it is usually not urgent, but if the pool +> is genuinely allowed to fill, virtual machines can pause or go offline. Run the +> **Quick triage** below to find which of two fixes applies. + +## At a glance + + + + + + + + + + + + + + + + + + +
Business impactUsually low — a reserve-capacity and update-readiness early + warning. (The page severity Medium reflects the signal, not day-to-day + impact.) High only if the pool is allowed to fill: thin-volume + writes can then fail and affected VMs can pause or go offline (unplanned outage).
Who owns thisThe customer's cluster / storage administrator (capacity + management). It is not a networking issue, and not an OEM issue + unless you are adding or replacing physical disks.
Typical time to resolveTriage: minutes. Adding disks (A1) or adjusting the alert (A4/A5): low and + online. Converting fixed→thin (A2) or the thin reclaim (Path B): a + maintenance window — slab consolidation can take hours on large + volumes.
Downtime / maintenance windowTriage and the add-capacity / alert options (A1, A4, A5) are + online. A2 (convert, then Path B), + A3 (evacuate + recreate a volume), and Path B + require data movement or a VM-offline window; Path B's window lasts through slab + consolidation.
+ +## Quick triage (start here) + +Run this on any cluster node. It shows how full the pool is and, crucially, whether +the volumes are **Fixed** or **Thin** — which decides the entire remediation path: + +```powershell +# 1) How full is the pool? (allocation vs total size) +Get-StoragePool | Where-Object IsPrimordial -eq $false | + Format-Table FriendlyName, Size, AllocatedSize, + @{N='UsedPct';E={[math]::Round(100*$_.AllocatedSize/$_.Size,1)}} -AutoSize + +# 2) Are the volumes Fixed or Thin? (this decides the fix) +Get-VirtualDisk | Format-Table FriendlyName, ProvisioningType, Size, FootprintOnPool -AutoSize + +# 3) Any active storage health faults? +Get-HealthFault +``` + +Then branch on `ProvisioningType`: + +- **`Fixed`** → the pool footprint is committed by design; the fix is to add + capacity, convert to thin, or adjust the alert — go to + [Path A](#path-a--fixed-provisioned-volumes). +- **`Thin`** → capacity from deleted data can be reclaimed — go to + [Path B](#path-b--thin-provisioned-volumes-reclaim-unused-capacity) (needs a + maintenance window). + +> [!NOTE] +> This is the short form of +> [Step 1](#step-1--determine-the-provisioning-type-required-first), surfaced up top. +> The full guide below explains *why* and covers every option in detail — read on if +> triage alone does not resolve it. + ## Overview This guide explains the Storage Spaces Direct (S2D) **storage pool capacity @@ -39,6 +111,31 @@ reclamation procedure (`Optimize-Volume -SlabConsolidate`, then waiting for the ReFS background unmap to release the freed slabs) **does nothing on a fixed-provisioned volume** and only applies to thin-provisioned volumes. +## Terminology + +Short definitions for the terms used in this guide: + +- **Storage pool** — the cluster-wide set of physical drives that Storage Spaces + Direct (S2D) manages as one unit. Every volume is carved out of the pool. +- **S2D (Storage Spaces Direct)** — the Azure Local software-defined storage layer + that pools each server's local drives into shared, resilient storage. +- **CSV (Cluster Shared Volume)** — a volume mounted under `C:\ClusterStorage\` that + every node can use at once; where VM disks (`.vhdx`) live. +- **ReFS (Resilient File System)** — the file system used for Azure Local volumes. +- **Thin vs Fixed provisioning** — a **thin** volume consumes pool capacity only as + data is written; a **fixed** volume reserves its full size in the pool the moment + it is created (see + [Why fixed-provisioned volumes hit this so easily](#why-fixed-provisioned-volumes-hit-this-so-easily)). +- **Footprint (`FootprintOnPool`)** — how much pool capacity a volume actually + occupies, *including* its resiliency copies (for example, a three-way mirror uses + 3× the written data). +- **Slab** — the 256 MB unit S2D allocates pool capacity in. A slab returns to the + pool only when every block in it is free. +- **Unmap** — the background ReFS operation that returns emptied slabs to the pool + after data is deleted or consolidated. +- **Primordial pool** — the built-in pool of drives not yet added to an S2D pool; + filter it out with `Where-Object IsPrimordial -eq $false`. + ## Symptoms **Observable behaviors:** @@ -126,7 +223,7 @@ Two different capacity signals exist, and they are frequently conflated: The pool has **no automatic 80/90/95 capacity ladder and no capacity-based read-only "block."** At the pool level the platform raises only two advisory, Warning-class health faults — `StoragePool.PoolCapacityThresholdExceeded` (the -configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacity` +configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacityFault` — and neither takes automatic action. A Storage Spaces pool is set **read-only** only on **quorum loss** (too many drives offline — operational state `Incomplete`) or by **administrator policy** (`Policy`), never from a capacity percentage. See @@ -462,6 +559,105 @@ Get-HealthFault For an upgrade, re-run the solution update readiness check and confirm the capacity finding is resolved or accepted. +## Data to Collect Before Opening a Support Case + +Collect the following from the cluster and attach it to the case. It captures which +capacity signal fired, the pool / volume / disk state, and the event-log history of +the threshold crossing and any allocation failures. + +**Health faults.** The pool-capacity signals surface as Storage Spaces health +faults. Collect the on-box faults with `Get-HealthFault`, and — for an Arc-connected +cluster — also check the resource's **Resource Health** / Insights health view in +the Azure portal: + +```powershell +Get-HealthFault +``` + +The fault types to look for (both Warning class): + +| Fault type | Meaning | Where it surfaces | +|---|---|---| +| `Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacityFault` | The pool no longer has the minimum reserve (about two drives' worth) needed to repair resiliency after a drive or node loss. | On-box `Get-HealthFault` (documented). | +| `Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded` | The storage pool is running out of capacity (the configurable thin-provisioning alert, default 70%). | Azure portal **Resource Health** / Insights health view; the on-box correlate is **EventID 103** below. | + +**Event log.** Collect these Storage Spaces events from +`Microsoft-Windows-StorageSpaces-Driver/Operational` on **every node** (the pool +owner logs them, and ownership can move between nodes): + +| EventID | Level | Meaning | +|---|---|---| +| **103** | Error | Pool capacity consumption exceeded the threshold set on the pool (the alert firing). | +| **104** | Information | Pool capacity consumption dropped back below the threshold (the alert clearing). | +| **310** | Error | An allocation for a virtual disk failed for lack of pool capacity; the disk can be taken offline / read-only until capacity is added. | + +> [!NOTE] +> These IDs are read from the Storage Spaces provider's own event manifest (confirm +> on the build with `wevtutil gp Microsoft-Windows-StorageSpaces-Driver /ge /gm:true`); +> they are not enumerated in public documentation and can vary by build. Do not +> confuse **310** with the documented **311** (*"virtual disk ... requires a data +> integrity scan"*), which is unrelated to capacity. + +```powershell +# Storage Spaces pool-capacity events (run on each node) +Get-WinEvent -FilterHashtable @{ + LogName = 'Microsoft-Windows-StorageSpaces-Driver/Operational' + Id = 103, 104, 310 +} -ErrorAction SilentlyContinue | + Select-Object TimeCreated, Id, LevelDisplayName, Message | + Format-Table -AutoSize -Wrap +``` + +**Pool, volume, and disk state, plus the cluster log:** + +```powershell +Get-StoragePool -IsPrimordial $false | + Format-List FriendlyName, Size, AllocatedSize, ThinProvisioningAlertThresholds, OperationalStatus, HealthStatus, ReadOnlyReason +Get-VirtualDisk | Format-List FriendlyName, ProvisioningType, Size, FootprintOnPool, OperationalStatus, HealthStatus +Get-PhysicalDisk | Format-Table FriendlyName, MediaType, Size, HealthStatus, Usage -AutoSize +Get-StorageJob + +# Last 2 hours of cluster log to C:\Temp +Get-ClusterLog -Destination C:\Temp -TimeSpan 120 +``` + +Also capture: the pool name and node count; whether the alert settings were changed +from default (`System.Storage.StoragePool.ThresholdAlert.Enabled`, +`System.Storage.StoragePool.CheckPoolReserveCapacity.Enabled`, and the pool's +`ThinProvisioningAlertThresholds`); and, for deeper analysis, the +[Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md) +storage report. + +## When to escalate + +Most capacity warnings are resolved by the paths above. Escalate when one of these +firm conditions is met — do not simply re-run the procedure. + +**Escalate to the hardware vendor / OEM when:** + +- The durable fix is to add capacity + ([Option A1](#option-a1--add-capacity-recommended-when-growth-is-expected--low-risk)), + but the OEM-supported drives are unavailable or the drive model is no longer + supported. +- Physical disks have failed or retired and pool capacity dropped as a result — the + reserve cannot be restored until the hardware is replaced (a drive replacement is + an OEM action). + +**Escalate to Microsoft support when:** + +- **EventID 310 appears, or a thin volume has gone read-only / offline** — the pool + reached true exhaustion and there is data-path impact. Collect the data above and + open the case now; do not wait for the pool to recover on its own. (If the pool + *operational state* is `Incomplete` / read-only from a drive-quorum loss rather + than capacity, that is a separate, higher-severity problem — escalate immediately.) +- **Path B completed with every precondition met** (confirmed real interior free + space, every VM on the volume stopped, checkpoints merged) and you waited out the + ReFS unmap, but pool `AllocatedSize` still does not drop. +- The reserve-capacity fault (`InsufficientReserveCapacityFault`) **persists after** + you have added capacity or reduced footprint. + +Include the data-collection output above with any Microsoft support case. + ## Related Issues - [How to add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) @@ -476,6 +672,7 @@ capacity finding is resolved or accepted. - [Optimize-StoragePool](https://learn.microsoft.com/powershell/module/storage/optimize-storagepool) - [Troubleshoot Storage Spaces Direct health and operational states](https://learn.microsoft.com/windows-server/storage/storage-spaces/storage-spaces-states) - [Azure Local Health Service settings (volume and pool capacity thresholds)](https://learn.microsoft.com/azure/azure-local/manage/health-service-settings) +- [Azure Local Health Service faults reference (`Get-HealthFault` fault types)](https://learn.microsoft.com/azure/azure-local/manage/health-service-faults) - [Set-VM (automatic stop action)](https://learn.microsoft.com/powershell/module/hyper-v/set-vm) --- From 8378f036bdfa2c831d1b403d495b2866928129b5 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Thu, 9 Jul 2026 14:30:09 -0400 Subject: [PATCH 5/5] Storage pool capacity TSG: revert reserve fault name (no "Fault" suffix) Addresses the CHANGES_REQUESTED review: Get-HealthFault on Azure Local emits Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacity with NO "Fault" suffix. Confirmed against fleet telemetry (HealthMgmt, last 30d): the no-suffix form has 79,140 occurrences and the suffixed form 0. The suffix came from a Windows Server 2016 doc that does not match what the cmdlet emits on Azure Local. Reverts all three occurrences (pool-vs-volume signals paragraph, the Data-to-Collect fault table, When-to-escalate) and drops the now- unsupportable "(documented)" tag on that table row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../Troubleshoot-Storage-StoragePoolCapacityThreshold.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 6164bf1d..e4328d5e 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -223,7 +223,7 @@ Two different capacity signals exist, and they are frequently conflated: The pool has **no automatic 80/90/95 capacity ladder and no capacity-based read-only "block."** At the pool level the platform raises only two advisory, Warning-class health faults — `StoragePool.PoolCapacityThresholdExceeded` (the -configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacityFault` +configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacity` — and neither takes automatic action. A Storage Spaces pool is set **read-only** only on **quorum loss** (too many drives offline — operational state `Incomplete`) or by **administrator policy** (`Policy`), never from a capacity percentage. See @@ -578,7 +578,7 @@ The fault types to look for (both Warning class): | Fault type | Meaning | Where it surfaces | |---|---|---| -| `Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacityFault` | The pool no longer has the minimum reserve (about two drives' worth) needed to repair resiliency after a drive or node loss. | On-box `Get-HealthFault` (documented). | +| `Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacity` | The pool no longer has the minimum reserve (about two drives' worth) needed to repair resiliency after a drive or node loss. | On-box `Get-HealthFault`. | | `Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded` | The storage pool is running out of capacity (the configurable thin-provisioning alert, default 70%). | Azure portal **Resource Health** / Insights health view; the on-box correlate is **EventID 103** below. | **Event log.** Collect these Storage Spaces events from @@ -653,7 +653,7 @@ firm conditions is met — do not simply re-run the procedure. - **Path B completed with every precondition met** (confirmed real interior free space, every VM on the volume stopped, checkpoints merged) and you waited out the ReFS unmap, but pool `AllocatedSize` still does not drop. -- The reserve-capacity fault (`InsufficientReserveCapacityFault`) **persists after** +- The reserve-capacity fault (`InsufficientReserveCapacity`) **persists after** you have added capacity or reduced footprint. Include the data-collection output above with any Microsoft support case.