Storage pool capacity TSG: capacity-pressure guidance and correct VM-offline handling (follow-up to #300)

[AlBurns-MSFT]

What

Follow-up to #300 addressing @Karl-WE's post-merge change request (relaying Patrick Lownds' field guidance): add capacity-escalation guidance with admin actions, and steer away from "pause VM" under capacity pressure.

Single-file change to TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md.

Changes

• Pool capacity vs volume capacity clarification. The 80% / 90% figures are the default volume health thresholds (System.Storage.Volume.CapacityThreshold.Warning = 80 / .Critical = 90), not pool auto-actions. The pool raises only the configurable 70% thin-provisioning alert (PoolCapacityThresholdExceeded) and the reserve-capacity fault, and it goes read-only only on quorum loss (Incomplete) or administrator policy (Policy) — there is no automatic 80/90/95 pool ladder and no capacity-based read-only "block."
• "What happens if the pool fills." Escalation (repair headroom shrinks; at true exhaustion thin-volume writes fail, the volume can go offline and VMs enter a paused state) plus an ordered admin response: audit & prune -> restrict new provisioning -> freeze automated expansion -> expand the pool.
• CAUTION against saving VM state under capacity pressure. Save (Save-VM / Stop-VM -Save / the Save the virtual machine state automatic stop action) writes a saved-state file the size of the VM's memory onto its volume; a cluster-wide stop with the Save stop action can flood an already-full CSV. Set the automatic stop action to Shut down the guest OS; note a node drain is space-safe (it live-migrates). For Arc VMs, stop from Azure.
• Path B (slab consolidation) reconciled to prefer a clean Stop-VM guest shutdown (releases the VHDX handles, no saved-state file), with Stop-VM -TurnOff as the fallback. Corrects that Suspend-VM only pauses the VM (memory stays in RAM and the disk handles stay open, so consolidation cannot proceed) and that Save is what writes the memory-sized file.
• References: add Health Service settings and Set-VM.

Accuracy note

The requested field guidance framed S2D as automatically flagging pool operational status Warning at 80%, Critical/P2 at 90%, and read-only "block" (with graceful guest pause) at 95%. This was verified against Microsoft Learn (Storage Spaces health and operational states; thin provisioning), the Azure Local Health Service settings doc, and the product health-fault model:

• The 80/90 thresholds are per-volume file-system settings, not pool thresholds.
• The only reasons a Storage Spaces pool becomes read-only are quorum loss and administrator policy — not a capacity percentage.
• Real pool exhaustion manifests as failed writes/allocations (auto-repair jobs go suspended; affected Hyper-V VMs enter paused-critical) — a Hyper-V/volume-layer reaction, not an S2D 95% pool "block."

This PR therefore captures the operational intent (escalation model + admin actions + the save-state hazard) with the accurate mechanism, so field engineers don't wait for an automatic pool block that won't come.

[Copilot]

Pull request overview

This is a documentation-only follow-up to #300 that refines the S2D storage pool capacity TSG (TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md). It corrects a common misconception that the pool auto-escalates at 80/90/95% (those are per-volume Health Service thresholds; the pool only raises the configurable 70% thin-provisioning alert and reserve-capacity fault, and goes read-only only on quorum loss or admin policy). It adds capacity-escalation guidance with an ordered admin response, warns against saving VM state under capacity pressure (a saved-state file the size of VM memory can flood an already-full CSV), and reconciles Path B to prefer a clean Stop-VM guest shutdown over Suspend-VM.

Changes:

• New sections clarifying pool allocation vs. volume fill signals, and "what happens if the pool is allowed to fill" with an ordered remediation ladder and a CAUTION against saving VM state.
• Path B rewritten to use Stop-VM / Stop-VM -TurnOff (shutdown, no saved-state file) and Start-VM, replacing Suspend-VM / Resume-VM, with an accompanying CAUTION explaining why suspend/save don't work for slab consolidation.
• Added references for Health Service settings and Set-VM.

[aldawson33]

Good, necessary changes on the wording to best align w/ the product operations.

[Karl-WE]

Thank you @AlBurns-MSFT for incorporating the feedback this way. Much appreciated.
What do you think about the S2D documentation is a similar PR and backlink to this PR advised to keep the guidance consistent for both AzL S2D and WS S2D?

[Karl-WE]

Shall a PS command be added at this point?

• Identify low free space CSV, then identifying the related VMs and disks placed on this one, then putting this into an array so PS can loop through it with Get-VM | Set-VM?

[Karl-WE]

Since 22H2 and later I am seeing a behaviour that when MSFT defrag runs slab consolidation on this provisioned CSV, file explorer will show an increased logical volume usage. This is usually not visible for CSV as the mounting point is hidden but is visible when using storage spaces thin provisioned volumes with a mount point displayed in file explorer.

Repro:
W10 or W11 22H2 or WS 2022 or later
Add at least two disks to HW or virtual disk in a VM.
Create Storage Pool
Create thin provisioned volume
Place amount of data and eventually remove it.
Run slab-optimization, or defrag GUI tool in Windows, repeat quickly 2 to 4 times
Observe temporary volume usage gain.

[1008covingtonlane]

TSG grader results, storage pool capacity (fixed vs thin)

I ran this through our TSG grader (static structure/safety lint plus a 13-reader persona usability panel) and also lab-tested the reclaim path. Overall it is a strong, technically precise guide: static lint A, usability 3.8/5. One correctness fix, a usability backlog, and one FYI.

1. Correctness fix, verified on a live cluster. In Path B, the documented command Optimize-Volume -Path "C:\ClusterStorage\<volume>" -SlabConsolidate fails on a CSV with:

Optimize-Volume : No MSFT_Volume objects found with property 'Path' equal to 'C:\ClusterStorage\<volume>'.

On a CSV the C:\ClusterStorage\<volume> mount is an access path, not the volume's Path (which is the \\?\Volume{GUID}\ form), so -Path cannot match it. The working form (reproduced 3 of 3 runs):

Get-Volume -FilePath "C:\ClusterStorage\<volume>" | Optimize-Volume -SlabConsolidate -Verbose

2. Usability work items (highest-leverage first).

• Add a top-of-document quick orientation: an at-a-glance box (owner, impact, downtime, typical time-to-resolve) plus a "start here" pointer to the single most common fix, before the full option matrix. (3 reviewers asked for this.)
• Add a short glossary (slab, unmap, footprint, resiliency, primordial, thin vs fixed) and a plain-language line on what the warning means.
• Turn the Fixed vs Thin decision into a short flowchart or decision table near the top.
• Add a one-line "who owns this / scope" note so a mis-assigned reader routes to the storage admin quickly.
• State up front (symptoms or overview) that the thin reclaim path needs a maintenance window.
• Add a single copy-paste triage block (Get-VirtualDisk, the pool-fill query, Get-HealthFault).

3. FYI, not a defect: I could not reproduce the Path B reclaim on a nested lab. Across three fragmented-slab patterns, SlabConsolidate reported the purgable slabs "pinned unmovable" and recovered 0 bytes, even with storage jobs idle and all file handles closed. This is almost certainly a nested-substrate and synthetic-workload fidelity limit, not a problem with your guidance; the real Path B condition is VM/VHDX churn accumulated over time on physical S2D. I would like to redo this on physical hardware with a realistic workload before we rely on the reclaim behavior. If it holds there, one small add would help: note that SlabConsolidate can legitimately report 0 recovered with pinned slabs, and when to escalate.

Nice touches already: the fixed-vs-thin framing and the explicit "SlabConsolidate does nothing on a fixed volume" guard are exactly right, and the Save-VM caution is a good catch.

[1008covingtonlane]

Update: Path B thin-volume reclaim validated live on physical hardware (ran cleanly).

Following up on my earlier note. I ran the full reclaim validation end to end on a physical 3-node S2D cluster (build 2607) and it completed cleanly. Results and timing:

• Created a scratch thin volume, wrote 20 GB, then deleted a scattered 60 of 80 files to leave partially-used slabs (footprint held at 76.5 GB, so ReFS auto-unmap could not reclaim them on its own).
• Get-Volume -FilePath "C:\ClusterStorage\<volume>" | Optimize-Volume -SlabConsolidate did real consolidation work in about 28 seconds.
• The ReFS background unmap then returned the freed slabs: footprint dropped from 76.5 GB to 31.5 GB (about 42 GB reclaimed) roughly 10 minutes after consolidation, landing right at the expected floor. End to end the validation took about 15 minutes.

Real hardware clusters are required to see this remediation take effect. On a nested / VM-based substrate the same procedure had zero effect (every purgable slab came back "pinned unmovable", 0 bytes recovered, footprint flat for 20+ minutes). Only on a physical S2D cluster does SlabConsolidate plus the background unmap actually reclaim, so Path B should be validated (and is only observable) on real hardware.

The one command fix still worth making is the CSV form from my first comment: use Get-Volume -FilePath "C:\ClusterStorage\<volume>" | Optimize-Volume -SlabConsolidate rather than Optimize-Volume -Path "C:\ClusterStorage\..." (the -Path form fails on any CSV).

[AlBurns-MSFT]

Thanks all for the careful review. Here is what I changed, how I validated it, and answers to the open questions.

Correctness fix: Optimize-Volume on a CSV (@1008covingtonlane)

You are right, and thanks for catching it. Optimize-Volume -Path "C:\ClusterStorage\<volume>" fails on a CSV because -Path binds to the volume's own path (the CIM MSFT_Volume.Path), whereas C:\ClusterStorage\<volume> is a mount/access path, so nothing matches and it errors with No MSFT_Volume objects found with property 'Path' equal to ....

I reproduced it independently on a live Azure Local cluster (build 10.2607), CSV mounted at C:\ClusterStorage\UserStorage_1, whose MSFT_Volume.Path is the \\?\Volume{GUID}\ form:

• Optimize-Volume -Path "C:\ClusterStorage\UserStorage_1" -SlabConsolidate gives the exact No MSFT_Volume objects found ... error.
• Get-Volume -FilePath "C:\ClusterStorage\UserStorage_1" resolves the volume.
• Get-Volume -FilePath "..." | Optimize-Volume -SlabConsolidate binds and runs.

Step 3 now resolves the CSV with Get-Volume -FilePath and pipes it to Optimize-Volume. I also added a confirm step (Format-List the resolved volume before acting): the pipeline form binds on any node, so echoing the resolved volume guards against running on a non-owner node or fat-fingering the path onto the OS disk. The note explains why -Path (and -FileSystemLabel) are wrong for a CSV mount.

Usability backlog (@1008covingtonlane)

Good list. The doc already gates on provisioning type up front (Step 1, "required first") and has the "Choose the right option" table, so the applicability guardrail is there. I would rather keep this PR to the correctness fix and take the orientation box / glossary / triage block as a focused follow-up so they get their own review. I will track that separately.

On "0 bytes recovered / pinned slabs": I folded that into the end-of-Path-B note, so an operator who gets little back checks the legitimate causes (footprint already matches live data, or slabs still pinned by running VMs / unmerged checkpoints) before escalating, instead of repeating the pass. The "only reproducible on physical S2D, not nested" observation matches expectations, but that is a lab-substrate caveat rather than customer guidance, so I left it out of the doc.

Set-VM helper to bulk-set the stop action (@Karl-WE)

I looked hard at scripting this and decided not to put a loop in this PR:

• The property that actually lands a memory-sized file on the volume is the VM's saved-state / smart-paging path, not its VHDX path, so "VMs whose disks are on this CSV" is the wrong selection key.
• Get-VM only sees the local node, so a loop would silently miss clustered VMs owned by other nodes (Step 2 already uses the cluster-aware Get-ClusterGroup ... OwnerNode pattern for that reason).
• A blanket Get-VM | Set-VM would also touch Arc-managed VMs, which the same procedure tells you to drive from Azure.
• AutomaticStopAction only fires at host shutdown or an ungraceful stop, so it is preventive hardening rather than a step inside the reclaim window.

The intent (set the stop action to "Shut down the guest OS" for non-Arc VMs) is already stated in the CAUTION, so I left it as guidance rather than a script here.

Temporary volume-usage rise during consolidation (@Karl-WE)

I think the increase you see in File Explorer is a layer mismatch rather than real growth. Optimize-Volume -SlabConsolidate works at the Storage Spaces pool layer (which slabs back the virtual disk); it adds or removes no files, so the filesystem "used" figure Explorer shows should not change from consolidation. The number that actually moves is pool allocation, and it settles only after the background unmap (~15 min). The doc already points at the pool-fill query in Verify (and the capacity CAUTION says to judge by pool / physical-disk utilization, not volume free space), so I did not add a "usage may rise" note, since pointing that at the Explorer figure would be misleading. If you have a repro where pool AllocatedSize itself rises before it drops, I would like to see it and will document that case specifically.

Parallel Windows Server S2D doc (@Karl-WE)

Agreed the mechanics are the same platform behavior and reciprocal backlinks would keep the two in sync. That is a separate doc and repo, so I would like to keep it out of this PR and open it as a follow-up that can point at this one once it merges.

[1008covingtonlane]

Re-ran the grader on the updated head: correctness (now clean) plus the 13-reader usability panel.

Correctness, re-validated. The Step 3 fix and the VM-offline rewrite both hold up, no correctness issues left on my side. Get-Volume -FilePath "C:\ClusterStorage\<volume>" | Optimize-Volume -SlabConsolidate is the form I validated live, and the Format-List confirm before acting is a good guard. The Suspend-VM to Stop-VM change is a real correctness improvement: pausing keeps the VHDX handles open (so consolidation cannot reclaim), and the old "suspend writes a saved-state file" line was conflating pause with save. Stop-VM (or -TurnOff for a hung guest) releases the handles without writing a memory-sized state file, which matches the documented Hyper-V behavior.

Usability panel: 3.7/5. The hands-on correctness reads are already at 5 (the fixed-vs-thin gate, the corrected command, the CAUTION and Arc-VM guidance). The remaining gap is orientation for skimmers and beginners, which lines up with the usability follow-up you said you would take separately. For when you get to it, these five reader-driven items are what would carry the panel to 5/5:

1. A top-of-doc at-a-glance box: business impact, who owns it (customer IT / storage, not network or OEM), rough duration, and downtime/maintenance-window risk. (4 readers)
2. A copy-paste quick-triage block up top: run the Step 1 provisioning-type check, then branch to Path A or Path B. (3 readers)
3. A short glossary (slab, unmap, footprint, thin vs fixed, primordial) and expand S2D/CSV/ReFS on first use. (beginner reader)
4. For CSS: the exact health-fault names / EventIDs to collect, and a firm "escalate to PG/OEM when..." rule beyond "open a support case."
5. A one-line plain-language orientation note for a mixed-skill global audience.

Items 1 and 2 alone cover nine of the thirteen readers. This is all the usability follow-up, not a blocker here; correctness is clean.

[AlBurns-MSFT]

Thanks again @1008covingtonlane — implemented all five usability items in 3137ff5:

1. At-a-glance box — business impact, owner (cluster/storage admin, not networking/OEM), time-to-resolve, downtime/maintenance-window risk.
2. Quick-triage block — copy-paste "start here" running the pool-fill + provisioning-type checks, branching to Path A/B (cross-linked to Step 1).
3. Terminology — glossary (slab, unmap, footprint, thin vs fixed, primordial) + S2D/CSV/ReFS expanded.
4. CSS data collection + escalation — a "Data to Collect Before Opening a Support Case" section with the exact fault types and event IDs, plus a firm "When to escalate" split into hardware/OEM vs Microsoft triggers.
5. Plain-language orientation line up top.

For item 4 I verified the identifiers rather than quoting from memory: ...StoragePool.InsufficientReserveCapacityFault (on-box Get-HealthFault) and ...PoolCapacityThresholdExceeded (Azure portal / Insights; on-box correlate EventID 103); StorageSpaces-Driver/Operational 103/104/310 read from the provider manifest on a live cluster (build-observed, and distinct from the documented 311). Every collection command was run read-only on a cluster to confirm it executes.

[1008covingtonlane]

Thanks for the follow-up. The usability sections read well, and I re-checked the new content in this commit before signing off.

What I re-verified in the new material:

• Fault type strings: Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded is correct and matches what the platform emits.
• The new PowerShell is all read-only, and every property it reads is real: ProvisioningType and FootprintOnPool on Get-VirtualDisk; ReadOnlyReason, IsPrimordial, and ThinProvisioningAlertThresholds on Get-StoragePool; Usage on Get-PhysicalDisk. Both the -IsPrimordial $false parameter and the Where-Object IsPrimordial -eq $false filter are valid, so the triage block and the collection block agree.
• The five new in-doc links (Path A, Path B, Step 1, the fixed-provisioning explainer, Option A1) all resolve to real headings.
• Terminology checks out: the slab and footprint definitions are right, a three-way mirror really is 3x written data, and the top-of-page Medium severity matches the "At a glance" box.

One thing to fix:

• The reserve fault type picked up a wrong suffix. This commit changed StoragePool.InsufficientReserveCapacity to StoragePool.InsufficientReserveCapacityFault. On an Azure Local cluster Get-HealthFault returns this fault as Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacity, with no Fault suffix, so the suffixed form is not a real fault name and an operator grepping for InsufficientReserveCapacityFault would match nothing. It is worth flagging because the "Data to Collect" table row is labeled "documented" and cites Get-HealthFault as its source, yet the string in that row does not match what Get-HealthFault emits. The suffixed form appears in three spots: the "two threshold controls" paragraph, the "Data to Collect" fault table, and the "When to escalate" section. Reverting all three to InsufficientReserveCapacity lines them up with what the cmdlet returns. One word each, low risk.

Minor, no action needed: I could not confirm EventIDs 103, 104, and 310 from public docs, but the note already tells the reader to verify them with wevtutil and warns they can vary by build, so that reads fine as written.

[AlBurns-MSFT]

Good catch — fixed in 8378f03. Reverted all three to InsufficientReserveCapacity (no Fault suffix).

I double-checked against fleet telemetry to be sure: in HealthMgmt over the last 30 days the no-suffix form appears 79,140 times and InsufficientReserveCapacityFault 0 times, so the suffixed form matched nothing Get-HealthFault emits. The suffix traced back to a Windows Server 2016 doc that disagrees with the cmdlet, so I also dropped the "(documented)" tag on that row.

[1008covingtonlane]

The reserve fault name is fixed in all three spots. It now reads InsufficientReserveCapacity without the Fault suffix, which matches what Get-HealthFault emits, and dropping the "documented" qualifier on that table row was a good call. Approving, thanks for the quick turnaround.