Skip to content

Qualify retained static iSCSI recovery on TrueNAS 25.10.7 - #1232

Draft
MrStarktastic wants to merge 10 commits into
mainfrom
codex/retained-iscsi-lab
Draft

MrStarktastic wants to merge 10 commits into
mainfrom
codex/retained-iscsi-lab

Conversation

@MrStarktastic

@MrStarktastic MrStarktastic commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

SQLite application data on NFS needs a storage path that survives complete cluster replacement without accidentally creating empty databases. This PR records the isolated TrueNAS 25.10.7 qualification of retained static iSCSI, using the stock democratic-csi node-manual plugin and an external NAS lifecycle. It changes test fixtures and documentation; it deploys no production storage or application migration.

The latest report is docs/superpowers/reports/2026-09-21-static-iscsi-recovery.md. Earlier REST-controller results and driver evaluations remain historical evidence; the selected candidate has no NAS API credential, dynamic provisioner, CSI snapshot controller or CSI expansion controller in Kubernetes.

Verified in the isolated lab:

  • Actual blank NodeStage refused mounting without changing the whole-device hash. External recovery gates also rejected wrong valid, corrupt and partitioned filesystems without writes.
  • Actual Argo and Sealed Secrets recovered existing retained bindings after two complete replacements of all three K3s VM disks. CAs and Kubernetes UIDs changed; NAS GUIDs, filesystem UUIDs and all acknowledged transactions survived.
  • RWOP contention, clean movement, storage-reachable control partition, stale identity, denied/timed-out fencing, and confirmed power-off before replacement.
  • Authenticated snapshot clone, discarded-reply reconciliation, explicit interrupted 2→3 GiB expansion, actual Argo capacity rebinding, prune/application-deletion retention, and independent backup restore while the lab NAS was off.
  • Final integrity and exact values: 31 acknowledged transactions per service.

All four lab VMs and disks, private networks, staged media, transient services, tunnel and private runtime were removed. Runner 300 was restored. Production remained 125/125 ready pods, 77/77 Synced/Healthy applications and 68/68 unchanged PV bindings.

Validation: static identity/capacity/disk tests, existing guard/lifecycle tests, repository pre-commit checks, and actual Kubernetes admission/reconciliation. See the report for exact counts and independent review findings. This is a run-specific correctness rehearsal, not a workload benchmark or production-ready recovery operator. Production-scoped fencing permissions, placement-specific writer authorization, onboarding interruption handling and a Jellyfin cold restore still need qualification before a pilot.

Production preparation now includes docs/superpowers/specs/2026-09-21-jellyfin-iscsi-pilot-design.md and sanitized read-only preflight evidence. The proposal covers a 64 GiB retained target, scoped Proxmox fencing, movable disposable cache, cold backup/restore, controlled cutover and rollback after new writes. The audit found sync=disabled on existing apps/pv; correcting it is a separate reviewed production change. No NAS property, ACL or storage binding was changed. Terraform #223 has been merged and both workers are verified at 28 GiB. The user approved the pilot design and confirmed the encrypted laptop backup destination. The implementation plan at docs/superpowers/plans/2026-09-21-jellyfin-iscsi-pilot.md separates four PR groups, shared maintenance coordination, failure-path tests and live qualification gates; it is ready for execution review. No production implementation or migration has run.

@MrStarktastic MrStarktastic changed the title Document retained iSCSI recovery rehearsal and storage audit Qualify retained static iSCSI recovery on TrueNAS 25.10.7 Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant