Skip to content

Migrate off self-hosted Postgres/Redis/RabbitMQ onto FuzeInfra-provisioned engines #255

Description

@izzywdev

@claude

FuzeAgent runs its own Postgres, Redis and RabbitMQ inside the fuzeagent namespace. That is architectural drift from the platform standard: FuzeInfra provides the shared engines, and consuming repos get a per-service database + role provisioned from FuzeInfra rather than running an engine of their own. This issue asks you to migrate off the self-hosted engines.

What was verified (live prod, 2026-09-01/02)

Read-only via FuzeInfra's cluster-query workflow — runs 33564509285 (PVCs), 33564639316 (pods), 33564766594 (PVs + node affinity).

PVC StorageClass Node pin Reclaim Status
fuzeagent/postgres-data (5Gi) longhorn none Retain Bound
fuzeagent/redis-data (1Gi) longhorn none Retain Bound
fuzeagent/rabbitmq-data (2Gi) longhorn none Retain Bound

Running in-namespace: postgres-69c75d55d-l8jd2, redis-56c76c5db-px4ms, rabbitmq-5c78d68cc6-hmhhh (Pending).

Correction to an earlier report: these three PVCs were previously described as sitting on local-path. They are not — they are on longhorn with reclaimPolicy: Retain. The storage-class emergency is over; do not treat this as an urgent storage migration. What remains is the architectural drift, and the earlier report that the FuzeAgent data directories were found absent is consistent with these volumes having been re-created empty after a node loss — i.e. assume the previous contents are gone and that these are not carrying irreplaceable state. Verify that assumption before you delete anything.

Remedy — consume FuzeInfra's shared engines

Postgres, Redis-as-datastore and RabbitMQ should not be self-hosted here.

  • Mechanism (already in production use, not dormant): helm/fuzeinfra/templates/service-db-provisioning.yaml, driven by .Values.serviceDatabases. Contrary to an earlier report that every entry is enabled: false — that is only true of the base values.yaml. In helm/fuzeinfra/values-contabo.yaml (prod) the entries for fuzesales, fuzecontact, fuzeservice, mendys, fuzequality, fuzehub and fuzesocial are enabled: true today. Use those as working precedents.
  • Your side of the contract: seal the password Secret for the fuzeinfra namespace and PR the serviceDatabases entry to FuzeInfra. Precedent: mendys-db-credentials, annotated fuzeinfra.dev/consumer: izzywdev/MendysRobotics.
  • Docs: docs/consuming-repos/MONGODB_PROVISIONING.md and docs/consuming-repos/CHROMADB_PROVISIONING.md in FuzeInfra. A Postgres equivalent is being written in parallel right now and may land shortly after this issue — check for it before hand-rolling the procedure.
  • Connect over in-cluster service DNS (fuzeinfra-postgres.fuzeinfra.svc, fuzeinfra-redis...) via DATABASE_URL / REDIS_URL from a SealedSecret. Never hardcode.
  • Redis: if it is a cache only, it may legitimately stay local — but then it should not have a PVC at all. Decide and state which it is.
  • RabbitMQ: FuzeInfra provides shared RabbitMQ. Consume it rather than running your own.

Acceptance criteria

  1. Each of Postgres / Redis / RabbitMQ is explicitly classified: migrate to shared FuzeInfra engine, or justified exception written down in the repo.
  2. For each one migrated: a serviceDatabases (or equivalent) entry PR'd to FuzeInfra, the password sealed into the fuzeinfra namespace, and the app wired via service DNS + SealedSecret.
  3. The self-hosted Deployment + its PVC are removed from deploy/helm/fuzeagent/templates/postgres.yaml (and the redis/rabbitmq equivalents) only after the app is verified running against the shared engine.
  4. Any data that must survive is dumped and restored into the shared engine before the old Deployment is removed — or it is explicitly recorded that the volume was empty.
  5. Prod is GitOps. Every change lands via a merged PR that Argo syncs. Never kubectl patch/edit a live prod resource — Argo selfHeal reverts out-of-band changes within seconds.
  6. Verify the result against the live cluster with FuzeInfra's cluster-query workflow (repository_dispatch type cluster-query, using the FUZEINFRA_DISPATCH_TOKEN you already hold) — see docs/consuming-repos/CLUSTER_QUERY.md. Do not report done from chart diffs alone.

STATE:

  • Why: FuzeAgent self-hosts DB/queue engines instead of consuming FuzeInfra's shared ones. Same class of exposure as the 2026-07-24 incident where node-local volumes were wiped and Neo4j was lost.
  • Done: Live prod census verified (3 PVCs, all longhorn/Retain, no node pin). Owning repo confirmed izzywdev/FuzeAgent via argocd/projects/fuzeagent.yaml sourceRepos. Provisioning mechanism confirmed live in prod values.
  • Remains: All of the acceptance criteria above. Nothing has been changed in this repo or in the cluster.
  • Decisions made: Storage class is NOT the issue here (already Longhorn/Retain) — the shared-engine migration is. Redis-as-cache is an acceptable exception if declared.
  • Open: Whether the three volumes carry any state worth preserving. Confirm before deleting.
  • Filed by: platform-governance (FuzeInfra), as cross-repo @claude delegation. FuzeInfra was not edited; nothing in this repo was edited.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions