[watchdog] stuck-containercreating: fuzepicker/fuzepicker-picker-bcd66cdd7-wz4rn:picker

[izzywdev]

@fuze the deployment-freeze watchdog detected stuck-containercreating.

fuzepicker/fuzepicker-picker-bcd66cdd7-wz4rn container picker has been PodInitializing for 9.2d (threshold 30m) — likely a stuck volume attach.

Diagnostics (verbatim from the cluster)

{
  "container": "picker",
  "namespace": "fuzepicker",
  "node": "mendys-worker-1",
  "pending_minutes": 13188.7,
  "pod": "fuzepicker-picker-bcd66cdd7-wz4rn",
  "threshold_minutes": 30.0,
  "waiting_message": null,
  "waiting_reason": "PodInitializing"
}

Automated action taken

• None. This watchdog is read-only against the cluster; prod is GitOps under Argo selfHeal, so any fix must land in git.

Watchdog run: https://github.com/izzywdev/FuzeInfra/actions/runs/33512624463

---

This watchdog exists because of the 2026-08-30..09-01 fleet-deployment freeze: a full Loki PVC crash-looped Loki 694 times, the permanently-unhealthy StatefulSet wedged one Argo sync operation in phase=Running for 37h, and every prod change queued silently behind it for 2d11h. Every component reported "running"; nothing alerted.

Thresholds: governance/watchdog-thresholds.json. Detection: scripts-tools/deployment_watchdog.py. This issue is deduplicated on the watchdog-key marker above — it will not be re-filed while it stays open, so close it once the condition is actually resolved.

[izzywdev]

⚠️ Fuze responder did not run to completion

The responder errored before finishing your request, so no changes were made and no branch was created.

• Reason: Gateway could not serve claude-opus-5 (HTTP 400): {"type":"error","error":{"type":"invalid_request_error","message":"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits."},"request_id":"req_011Cecohmwxqtk7mwJXuRe4f"}. Received Model Group=claude-opus-5
  Available Model Group Fallbacks=['gpt-4.1', 'gemini-flash-latest']
  Error doing the fallback: litellm.ServiceUnavailableError: GeminiException - {
  "error": {
  "code": 503,
  "message": "This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.",
  "status": "UNAVAILABLE"
  }
  }
  No fallback model group found for original model_group=gemini-flash-latest. Fallbacks=[{'claude-opus-4-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-opus-4-6': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-opus-4-8': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-opus-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-sonnet-4-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-sonnet-4-6': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-sonnet-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-fable-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-haiku-4-5': ['gpt-4.1-mini', 'gemini-flash-lite-latest']}]. Received Model Group=gemini-flash-latest
  Available Model Group Fallbacks=None
  Error doing the fallback: litellm.ServiceUnavailableError: GeminiException - {
  "error": {
  "code": 503,
  "message": "This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.",
  "status": "UNAVAILABLE"
  }
  }
  No fallback model group found for original model_group=gemini-flash-latest. Fallbacks=[{'claude-opus-4-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-opus-4-6': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-opus-4-8': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-opus-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-sonnet-4-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-sonnet-4-6': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-sonnet-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-fable-5': ['gpt-4.1', 'gemini-flash-latest']}, {'claude-haiku-4-5': ['gpt-4.1-mini', 'gemini-flash-lite-latest']}] LiteLLM Retried: 2 times, LiteLLM Max Retries: 2 LiteLLM Retried: 2 times, LiteLLM Max Retries: 2
• Run log: https://github.com/izzywdev/FuzeInfra/actions/runs/33513230143

This notice is automated and will not retry on its own. If the LiteLLM gateway is the problem, check the LITELLM_FUZE_KEY secret and gateway health (helm/litellm/), then re-trigger by posting a new comment mentioning @fuze.