ci: TEM-13 clean up

[Chmokachka]

Summary

Build-hygiene and cleanup work for TEM-13. Removes committed OS cruft and dead
config, and adds three automated cleanup mechanisms so ephemeral artifacts
(RC/dev image tags, orphaned signatures, leaked test pods) don't pile up and
burn storage/credits.

Changes

Build hygiene

• Removed the committed official-templates/.DS_Store and added .gitignore
  rules for OS/editor artifacts (.DS_Store, Thumbs.db, desktop.ini).
• Pruned the docker ecosystem entry from .github/dependabot.yml: it pointed
  at / and never matched anything useful (our base images live in
  docker-bake.hcl files, which Dependabot doesn't parse), so it only added
  noise.

Docker Hub tag cleanup

• New .github/workflows/cleanup.yml - sweeps aged-out ephemeral tags:
  deletes -rc.* / -dev image tags older than a retention window and their
  now-orphaned Cosign signature tags (sha256-*.sig / *.att). Runs on a
  schedule and via workflow_dispatch, and is also invoked after a successful
  release (see release.yml). Supports dry_run to preview.
• New .github/workflows/cleanup-closed-pr.yml - promptly removes a PR's
  -rc.<PR#> images (and their signatures) when the PR is closed/merged, so
  merged-PR release candidates don't linger until the age-based sweep.
• release.yml - runs the cleanup workflow after a real release
  (should-release), so retention is enforced right after new tags land.

RunPod pod reaper

• New .github/workflows/reap-pods.yml - hourly cron (and manual
  workflow_dispatch) that deletes leaked smoketest-* pods older than a
  threshold. A cancel-in-progress PR cancel can SIGKILL the smoke-test runner
  before cleanup_all() finishes, leaving pods running for days; this reaper is
  the reliable, job-lifecycle-independent safety net. It only ever touches
  smoketest-* pods and reads each pod's age from its name, so it never deletes
  a human's pod.
• tests/runpod_smoke/config.py / pod.py - pass --terminate-after
  (RFC3339 datetime) as a best-effort server-side backstop, recomputed per pod.
  Documented that it's unreliable (runpodctl drops it on the CPU REST path and
  doesn't honor it promptly for GPU pods), so the reaper cron - not this flag -
  is the real safety net.
• tests/README.md - documents the reaper and the --terminate-after
  caveat.

[kodxana]

Had another look at the latest commits. The 404 issue is fixed, and I checked the test runs: both Docker Hub cleanup workflows completed successfully in dry-run, and the reaper successfully deleted the stale test pod.

I still see three things to fix before merging:

• In release.yml, cleanup checks needs.release.outputs.should-release, but the release job has no outputs. That value comes from version, so the cleanup job will always skip. It should depend on both version and release and check needs.version.outputs.should-release.
• In reap-pods.yml, the pod-list and jq errors are swallowed with || true. A failed list or parse would look like zero pods, and a failed deletion is still counted as reaped.
• In cleanup.yml, an invalid or missing last_updated becomes 0, which makes the tag look ancient and eligible for deletion. Please keep the tag and treat its digest as live if the timestamp cannot be parsed.

Could we also validate pr_number, retention_days, and max_age_minutes before using them?

One small comment: pytorch-cluster doesn’t need a future Docker Hub repository—#138 publishes those tags under runpod/pytorch, so that comment is misleading.

[kodxana]

Thanks, this addresses the earlier points and the verification runs look good.

I found one last edge case in the new validation: the current case only checks that the value contains digits, so it still accepts 0 and values with leading zeroes.

retention_days=0 with deletion enabled would select essentially every ephemeral tag, while max_age_minutes=0 would select every current smoke-test pod. Values like 08 also fail in Bash arithmetic because they’re interpreted as octal.

Could we validate all three inputs with ^[1-9][0-9]*$ instead? After that I’m good with this.

[Madiator2011Work]

LGTM