Skip to content

fix(infra): pods list live by default, and a terminated pod is "Ended" not "Down" - #751

Merged
Makisuo merged 1 commit into
mainfrom
fix/pods-live-lifecycle
Sep 2, 2026
Merged

fix(infra): pods list live by default, and a terminated pod is "Ended" not "Down"#751
Makisuo merged 1 commit into
mainfrom
fix/pods-live-lifecycle

Conversation

@Makisuo

@Makisuo Makisuo commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

The Kubernetes pods view painted most of an autoscaled fleet red — hundreds of rows marked Down on a fleet where nothing was actually failing. Two causes, both fixed here.

Why "Down" fired on the normal case

deriveHostStatus returned "down" — styled --severity-error — for any resource whose newest datapoint was older than 5 minutes. But lastSeen is max(TimeUnix) over a whole query window (12h by default), so "down" really meant "this pod's metric series ended". On an autoscaled fleet that is the normal end of a pod's life: HPA scale-in, a rolling deploy, a replaced Fargate task, a reclaimed spot node.

Continuity alone cannot prove a resource is down, so the state is now ended and reads neutral (text-muted-foreground, bg-muted-foreground/40 dot) instead of error red. A genuine down signal needs an expectationk8s.pod.phase, or a workload's available-vs-desired replicas — and belongs beside these states, not instead of them.

Why the list was full of dead pods

The list returned the union of everything that reported anywhere in the window, with no notion of who was still running. Dead pods inflated the "N pods in scope" denominator and outranked live ones in the default saturation sort.

PodLifecycle ("live" | "ended" | "all") now defaults to live, cutting at the same 300s-before-endTime threshold the badge uses — so the predicate that selected a row and the badge on it cannot disagree. It rides in listPodsFilters rather than with the scope, so listPodsCount narrows identically and the page can't read "50 of 656".

What changed

  • HostStatus is "active" | "idle" | "ended"; down is gone. Shared policy, so nodes, hosts and workloads get the same treatment — an ASG or Karpenter scaling a node out is the same non-event.
  • listPodsQuery applies the lifecycle outside the grouping, alongside the scope predicate.
  • listPodsSummaryQuery returns livePods/endedPods as absolute counts, with the three saturation buckets counted within the requested lifecycle so the band stays a valid denominator for the list beside it.
  • The old stale scope was the same idea under a worse name — it is now the ended lifecycle, and the band's fourth cell. Clicking it swaps which fleet is on screen rather than narrowing the live one.
  • The band header reads "N live pods in scope · share of the live fleet by peak utilization".
  • The unfiltered empty state no longer says "install the Helm chart" when the fleet has simply all ended (a finished job, an environment scaled to zero) — it points at the Ended scope with the count.

Reviewer notes

  • Breaking URLs: shared links carrying ?scope=stale (pods) or ?status=down (nodes) will now fail validateSearch. I took the clean rename over carrying both literals — flag it if you'd rather keep the old values decoding.
  • Not verified against real data. The local dev org has no Kubernetes metrics, so this is covered by SQL-shape tests but the visual pass on the screen is unverified. Worth a look on an org with a live fleet.
  • Follow-up left out: facet counts in the filter sidebar are still window-wide rather than live-scoped, so the sidebar can offer a namespace that yields zero rows. Bounded the diff deliberately.
  • The natural next step is the expectation signal — k8s.pod.phase, k8s.pod.status_reason, k8s.container.restarts, and k8s.deployment.desired vs .available are all collected by default in deploy/k8s-infra but unqueried. That is what would let "Down" mean something again, at the workload level where replicas missing (rather than churning) is the actual incident.

bun typecheck, bun run lint, and the query-engine + web infra test suites pass.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…" not "Down"

The pods view painted most of an autoscaled fleet red. Two causes, both here.

`deriveHostStatus` returned "down" — styled `--severity-error` — for any
resource whose newest datapoint was older than 5 min. But `lastSeen` is
`max(TimeUnix)` over a whole window (12h by default), so "down" really meant
"this pod's series ended", which for a pod on an autoscaled fleet is the normal
end of its life: HPA scale-in, a rollout, a replaced Fargate task, a reclaimed
spot node. Continuity alone cannot prove a resource is down, so the state is now
`ended` and reads neutral. A real down signal needs an expectation — `k8s.pod.phase`,
or a workload's available-vs-desired replicas — and belongs beside these, not
instead of them. The status policy is shared, so nodes, hosts and workloads get
the same treatment.

The list also had no notion of who was still running: it returned the union of
everything that reported in the window, so dead pods inflated the denominator
and outranked live ones in the saturation sort. `PodLifecycle` ("live" | "ended"
| "all") now defaults to live, cutting at the same 300s-before-endTime threshold
the badge uses so the predicate and the badge cannot disagree. It rides with the
filters rather than the scope, so the count query narrows identically. The old
`stale` scope was the same idea under a worse name and is now the `ended`
lifecycle, offered as the band's fourth cell — clicking it swaps which fleet is
on screen rather than narrowing the live one.

`listPodsSummaryQuery` returns `livePods`/`endedPods` as absolute counts, with
the saturation buckets counted within the requested lifecycle so the band stays
a valid denominator for the list beside it.

The unfiltered empty state no longer says "install the Helm chart" when the
fleet has simply all ended — a finished job, an environment scaled to zero — and
points at the Ended scope instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Makisuo
Makisuo merged commit c212a59 into main Sep 2, 2026
34 checks passed
@Makisuo
Makisuo deleted the fix/pods-live-lifecycle branch September 2, 2026 23:54
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

🍁 Maple PR preview

Warning

Preview cleanup could not be confirmed. The Alchemy teardown outcome was skipped.

Final commit 0e69bd6 · View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant