Skip to content

fix(httpapi): log dependency state on transition, not every probe - #6

Merged
crypto-a merged 1 commit into
mainfrom
5/probe-log-transitions
Aug 20, 2026
Merged

crypto-a merged 1 commit into
mainfrom
5/probe-log-transitions

Conversation

@crypto-a

Copy link
Copy Markdown
Contributor

A kubelet polls /readyz every few seconds for the life of a pod, so
logging steady state there emitted the same warning thousands of times a
day per replica. One service with an unconfigured cache produced 290
identical lines in minutes across three replicas.

That is not a louder signal, it is a quieter one: the line that matters,
a dependency that just broke, is buried under identical copies of a line
that has been true since boot, and the log store bills for every copy.

Keyed by dependency and holding the last state logged, so a flap logs
both edges while a dependency unconfigured since startup logs once. The
recovery line is new and deliberate: without it a log shows only half of
an incident, the moment it broke and never the moment it came back.

A kubelet polls /readyz every few seconds for the life of a pod, so
logging steady state there emitted the same warning thousands of times a
day per replica. One service with an unconfigured cache produced 290
identical lines in minutes across three replicas.

That is not a louder signal, it is a quieter one: the line that matters,
a dependency that just broke, is buried under identical copies of a line
that has been true since boot, and the log store bills for every copy.

Keyed by dependency and holding the last state logged, so a flap logs
both edges while a dependency unconfigured since startup logs once. The
recovery line is new and deliberate: without it a log shows only half of
an incident, the moment it broke and never the moment it came back.
@crypto-a
crypto-a merged commit 56c3453 into main Aug 20, 2026
5 checks passed
@crypto-a
crypto-a deleted the 5/probe-log-transitions branch August 20, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant