fix(httpapi): log dependency state on transition, not every probe - #6
Merged
Merged
Conversation
A kubelet polls /readyz every few seconds for the life of a pod, so logging steady state there emitted the same warning thousands of times a day per replica. One service with an unconfigured cache produced 290 identical lines in minutes across three replicas. That is not a louder signal, it is a quieter one: the line that matters, a dependency that just broke, is buried under identical copies of a line that has been true since boot, and the log store bills for every copy. Keyed by dependency and holding the last state logged, so a flap logs both edges while a dependency unconfigured since startup logs once. The recovery line is new and deliberate: without it a log shows only half of an incident, the moment it broke and never the moment it came back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A kubelet polls /readyz every few seconds for the life of a pod, so
logging steady state there emitted the same warning thousands of times a
day per replica. One service with an unconfigured cache produced 290
identical lines in minutes across three replicas.
That is not a louder signal, it is a quieter one: the line that matters,
a dependency that just broke, is buried under identical copies of a line
that has been true since boot, and the log store bills for every copy.
Keyed by dependency and holding the last state logged, so a flap logs
both edges while a dependency unconfigured since startup logs once. The
recovery line is new and deliberate: without it a log shows only half of
an incident, the moment it broke and never the moment it came back.