Skip to content

Fix Tenzir error flood on Kubernetes: suppression checks a string that is never returned - #7

Open
torvalstrom wants to merge 1 commit into
Shuffle:nightlyfrom
torvalstrom:fix-tenzir-k8s-log-flood
Open

Fix Tenzir error flood on Kubernetes: suppression checks a string that is never returned#7
torvalstrom wants to merge 1 commit into
Shuffle:nightlyfrom
torvalstrom:fix-tenzir-k8s-log-flood

Conversation

@torvalstrom

Copy link
Copy Markdown

Problem

On Kubernetes, Orborus logs this on every pipeline health poll — about one line every two seconds, indefinitely:

[ERROR] Tenzir node connection problem: Tenzir not implemented for k8s

On one cluster that is ~43,000 lines per day shipped into central logging.

Cause

deployTenzirNode() returns the error "Tenzir not implemented for k8s" when running under Kubernetes:

if isKubernetes == "true" {
    return errors.New("Tenzir not implemented for k8s")
}

sendPipelineHealthStatus() tries to suppress exactly this case, but tests for a different string — "Kubernetes not implemented for Tenzir node":

if (!strings.Contains(err.Error(), "SHUFFLE_SKIP_PIPELINES") && !strings.Contains(err.Error(), "Kubernetes not implemented for Tenzir node")) && ... {
    log.Printf("[ERROR] Tenzir node connection problem: %s", err)
} else {
    //tenzirDisabled = true
    if debug {
        log.Printf("[WARNING] Disabling pipelines: %s. ...")
    }
}

The two strings never match, so the else branch is unreachable for the Kubernetes case and the error is logged every time. The commented-out tenzirDisabled = true and the debug-level "Disabling pipelines" message show the quiet path is what was intended.

Fix

Point the existing check at the message that is actually produced. Neither string occurs anywhere else in the repository, so this is a one-line change with no other call sites affected.

Notes

  • Not a regression in behaviour, only in logging — Tenzir genuinely is not implemented for Kubernetes, and this does not change that.
  • Operators can already work around it with SHUFFLE_SKIP_PIPELINES=true, which the same condition matches. That is what I am running in the meantime.
  • Verified against nightly; the same mismatch is present on main.

… returned

deployTenzirNode() returns "Tenzir not implemented for k8s" when
isKubernetes is true, but sendPipelineHealthStatus() suppresses the error
log by testing for "Kubernetes not implemented for Tenzir node". Those two
strings never match, so the suppression never fires and every Kubernetes
deployment logs

    [ERROR] Tenzir node connection problem: Tenzir not implemented for k8s

on every poll - roughly one line every two seconds, indefinitely.

That the quiet path is intended is clear from the else branch, which
carries a commented-out tenzirDisabled = true and a debug-level
"Disabling pipelines" message.

Neither string appears anywhere else in the repository, so this points the
existing check at the message that is actually produced. Users can already
work around it by setting SHUFFLE_SKIP_PIPELINES=true, which is matched by
the same condition.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant