fix(alerts): ContainerHighMemory measured against a limit nobody set - #113
Merged
Conversation
The rule divided container_memory_working_set_bytes by container_spec_memory_limit_bytes and then guarded on that limit being non-zero. No service in compose.yaml sets a memory limit, so cAdvisor reports 0 for all six containers and the guard excluded every one of them. The alert could not fire for any input while showing as loaded and healthy on the status page — the failure ups.rules.yaml warns about at length. It now measures against machine_memory_bytes, which needs no per-service tuning and, on a single-host stack, is the real budget anyway. Note the join is load-bearing: the obvious `/ machine_memory_bytes` returns no series at all, because machine_memory_bytes carries boot_id, machine_id and system_uuid while the container series carry name, id and image. The on(instance) join is what makes it match, and group_left is mandatory — six containers to one machine series is many-to-one, which Prometheus rejects unless the grouping is explicit. Threshold is 0.40 of host. The whole stack runs at ~9.6% today, so this is a single-container runaway detector; HostMemoryPressure still owns the aggregate case, and this one fires earlier and names the culprit. promtool check rules had passed this the entire time, because it parses PromQL and never asks whether an expression can be true. So this adds the first promtool test rules unit tests in the repo, wired into make check-rules, validate.sh and CI. They were mutation-tested: the suite fails against the old expression, the naive rewrite, a dropped group_left, a wrong join key, and a retuned threshold. Rule counts in the docs were already stale before this change (32+8 vs an actual 34+13), as was the panel count (79 vs 84); both corrected. Closes #63
4 tasks
Owner
Author
|
Follow-up for the other half of #63's "two ways out" is now #114 — memory limits on the six services, kept deliberately independent of this rule. Worth noting what came out of scoping it: |
#114 is the other half of #63's "two ways out". It goes in Monitoring rather than Infrastructure because it is about the observability stack, not the rack, and it records why it is not simply actionable: cAdvisor has only reported correctly since #62, so there are hours of history to size a limit from rather than the 30 days retention implies. The #63 paragraph now points at it, since "so should we set limits?" is the next question a reader has.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #63.
What was wrong
The rule divided by
container_spec_memory_limit_bytesand guarded on thatlimit being non-zero. No service in
compose.yamlsets a memory limit, so theguard excluded every container and the alert could not fire for any input —
while showing as loaded and healthy on the status page.
Two corrections to the issue's write-up, both verified against live Prometheus:
container_spec_memory_limit_bytesreturns 7 series, not 0. Six namedcontainers all report value
0; the root cgroup reports the host total. The> 0guard, not a missing metric, is what made the rule unreachable.container_memory_working_set_bytes / machine_memory_bytesreturns 0series —
machine_memory_bytescarriesboot_id/machine_id/system_uuidand the container series carry
name/id/image, so nothing matches. Takingit verbatim would have swapped one unfireable rule for another.
The fix
group_leftis mandatory, not stylistic: six containers to one machine seriesis many-to-one, which Prometheus rejects outright without explicit grouping.
Threshold 0.40 of host (~3 GB). The whole stack runs at ~9.6% today, so this is
a single-container runaway detector —
HostMemoryPressurestill owns theaggregate case. This one fires earlier and names the culprit, which is the
attribution the host-level rule cannot give.
Chose this over setting
mem_limiton the six services because a limit and analert answer different questions: making detection depend on enforcement is what
created #63 in the first place. Setting limits stays worthwhile for blast radius
and is now independent of this rule working.
Regression guard
promtool check ruleshad passed this the entire time — it parses PromQL andnever asks whether an expression can be true. So this adds the repo's first
promtool test rulesunit tests, wired intomake check-rules,validate.shand CI.
They were mutation-tested rather than assumed. The suite fails against:
/ machine_memory_bytesrewritegroup_leftdroppedon(job)for: 15mchanged to 45mNot caught: swapping
on(instance)foron(host). Under this data those areequivalent, so it's a preference rather than a break — left uncaught deliberately
rather than contorting the fixtures.
Coverage is 1 rule of 34. The other 33 are still syntax-checked only, which
is exactly the standing #63 had — stated plainly in
docs/observability.mdanddocs/roadmap.mdrather than left implied.Docs
Rule counts were already stale before this change: docs said 32 Prometheus + 8
Loki = 40, actual is 34 + 13 = 47. Panel count was likewise 79 vs an actual 84
(the repo's own
check_dashboards.pyreports it). Both corrected.docs/adr/0007and the completed-work line inroadmap.mdkeep their "40" —both are past-tense records of a decision when 40 was true, and editing them
would rewrite the record.
Verification
make validate— all checks pass (gitleaks skipped, not installed locally).ContainerHighMemoryhealthok.> 0.001the expression returns one seriesper container; at the real
> 0.40it returns zero — quiet because nothing iswrong, not because it is broken.