Skip to content

fix(alerts): ContainerHighMemory measured against a limit nobody set - #113

Merged
Gerrrt merged 2 commits into
mainfrom
fix/container-high-memory-unfireable
Aug 21, 2026
Merged

fix(alerts): ContainerHighMemory measured against a limit nobody set#113
Gerrrt merged 2 commits into
mainfrom
fix/container-high-memory-unfireable

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Closes #63.

What was wrong

The rule divided by container_spec_memory_limit_bytes and guarded on that
limit being non-zero. No service in compose.yaml sets a memory limit, so the
guard excluded every container and the alert could not fire for any input —
while showing as loaded and healthy on the status page.

Two corrections to the issue's write-up, both verified against live Prometheus:

  • container_spec_memory_limit_bytes returns 7 series, not 0. Six named
    containers all report value 0; the root cgroup reports the host total. The
    > 0 guard, not a missing metric, is what made the rule unreachable.
  • The rewrite the issue proposes does not work as written.
    container_memory_working_set_bytes / machine_memory_bytes returns 0
    series
    machine_memory_bytes carries boot_id/machine_id/system_uuid
    and the container series carry name/id/image, so nothing matches. Taking
    it verbatim would have swapped one unfireable rule for another.

The fix

container_memory_working_set_bytes{name!=""}
/ on(instance) group_left machine_memory_bytes > 0.40

group_left is mandatory, not stylistic: six containers to one machine series
is many-to-one, which Prometheus rejects outright without explicit grouping.

Threshold 0.40 of host (~3 GB). The whole stack runs at ~9.6% today, so this is
a single-container runaway detectorHostMemoryPressure still owns the
aggregate case. This one fires earlier and names the culprit, which is the
attribution the host-level rule cannot give.

Chose this over setting mem_limit on the six services because a limit and an
alert answer different questions: making detection depend on enforcement is what
created #63 in the first place. Setting limits stays worthwhile for blast radius
and is now independent of this rule working.

Regression guard

promtool check rules had passed this the entire time — it parses PromQL and
never asks whether an expression can be true. So this adds the repo's first
promtool test rules unit tests, wired into make check-rules, validate.sh
and CI.

They were mutation-tested rather than assumed. The suite fails against:

mutation caught
the old limit-based expression
the naive / machine_memory_bytes rewrite
group_left dropped
join key changed to on(job)
threshold retuned to 0.25 or 0.50
for: 15m changed to 45m

Not caught: swapping on(instance) for on(host). Under this data those are
equivalent, so it's a preference rather than a break — left uncaught deliberately
rather than contorting the fixtures.

Coverage is 1 rule of 34. The other 33 are still syntax-checked only, which
is exactly the standing #63 had — stated plainly in docs/observability.md and
docs/roadmap.md rather than left implied.

Docs

Rule counts were already stale before this change: docs said 32 Prometheus + 8
Loki = 40, actual is 34 + 13 = 47. Panel count was likewise 79 vs an actual 84
(the repo's own check_dashboards.py reports it). Both corrected.

docs/adr/0007 and the completed-work line in roadmap.md keep their "40" —
both are past-tense records of a decision when 40 was true, and editing them
would rewrite the record.

Verification

  • make validate — all checks pass (gitleaks skipped, not installed locally).
  • Reloaded live Prometheus: 34 rules loaded, none in error,
    ContainerHighMemory health ok.
  • Reachability proven directly: at > 0.001 the expression returns one series
    per container; at the real > 0.40 it returns zero — quiet because nothing is
    wrong, not because it is broken.

The rule divided container_memory_working_set_bytes by
container_spec_memory_limit_bytes and then guarded on that limit being
non-zero. No service in compose.yaml sets a memory limit, so cAdvisor
reports 0 for all six containers and the guard excluded every one of
them. The alert could not fire for any input while showing as loaded and
healthy on the status page — the failure ups.rules.yaml warns about at
length.

It now measures against machine_memory_bytes, which needs no per-service
tuning and, on a single-host stack, is the real budget anyway. Note the
join is load-bearing: the obvious `/ machine_memory_bytes` returns no
series at all, because machine_memory_bytes carries boot_id, machine_id
and system_uuid while the container series carry name, id and image. The
on(instance) join is what makes it match, and group_left is mandatory —
six containers to one machine series is many-to-one, which Prometheus
rejects unless the grouping is explicit.

Threshold is 0.40 of host. The whole stack runs at ~9.6% today, so this
is a single-container runaway detector; HostMemoryPressure still owns the
aggregate case, and this one fires earlier and names the culprit.

promtool check rules had passed this the entire time, because it parses
PromQL and never asks whether an expression can be true. So this adds the
first promtool test rules unit tests in the repo, wired into make
check-rules, validate.sh and CI. They were mutation-tested: the suite
fails against the old expression, the naive rewrite, a dropped
group_left, a wrong join key, and a retuned threshold.

Rule counts in the docs were already stale before this change (32+8 vs an
actual 34+13), as was the panel count (79 vs 84); both corrected.

Closes #63
@Gerrrt

Gerrrt commented Aug 21, 2026

Copy link
Copy Markdown
Owner Author

Follow-up for the other half of #63's "two ways out" is now #114 — memory limits on the six services, kept deliberately independent of this rule.

Worth noting what came out of scoping it: container_memory_working_set_bytes only has ~6 hours of history, not 30 days, because cAdvisor was only fixed in #62. So the numbers needed to size a limit safely don't exist yet — grafana alone swings 3.7x inside that window. #114 records that as the blocker rather than guessing at values.

#114 is the other half of #63's "two ways out". It goes in Monitoring
rather than Infrastructure because it is about the observability stack,
not the rack, and it records why it is not simply actionable: cAdvisor
has only reported correctly since #62, so there are hours of history to
size a limit from rather than the 30 days retention implies.

The #63 paragraph now points at it, since "so should we set limits?" is
the next question a reader has.
@Gerrrt
Gerrrt merged commit 02a87ea into main Aug 21, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the fix/container-high-memory-unfireable branch August 21, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ContainerHighMemory cannot fire — no service sets a memory limit

1 participant