Skip to content

Set memory limits on the six services to bound the blast radius of a leak #114

Description

@Gerrrt

Follow-up from #63 / #113. Not a continuation of that fix — ContainerHighMemory
now measures against machine_memory_bytes and works whether or not this is ever
done. This is the other half of #63's "two ways out", kept separate on purpose:
a limit is an enforcement control, the alert is a detection control, and
coupling them is what made #63 unfireable in the first place.

Why

No service in compose.yaml sets mem_limit, cpus, deploy.resources,
pids_limit or ulimits. A leak in any one container can take the host down,
and the host is a 2012 MacBook Pro with 8 GB soldered (docs/hardware.md:29).
Setting limits also makes the existing ContainerOomKilled rule meaningful
per-service rather than only firing on a host-wide OOM.

The blocker: there is not enough data to choose numbers yet

This is the part worth recording before anyone starts.

cAdvisor only began reporting per-container metrics correctly after #62 fixed
cgroup: host. Measured just now, container_memory_working_set_bytes has
~6 hours of history, not the 30 days --storage.tsdb.retention.time
suggests. Peaks over that window (MiB):

service current peak (~6h) 2x peak 3x peak
grafana 159 585 1169 1754
alloy 185 187 375 562
prometheus 129 168 337 505
loki 107 152 305 458
alertmanager 21 25 51 76
snmp-exporter 17 19 37 56
total 617 1137

Host is 7816 MiB. 3x every peak is 3410 MiB — it fits comfortably, so the
constraint is not "is there room", it is "is the number right".

Two reasons the window is too short to trust:

  • grafana already swings 3.7x inside six hours (159 → 585). A limit picked
    from a quiet sample would OOM-kill it during whatever caused that.
  • Prometheus is the genuinely hard one. Its working set grows with the TSDB
    as retention fills, and it spikes during compaction and wide range queries.
    Six hours of a mostly-empty TSDB says nothing about steady state at day 30.
    It is also the worst thing on the host to OOM-kill, since it is what would
    otherwise tell you the kill happened.

Suggested approach: let the stack run undisturbed for a couple of weeks now that
collection is fixed, then size from max_over_time(...[14d]) with generous
headroom — 3x peak, not 2x — and start with the low-risk services
(snmp-exporter, alertmanager, loki, alloy) before touching prometheus
or grafana.

Two things to decide, not just numbers

  • Swap. mem_limit without memswap_limit lets a constrained container
    spill to swap rather than be killed. /swap.img is 4 GiB and unencrypted
    (docs/security.md:134, SECURITY.md:79), and the repo already treats it as
    a data-at-rest exposure. Pushing container memory toward swap makes that
    exposure more likely, so memswap_limit should be set deliberately either
    way rather than left to default.
  • A limit-relative alert is a possible complement, not a replacement. Once
    limits exist, container_spec_memory_limit_bytes becomes non-zero and a
    "near its own limit" rule becomes possible alongside the host-relative one.
    If that gets added, it needs its own promtool test rules case — ContainerHighMemory cannot fire — no service sets a memory limit #63 is
    precisely what happens when a memory rule is added without one.

Done when

  • At least 14 days of container memory history exists to size from
  • mem_limit (and a deliberate memswap_limit) on all six services, each
    with the comment style compose.yaml uses — what the number is derived
    from, not just what it is
  • Stack restarted and run through make validate, with no container
    OOM-killed and ContainerOomKilled quiet
  • docs/security.md mitigations table updated if the swap posture changes

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions