fix(alerts): ContainerHighMemory measured against a limit nobody set

.github/workflows/ci.yml

@@ -128,6 +128,14 @@ jobs:
             -v "$PWD:/repo" -w /repo "$PROM_IMAGE" \
             check rules "$STACK"/prometheus/rules/*.rules.yaml
 
+      # check rules only parses PromQL. It passed for months against a rule that
+      # could not fire for any input (#63); these are the tests that catch that.
+      - name: promtool test rules
+        run: |
+          docker run --rm --entrypoint promtool \
+            -v "$PWD:/repo" -w /repo "$PROM_IMAGE" \
+            test rules "$STACK"/prometheus/tests/*.test.yaml
+
       # No secret needed: the receiver URL comes from url_file, which
       # Alertmanager reads at notify time rather than at config load time.
       - name: amtool check-config

Makefile

@@ -117,9 +117,10 @@ check-dashboards: ## Validate dashboard JSON and datasource references
 	python3 scripts/check_dashboards.py
 
 .PHONY: check-rules
-check-rules: ## Validate Prometheus rules and config
+check-rules: ## Validate and unit-test Prometheus rules and config
 	promtool check config $(STACK_DIR)/prometheus/prometheus.yaml
 	promtool check rules $(STACK_DIR)/prometheus/rules/*.rules.yaml
+	promtool test rules $(STACK_DIR)/prometheus/tests/*.test.yaml
 
 .PHONY: check-compose-health
 check-compose-health: ## Verify compose health dependencies can be satisfied

README.md

@@ -44,8 +44,8 @@ incident.
   metrics and logs from Linux hosts; `snmp_exporter` polls the four devices that
   can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
   identically everywhere. [How](docs/architecture.md#observability-data-flow)
-- **Dashboards and alerting as code.** 5 provisioned dashboards, 79 panels, and
-  40 alert rules — 32 metric-based in Prometheus, 8 log-based in Loki — sharing
+- **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and
+  47 alert rules — 34 metric-based in Prometheus, 13 log-based in Loki — sharing
   one Alertmanager routing tree. No dashboard exists only in a database.
 - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
   decrypted at deploy time into gitignored paths, with `git log` showing which
@@ -143,9 +143,9 @@ rack; a dashed border means egress only. Full topology and data flow in
 .
 ├── stacks/observability/     # the deployed stack — one compose file, six services
 │   ├── compose.yaml
-│   ├── prometheus/           # config, file_sd targets, 32 alert rules
+│   ├── prometheus/           # config, file_sd targets, 34 alert rules
 │   ├── alertmanager/         # routing and inhibition
-│   ├── loki/                 # single-binary config + 8 LogQL rules
+│   ├── loki/                 # single-binary config + 13 LogQL rules
 │   ├── alloy/                # one agent config, used on every host
 │   ├── snmp-exporter/        # generator.yaml is the source of truth
 │   └── grafana/              # provisioning + 5 dashboards

docs/observability.md

@@ -70,7 +70,7 @@ expression in every panel is syntactically valid.
 
 ## Alerting
 
-40 rules in total: 32 metric-based in `prometheus/rules/`, and 8 log-based in
+47 rules in total: 34 metric-based in `prometheus/rules/`, and 13 log-based in
 `loki/rules/`.
 
 ### Log-based (Loki ruler)
@@ -99,14 +99,25 @@ boot check.
 
 ### Metric-based (Prometheus)
 
-32 rules across four files in `prometheus/rules/`:
+34 rules across four files in `prometheus/rules/`:
 
 | File | Covers |
 | --- | --- |
 | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots |
 | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware |
 | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature |
-| `containers.rules.yaml` | Restart loops, OOM kills, throttling, and the stack watching itself |
+| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself |
+
+`promtool check rules` validates that these parse. It does not — and cannot —
+tell you whether a rule can ever be true: `ContainerHighMemory` passed it for
+months while dividing by a memory limit no service sets, so it showed as loaded
+and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/HomeLab/issues/63)).
+`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
+feed a rule synthetic series and assert it fires — paired with a case asserting
+it stays quiet, because a test that only ever expects silence would have passed
+against the broken rule too. Coverage is one rule of 34 so far:
+`ContainerHighMemory`. The other 33 are still validated for syntax only, which
+is exactly the standing #63 had.
 
 Routing is by `severity` and `category` (see `alertmanager/alertmanager.yaml`).
 `critical` + `category=power` pages immediately and repeats every 30 minutes;

docs/roadmap.md

@@ -44,6 +44,14 @@ issues intact. Nothing was summarised away.
 - **[#91](https://github.com/Gerrrt/HomeLab/issues/91) Add blackbox-exporter**
   for uptime and TLS-expiry on internal services. Until it exists, expiry is
   something you find out about from a browser warning.
+- **[#114](https://github.com/Gerrrt/HomeLab/issues/114) Set memory limits on
+  the six services.** Nothing in `compose.yaml` bounds a leak, so one container
+  can take the host down — and the host has 8 GB soldered. Blocked on data
+  rather than on agreement: cAdvisor has only reported correctly since
+  [#62](https://github.com/Gerrrt/HomeLab/pull/62), so there are hours of
+  history to size from, not the 30 days retention implies, and `grafana` alone
+  swings 3.7x inside that window. A limit picked from it would be a guess at an
+  OOM kill.
 - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard
   screenshots.** `make screenshots` does four of the five; the Logs dashboard is
   deliberately excluded. → [`images/README.md`](images/README.md)
@@ -59,14 +67,26 @@ new since this file was last honest:
   to extract.
 - **[#67](https://github.com/Gerrrt/HomeLab/issues/67)** No dead man's switch on
   the notification path — a 200 into a dead topic is a successful notification.
-- **[#63](https://github.com/Gerrrt/HomeLab/issues/63)** `ContainerHighMemory`
-  cannot fire, because nothing sets a memory limit.
 
 Two collection faults of the same kind were fixed in
 [#62](https://github.com/Gerrrt/HomeLab/pull/62): the agent was answering to the
 name of the server, and cAdvisor could only see its own cgroup. Both ran healthy
 and produced nothing.
 
+[#63](https://github.com/Gerrrt/HomeLab/issues/63) was the same fault one layer
+up. `ContainerHighMemory` divided by a memory limit no service sets and guarded
+on it being non-zero, so it could not fire for any input while showing as loaded
+and healthy. It now measures against the host total instead. `promtool check
+rules` had passed it the entire time — it parses PromQL and never asks whether
+an expression can be true — so the fix came with the first `promtool test rules`
+unit tests in the repo, which fail if the rule stops being able to fire. They
+cover that one rule. The other 33 are still syntax-checked only, so the same
+class of fault could be sitting in any of them and would look just as healthy.
+Setting the memory limits themselves is
+[#114](https://github.com/Gerrrt/HomeLab/issues/114), deliberately separate: a
+limit enforces, a rule detects, and making the second depend on the first is
+what left this one unfireable for months.
+
 ## Infrastructure
 
 - **[#92](https://github.com/Gerrrt/HomeLab/issues/92) Get the firewall backup
@@ -159,12 +179,12 @@ months.
 - [x] Stand up Prometheus, Grafana, Loki, snmp-exporter and Alloy
 - [x] Consolidate five broken compose files into one working stack
 - [x] Provision Grafana datasources and dashboards from files
-- [x] Add alerting (32 rules) and Alertmanager routing
+- [x] Add alerting (34 rules) and Alertmanager routing
 - [x] Move secrets to SOPS + age
 - [x] Add CI: lint, config validation, secret scanning
 - [x] Pin every image by digest, not just tag, with drift detection in CI
 - [x] Add SECURITY.md with a disclosure policy and known-exposure summary
-- [x] Loki alerting rules (8) for auth, SSH brute force and disk/OOM events,
+- [x] Loki alerting rules (13) for auth, SSH brute force and disk/OOM events,
       validated in CI by booting the pinned Loki image against them
 - [x] Replace the CA and leaf certificates that leaked, and add tooling so
       issuing one is a command rather than a research project

docs/runbooks/deploy-stack.md

@@ -43,7 +43,7 @@ Then in the UI:
 
 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
    take up to 45 seconds on their first scrape.
-2. **Prometheus → Status → Rules.** 32 rules loaded, none in error.
+2. **Prometheus → Status → Rules.** 34 rules loaded, none in error.
 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated.
 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving.
 5. Confirm level normalisation is working — this has been silently broken

docs/security.md

@@ -14,7 +14,7 @@ What this network is actually built to survive:
 | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only |
 | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
 | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
-| Losing visibility of a failure | 32 alert rules, 30 days of metrics and logs |
+| Losing visibility of a failure | 34 alert rules, 30 days of metrics and logs |
 | Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below |
 
 What it explicitly does **not** defend against: a determined attacker with

scripts/validate.sh

@@ -108,6 +108,16 @@ if ((${#PROMTOOL[@]})); then
     "${PROMTOOL[@]}" check rules "${STACK}"/prometheus/rules/*.rules.yaml
     fail "promtool check rules"
   fi
+
+  # `check rules` only parses PromQL; it cannot tell whether an expression can
+  # ever be true. ContainerHighMemory passed it for months while being
+  # unfireable (#63). The unit tests are what actually assert the rules fire.
+  if "${PROMTOOL[@]}" test rules "${STACK}"/prometheus/tests/*.test.yaml >/dev/null 2>&1; then
+    pass "promtool test rules"
+  else
+    "${PROMTOOL[@]}" test rules "${STACK}"/prometheus/tests/*.test.yaml
+    fail "promtool test rules"
+  fi
 else
   skip "no promtool and no docker daemon"
 fi

stacks/observability/README.md

@@ -23,7 +23,8 @@ compose.yaml               all six services, one network, health-gated ordering
 prometheus/
   prometheus.yaml          scrape config; SNMP via file_sd
   targets/snmp.yaml        SNMP targets — hot-reloaded, no restart needed
-  rules/*.rules.yaml       32 alert rules across host/network/ups/containers
+  rules/*.rules.yaml       34 alert rules across host/network/ups/containers
+  tests/*.test.yaml        promtool unit tests — assert the rules can fire
 alertmanager/
   alertmanager.yaml        severity + category routing, inhibition
 loki/loki-config.yaml      single-binary, filesystem, 30-day retention
@@ -33,7 +34,7 @@ snmp-exporter/
   snmp.yaml                generated, 14k lines, ${PLACEHOLDER} communities
 grafana/
   provisioning/            datasources + dashboard provider
-  dashboards/*.json        5 dashboards, 79 panels
+  dashboards/*.json        5 dashboards, 84 panels
 ```
 
 ## Things worth knowing before editing
@@ -58,5 +59,6 @@ make validate
 ```
 
 Runs `docker compose config`, `promtool check config`, `promtool check rules`,
-`amtool check-config`, `alloy fmt --verify`, the dashboard checks, yamllint,
-markdownlint, shellcheck and gitleaks. Same set CI runs.
+`promtool test rules`, `amtool check-config`, `alloy fmt --verify`, the
+dashboard checks, yamllint, markdownlint, shellcheck and gitleaks. Same set CI
+runs.

stacks/observability/prometheus/rules/containers.rules.yaml

@@ -27,19 +27,34 @@ groups:
           summary: "Container {{ $labels.name }} was OOM-killed"
 
       - alert: ContainerHighMemory
-        # Only meaningful where a limit is actually set; the spec_memory_limit
-        # guard excludes unlimited containers, which report the host total.
+        # Measured against the host, not against a limit. The limit-based form
+        # this replaces could not fire: it divided by
+        # container_spec_memory_limit_bytes and guarded on that being > 0, but
+        # no service in compose.yaml sets a memory limit, so cAdvisor reports 0
+        # for all six and the guard excluded every one of them. See #63.
+        #
+        # Do not "simplify" this to `/ machine_memory_bytes`. That form returns
+        # no series at all: machine_memory_bytes carries boot_id, machine_id and
+        # system_uuid, the container series carry name, id and image, so nothing
+        # matches and the rule silently goes unfireable again. The on(instance)
+        # join is what makes it work, and group_left is mandatory — six
+        # containers to one machine series is many-to-one, and Prometheus
+        # rejects that as an error unless the grouping is explicit.
+        #
+        # 0.40 of 7.6 GiB is ~3 GB in one container, against a whole-stack
+        # working set of ~790 MB. HostMemoryPressure in host.rules.yaml already
+        # catches the host running out; this one fires earlier and, unlike that
+        # rule, names which container is responsible.
         expr: |
           container_memory_working_set_bytes{name!=""}
-          / container_spec_memory_limit_bytes{name!=""} > 0.90
-          and container_spec_memory_limit_bytes{name!=""} > 0
+          / on(instance) group_left machine_memory_bytes > 0.40
         for: 15m
         labels:
           component: containers
           severity: warning
           category: capacity
         annotations:
-          summary: "Container {{ $labels.name }} at {{ $value | humanizePercentage }} of its memory limit"
+          summary: "Container {{ $labels.name }} is using {{ $value | humanizePercentage }} of host memory"
 
       - alert: ContainerCpuThrottled
         expr: |