diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 25ffdef..1873a3f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -128,6 +128,14 @@ jobs: -v "$PWD:/repo" -w /repo "$PROM_IMAGE" \ check rules "$STACK"/prometheus/rules/*.rules.yaml + # check rules only parses PromQL. It passed for months against a rule that + # could not fire for any input (#63); these are the tests that catch that. + - name: promtool test rules + run: | + docker run --rm --entrypoint promtool \ + -v "$PWD:/repo" -w /repo "$PROM_IMAGE" \ + test rules "$STACK"/prometheus/tests/*.test.yaml + # No secret needed: the receiver URL comes from url_file, which # Alertmanager reads at notify time rather than at config load time. - name: amtool check-config diff --git a/Makefile b/Makefile index 77690f6..191aa82 100644 --- a/Makefile +++ b/Makefile @@ -117,9 +117,10 @@ check-dashboards: ## Validate dashboard JSON and datasource references python3 scripts/check_dashboards.py .PHONY: check-rules -check-rules: ## Validate Prometheus rules and config +check-rules: ## Validate and unit-test Prometheus rules and config promtool check config $(STACK_DIR)/prometheus/prometheus.yaml promtool check rules $(STACK_DIR)/prometheus/rules/*.rules.yaml + promtool test rules $(STACK_DIR)/prometheus/tests/*.test.yaml .PHONY: check-compose-health check-compose-health: ## Verify compose health dependencies can be satisfied diff --git a/README.md b/README.md index 00980fc..5fbe32f 100644 --- a/README.md +++ b/README.md @@ -44,8 +44,8 @@ incident. metrics and logs from Linux hosts; `snmp_exporter` polls the four devices that can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) -- **Dashboards and alerting as code.** 5 provisioned dashboards, 79 panels, and - 40 alert rules — 32 metric-based in Prometheus, 8 log-based in Loki — sharing +- **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and + 47 alert rules — 34 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -143,9 +143,9 @@ rack; a dashed border means egress only. Full topology and data flow in . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 32 alert rules +│ ├── prometheus/ # config, file_sd targets, 34 alert rules │ ├── alertmanager/ # routing and inhibition -│ ├── loki/ # single-binary config + 8 LogQL rules +│ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # one agent config, used on every host │ ├── snmp-exporter/ # generator.yaml is the source of truth │ └── grafana/ # provisioning + 5 dashboards diff --git a/docs/observability.md b/docs/observability.md index 98efcf1..0cc694e 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -70,7 +70,7 @@ expression in every panel is syntactically valid. ## Alerting -40 rules in total: 32 metric-based in `prometheus/rules/`, and 8 log-based in +47 rules in total: 34 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -99,14 +99,25 @@ boot check. ### Metric-based (Prometheus) -32 rules across four files in `prometheus/rules/`: +34 rules across four files in `prometheus/rules/`: | File | Covers | | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature | -| `containers.rules.yaml` | Restart loops, OOM kills, throttling, and the stack watching itself | +| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | + +`promtool check rules` validates that these parse. It does not — and cannot — +tell you whether a rule can ever be true: `ContainerHighMemory` passed it for +months while dividing by a memory limit no service sets, so it showed as loaded +and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/HomeLab/issues/63)). +`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which +feed a rule synthetic series and assert it fires — paired with a case asserting +it stays quiet, because a test that only ever expects silence would have passed +against the broken rule too. Coverage is one rule of 34 so far: +`ContainerHighMemory`. The other 33 are still validated for syntax only, which +is exactly the standing #63 had. Routing is by `severity` and `category` (see `alertmanager/alertmanager.yaml`). `critical` + `category=power` pages immediately and repeats every 30 minutes; diff --git a/docs/roadmap.md b/docs/roadmap.md index 4d80271..4e3d8bb 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -44,6 +44,14 @@ issues intact. Nothing was summarised away. - **[#91](https://github.com/Gerrrt/HomeLab/issues/91) Add blackbox-exporter** for uptime and TLS-expiry on internal services. Until it exists, expiry is something you find out about from a browser warning. +- **[#114](https://github.com/Gerrrt/HomeLab/issues/114) Set memory limits on + the six services.** Nothing in `compose.yaml` bounds a leak, so one container + can take the host down — and the host has 8 GB soldered. Blocked on data + rather than on agreement: cAdvisor has only reported correctly since + [#62](https://github.com/Gerrrt/HomeLab/pull/62), so there are hours of + history to size from, not the 30 days retention implies, and `grafana` alone + swings 3.7x inside that window. A limit picked from it would be a guess at an + OOM kill. - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard screenshots.** `make screenshots` does four of the five; the Logs dashboard is deliberately excluded. → [`images/README.md`](images/README.md) @@ -59,14 +67,26 @@ new since this file was last honest: to extract. - **[#67](https://github.com/Gerrrt/HomeLab/issues/67)** No dead man's switch on the notification path — a 200 into a dead topic is a successful notification. -- **[#63](https://github.com/Gerrrt/HomeLab/issues/63)** `ContainerHighMemory` - cannot fire, because nothing sets a memory limit. Two collection faults of the same kind were fixed in [#62](https://github.com/Gerrrt/HomeLab/pull/62): the agent was answering to the name of the server, and cAdvisor could only see its own cgroup. Both ran healthy and produced nothing. +[#63](https://github.com/Gerrrt/HomeLab/issues/63) was the same fault one layer +up. `ContainerHighMemory` divided by a memory limit no service sets and guarded +on it being non-zero, so it could not fire for any input while showing as loaded +and healthy. It now measures against the host total instead. `promtool check +rules` had passed it the entire time — it parses PromQL and never asks whether +an expression can be true — so the fix came with the first `promtool test rules` +unit tests in the repo, which fail if the rule stops being able to fire. They +cover that one rule. The other 33 are still syntax-checked only, so the same +class of fault could be sitting in any of them and would look just as healthy. +Setting the memory limits themselves is +[#114](https://github.com/Gerrrt/HomeLab/issues/114), deliberately separate: a +limit enforces, a rule detects, and making the second depend on the first is +what left this one unfireable for months. + ## Infrastructure - **[#92](https://github.com/Gerrrt/HomeLab/issues/92) Get the firewall backup @@ -159,12 +179,12 @@ months. - [x] Stand up Prometheus, Grafana, Loki, snmp-exporter and Alloy - [x] Consolidate five broken compose files into one working stack - [x] Provision Grafana datasources and dashboards from files -- [x] Add alerting (32 rules) and Alertmanager routing +- [x] Add alerting (34 rules) and Alertmanager routing - [x] Move secrets to SOPS + age - [x] Add CI: lint, config validation, secret scanning - [x] Pin every image by digest, not just tag, with drift detection in CI - [x] Add SECURITY.md with a disclosure policy and known-exposure summary -- [x] Loki alerting rules (8) for auth, SSH brute force and disk/OOM events, +- [x] Loki alerting rules (13) for auth, SSH brute force and disk/OOM events, validated in CI by booting the pinned Loki image against them - [x] Replace the CA and leaf certificates that leaked, and add tooling so issuing one is a command rather than a research project diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index a8829eb..a3455d3 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -43,7 +43,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 32 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 34 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/security.md b/docs/security.md index a45fbe6..a61844d 100644 --- a/docs/security.md +++ b/docs/security.md @@ -14,7 +14,7 @@ What this network is actually built to survive: | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only | | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | -| Losing visibility of a failure | 32 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 34 alert rules, 30 days of metrics and logs | | Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | What it explicitly does **not** defend against: a determined attacker with diff --git a/scripts/validate.sh b/scripts/validate.sh index e4bdd39..afa7340 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -108,6 +108,16 @@ if ((${#PROMTOOL[@]})); then "${PROMTOOL[@]}" check rules "${STACK}"/prometheus/rules/*.rules.yaml fail "promtool check rules" fi + + # `check rules` only parses PromQL; it cannot tell whether an expression can + # ever be true. ContainerHighMemory passed it for months while being + # unfireable (#63). The unit tests are what actually assert the rules fire. + if "${PROMTOOL[@]}" test rules "${STACK}"/prometheus/tests/*.test.yaml >/dev/null 2>&1; then + pass "promtool test rules" + else + "${PROMTOOL[@]}" test rules "${STACK}"/prometheus/tests/*.test.yaml + fail "promtool test rules" + fi else skip "no promtool and no docker daemon" fi diff --git a/stacks/observability/README.md b/stacks/observability/README.md index d7f8bc9..b615f11 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -23,7 +23,8 @@ compose.yaml all six services, one network, health-gated ordering prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed - rules/*.rules.yaml 32 alert rules across host/network/ups/containers + rules/*.rules.yaml 34 alert rules across host/network/ups/containers + tests/*.test.yaml promtool unit tests — assert the rules can fire alertmanager/ alertmanager.yaml severity + category routing, inhibition loki/loki-config.yaml single-binary, filesystem, 30-day retention @@ -33,7 +34,7 @@ snmp-exporter/ snmp.yaml generated, 14k lines, ${PLACEHOLDER} communities grafana/ provisioning/ datasources + dashboard provider - dashboards/*.json 5 dashboards, 79 panels + dashboards/*.json 5 dashboards, 84 panels ``` ## Things worth knowing before editing @@ -58,5 +59,6 @@ make validate ``` Runs `docker compose config`, `promtool check config`, `promtool check rules`, -`amtool check-config`, `alloy fmt --verify`, the dashboard checks, yamllint, -markdownlint, shellcheck and gitleaks. Same set CI runs. +`promtool test rules`, `amtool check-config`, `alloy fmt --verify`, the +dashboard checks, yamllint, markdownlint, shellcheck and gitleaks. Same set CI +runs. diff --git a/stacks/observability/prometheus/rules/containers.rules.yaml b/stacks/observability/prometheus/rules/containers.rules.yaml index 07a7fde..234497b 100644 --- a/stacks/observability/prometheus/rules/containers.rules.yaml +++ b/stacks/observability/prometheus/rules/containers.rules.yaml @@ -27,19 +27,34 @@ groups: summary: "Container {{ $labels.name }} was OOM-killed" - alert: ContainerHighMemory - # Only meaningful where a limit is actually set; the spec_memory_limit - # guard excludes unlimited containers, which report the host total. + # Measured against the host, not against a limit. The limit-based form + # this replaces could not fire: it divided by + # container_spec_memory_limit_bytes and guarded on that being > 0, but + # no service in compose.yaml sets a memory limit, so cAdvisor reports 0 + # for all six and the guard excluded every one of them. See #63. + # + # Do not "simplify" this to `/ machine_memory_bytes`. That form returns + # no series at all: machine_memory_bytes carries boot_id, machine_id and + # system_uuid, the container series carry name, id and image, so nothing + # matches and the rule silently goes unfireable again. The on(instance) + # join is what makes it work, and group_left is mandatory — six + # containers to one machine series is many-to-one, and Prometheus + # rejects that as an error unless the grouping is explicit. + # + # 0.40 of 7.6 GiB is ~3 GB in one container, against a whole-stack + # working set of ~790 MB. HostMemoryPressure in host.rules.yaml already + # catches the host running out; this one fires earlier and, unlike that + # rule, names which container is responsible. expr: | container_memory_working_set_bytes{name!=""} - / container_spec_memory_limit_bytes{name!=""} > 0.90 - and container_spec_memory_limit_bytes{name!=""} > 0 + / on(instance) group_left machine_memory_bytes > 0.40 for: 15m labels: component: containers severity: warning category: capacity annotations: - summary: "Container {{ $labels.name }} at {{ $value | humanizePercentage }} of its memory limit" + summary: "Container {{ $labels.name }} is using {{ $value | humanizePercentage }} of host memory" - alert: ContainerCpuThrottled expr: | diff --git a/stacks/observability/prometheus/tests/containers.test.yaml b/stacks/observability/prometheus/tests/containers.test.yaml new file mode 100644 index 0000000..b08b860 --- /dev/null +++ b/stacks/observability/prometheus/tests/containers.test.yaml @@ -0,0 +1,150 @@ +--- +# Unit tests for containers.rules.yaml. +# +# These exist because of #63. ContainerHighMemory sat in the rule file for +# months unable to fire for any input: it divided by +# container_spec_memory_limit_bytes and guarded on that being > 0, and nothing +# in compose.yaml sets a memory limit. `promtool check rules` passed the whole +# time — it parses PromQL, it does not ask whether an expression can ever be +# true. The rule showed as loaded and healthy on the Prometheus status page, +# which is the failure mode ups.rules.yaml warns about at length: a rule that +# cannot fire is worse than no rule. +# +# So the assertion that matters is not "the YAML is valid", it is "this +# expression produces an alert for a real input". Every case below is paired +# with a quiet case, because a test that only ever expects silence would have +# passed against the broken rule too. +# +# The firing fixture carries container_spec_memory_limit_bytes at 0 on purpose. +# That is what cAdvisor really reports for these containers, so the fixture is +# faithful to production — and the old expression still cannot fire against it, +# which is the whole finding rather than an artefact of a thin fixture. +# +# What these do NOT prove: the fixtures are synthetic, so nothing here notices +# if cAdvisor renames or drops a label. They guard the expression, not the data +# feeding it. A green run means the rule is still able to fire, not that it is +# still wired to anything. +rule_files: + - ../rules/containers.rules.yaml + +evaluation_interval: 1m + +tests: + # --- ContainerHighMemory: fires ------------------------------------------ + # 4097804288 / 8195608576 is exactly 0.5 of the host, held past for: 15m. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "4097804288x25" + - series: 'container_spec_memory_limit_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "0x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: + - exp_labels: + alertname: ContainerHighMemory + name: loki + instance: prometheus + host: prometheus + job: integrations/cadvisor + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Container loki is using 50% of host memory" + + # Not yet held long enough — for: 15m has not elapsed at 10m. + - eval_time: 10m + alertname: ContainerHighMemory + exp_alerts: [] + + # --- ContainerHighMemory: stays quiet at normal usage --------------------- + # 163912171 / 8195608576 is ~2%, which is what the stack actually runs at. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "163912171x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: [] + + # --- ContainerHighMemory: 0.40 is load-bearing ---------------------------- + # 3196287344 / 8195608576 is 39% — just under. With the 50% firing case above + # this brackets the threshold to (0.39, 0.50]. It does not pin it to exactly + # 0.40, but a drive-by retune can no longer land silently. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="prometheus",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "3196287344x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: [] + + # --- ContainerHighMemory: many containers, one machine -------------------- + # Two series on the left of the join and one on the right is many-to-one, + # which Prometheus refuses to evaluate without an explicit group_left. With a + # single container the join is 1:1 and a dropped group_left only quietly + # strips `name`; this case is what makes that error real. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "4097804288x25" + - series: 'container_memory_working_set_bytes{name="grafana",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "163912171x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: + - exp_labels: + alertname: ContainerHighMemory + name: loki + instance: prometheus + host: prometheus + job: integrations/cadvisor + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Container loki is using 50% of host memory" + + # --- ContainerHighMemory: each host against its own total ----------------- + # config.alloy is written to run on every host, so this rule has to survive a + # second agent joining. Same container name on both, different host sizes: a + # join keyed on anything the two hosts share — `job`, or an absent `host` — + # collapses them into one match group and errors instead of alerting. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="alloy",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "163912171x25" + - series: 'container_memory_working_set_bytes{name="alloy",instance="saruman",host="saruman",job="integrations/cadvisor"}' + values: "17179869184x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + - series: 'machine_memory_bytes{instance="saruman",host="saruman",job="integrations/cadvisor",boot_id="4d1a9f70",machine_id="a3c2e011"}' + values: "34359738368x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: + - exp_labels: + alertname: ContainerHighMemory + name: alloy + instance: saruman + host: saruman + job: integrations/cadvisor + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Container alloy is using 50% of host memory"