From f4b982436f25436b5107388f2c3c091ee61c8608 Mon Sep 17 00:00:00 2001 From: Your Name Date: Fri, 21 Aug 2026 21:37:56 +0000 Subject: [PATCH 1/2] fix(alerts): ContainerHighMemory measured against a limit nobody set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rule divided container_memory_working_set_bytes by container_spec_memory_limit_bytes and then guarded on that limit being non-zero. No service in compose.yaml sets a memory limit, so cAdvisor reports 0 for all six containers and the guard excluded every one of them. The alert could not fire for any input while showing as loaded and healthy on the status page — the failure ups.rules.yaml warns about at length. It now measures against machine_memory_bytes, which needs no per-service tuning and, on a single-host stack, is the real budget anyway. Note the join is load-bearing: the obvious `/ machine_memory_bytes` returns no series at all, because machine_memory_bytes carries boot_id, machine_id and system_uuid while the container series carry name, id and image. The on(instance) join is what makes it match, and group_left is mandatory — six containers to one machine series is many-to-one, which Prometheus rejects unless the grouping is explicit. Threshold is 0.40 of host. The whole stack runs at ~9.6% today, so this is a single-container runaway detector; HostMemoryPressure still owns the aggregate case, and this one fires earlier and names the culprit. promtool check rules had passed this the entire time, because it parses PromQL and never asks whether an expression can be true. So this adds the first promtool test rules unit tests in the repo, wired into make check-rules, validate.sh and CI. They were mutation-tested: the suite fails against the old expression, the naive rewrite, a dropped group_left, a wrong join key, and a retuned threshold. Rule counts in the docs were already stale before this change (32+8 vs an actual 34+13), as was the panel count (79 vs 84); both corrected. Closes #63 --- .github/workflows/ci.yml | 8 + Makefile | 3 +- README.md | 8 +- docs/observability.md | 17 +- docs/roadmap.md | 16 +- docs/runbooks/deploy-stack.md | 2 +- docs/security.md | 2 +- scripts/validate.sh | 10 ++ stacks/observability/README.md | 10 +- .../prometheus/rules/containers.rules.yaml | 25 ++- .../prometheus/tests/containers.test.yaml | 150 ++++++++++++++++++ 11 files changed, 228 insertions(+), 23 deletions(-) create mode 100644 stacks/observability/prometheus/tests/containers.test.yaml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 25ffdef..1873a3f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -128,6 +128,14 @@ jobs: -v "$PWD:/repo" -w /repo "$PROM_IMAGE" \ check rules "$STACK"/prometheus/rules/*.rules.yaml + # check rules only parses PromQL. It passed for months against a rule that + # could not fire for any input (#63); these are the tests that catch that. + - name: promtool test rules + run: | + docker run --rm --entrypoint promtool \ + -v "$PWD:/repo" -w /repo "$PROM_IMAGE" \ + test rules "$STACK"/prometheus/tests/*.test.yaml + # No secret needed: the receiver URL comes from url_file, which # Alertmanager reads at notify time rather than at config load time. - name: amtool check-config diff --git a/Makefile b/Makefile index 77690f6..191aa82 100644 --- a/Makefile +++ b/Makefile @@ -117,9 +117,10 @@ check-dashboards: ## Validate dashboard JSON and datasource references python3 scripts/check_dashboards.py .PHONY: check-rules -check-rules: ## Validate Prometheus rules and config +check-rules: ## Validate and unit-test Prometheus rules and config promtool check config $(STACK_DIR)/prometheus/prometheus.yaml promtool check rules $(STACK_DIR)/prometheus/rules/*.rules.yaml + promtool test rules $(STACK_DIR)/prometheus/tests/*.test.yaml .PHONY: check-compose-health check-compose-health: ## Verify compose health dependencies can be satisfied diff --git a/README.md b/README.md index 00980fc..5fbe32f 100644 --- a/README.md +++ b/README.md @@ -44,8 +44,8 @@ incident. metrics and logs from Linux hosts; `snmp_exporter` polls the four devices that can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) -- **Dashboards and alerting as code.** 5 provisioned dashboards, 79 panels, and - 40 alert rules — 32 metric-based in Prometheus, 8 log-based in Loki — sharing +- **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and + 47 alert rules — 34 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -143,9 +143,9 @@ rack; a dashed border means egress only. Full topology and data flow in . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 32 alert rules +│ ├── prometheus/ # config, file_sd targets, 34 alert rules │ ├── alertmanager/ # routing and inhibition -│ ├── loki/ # single-binary config + 8 LogQL rules +│ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # one agent config, used on every host │ ├── snmp-exporter/ # generator.yaml is the source of truth │ └── grafana/ # provisioning + 5 dashboards diff --git a/docs/observability.md b/docs/observability.md index 98efcf1..0cc694e 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -70,7 +70,7 @@ expression in every panel is syntactically valid. ## Alerting -40 rules in total: 32 metric-based in `prometheus/rules/`, and 8 log-based in +47 rules in total: 34 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -99,14 +99,25 @@ boot check. ### Metric-based (Prometheus) -32 rules across four files in `prometheus/rules/`: +34 rules across four files in `prometheus/rules/`: | File | Covers | | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature | -| `containers.rules.yaml` | Restart loops, OOM kills, throttling, and the stack watching itself | +| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | + +`promtool check rules` validates that these parse. It does not — and cannot — +tell you whether a rule can ever be true: `ContainerHighMemory` passed it for +months while dividing by a memory limit no service sets, so it showed as loaded +and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/HomeLab/issues/63)). +`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which +feed a rule synthetic series and assert it fires — paired with a case asserting +it stays quiet, because a test that only ever expects silence would have passed +against the broken rule too. Coverage is one rule of 34 so far: +`ContainerHighMemory`. The other 33 are still validated for syntax only, which +is exactly the standing #63 had. Routing is by `severity` and `category` (see `alertmanager/alertmanager.yaml`). `critical` + `category=power` pages immediately and repeats every 30 minutes; diff --git a/docs/roadmap.md b/docs/roadmap.md index 4d80271..2695e27 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -59,14 +59,22 @@ new since this file was last honest: to extract. - **[#67](https://github.com/Gerrrt/HomeLab/issues/67)** No dead man's switch on the notification path — a 200 into a dead topic is a successful notification. -- **[#63](https://github.com/Gerrrt/HomeLab/issues/63)** `ContainerHighMemory` - cannot fire, because nothing sets a memory limit. Two collection faults of the same kind were fixed in [#62](https://github.com/Gerrrt/HomeLab/pull/62): the agent was answering to the name of the server, and cAdvisor could only see its own cgroup. Both ran healthy and produced nothing. +[#63](https://github.com/Gerrrt/HomeLab/issues/63) was the same fault one layer +up. `ContainerHighMemory` divided by a memory limit no service sets and guarded +on it being non-zero, so it could not fire for any input while showing as loaded +and healthy. It now measures against the host total instead. `promtool check +rules` had passed it the entire time — it parses PromQL and never asks whether +an expression can be true — so the fix came with the first `promtool test rules` +unit tests in the repo, which fail if the rule stops being able to fire. They +cover that one rule. The other 33 are still syntax-checked only, so the same +class of fault could be sitting in any of them and would look just as healthy. + ## Infrastructure - **[#92](https://github.com/Gerrrt/HomeLab/issues/92) Get the firewall backup @@ -159,12 +167,12 @@ months. - [x] Stand up Prometheus, Grafana, Loki, snmp-exporter and Alloy - [x] Consolidate five broken compose files into one working stack - [x] Provision Grafana datasources and dashboards from files -- [x] Add alerting (32 rules) and Alertmanager routing +- [x] Add alerting (34 rules) and Alertmanager routing - [x] Move secrets to SOPS + age - [x] Add CI: lint, config validation, secret scanning - [x] Pin every image by digest, not just tag, with drift detection in CI - [x] Add SECURITY.md with a disclosure policy and known-exposure summary -- [x] Loki alerting rules (8) for auth, SSH brute force and disk/OOM events, +- [x] Loki alerting rules (13) for auth, SSH brute force and disk/OOM events, validated in CI by booting the pinned Loki image against them - [x] Replace the CA and leaf certificates that leaked, and add tooling so issuing one is a command rather than a research project diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index a8829eb..a3455d3 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -43,7 +43,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 32 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 34 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/security.md b/docs/security.md index a45fbe6..a61844d 100644 --- a/docs/security.md +++ b/docs/security.md @@ -14,7 +14,7 @@ What this network is actually built to survive: | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only | | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | -| Losing visibility of a failure | 32 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 34 alert rules, 30 days of metrics and logs | | Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | What it explicitly does **not** defend against: a determined attacker with diff --git a/scripts/validate.sh b/scripts/validate.sh index e4bdd39..afa7340 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -108,6 +108,16 @@ if ((${#PROMTOOL[@]})); then "${PROMTOOL[@]}" check rules "${STACK}"/prometheus/rules/*.rules.yaml fail "promtool check rules" fi + + # `check rules` only parses PromQL; it cannot tell whether an expression can + # ever be true. ContainerHighMemory passed it for months while being + # unfireable (#63). The unit tests are what actually assert the rules fire. + if "${PROMTOOL[@]}" test rules "${STACK}"/prometheus/tests/*.test.yaml >/dev/null 2>&1; then + pass "promtool test rules" + else + "${PROMTOOL[@]}" test rules "${STACK}"/prometheus/tests/*.test.yaml + fail "promtool test rules" + fi else skip "no promtool and no docker daemon" fi diff --git a/stacks/observability/README.md b/stacks/observability/README.md index d7f8bc9..b615f11 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -23,7 +23,8 @@ compose.yaml all six services, one network, health-gated ordering prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed - rules/*.rules.yaml 32 alert rules across host/network/ups/containers + rules/*.rules.yaml 34 alert rules across host/network/ups/containers + tests/*.test.yaml promtool unit tests — assert the rules can fire alertmanager/ alertmanager.yaml severity + category routing, inhibition loki/loki-config.yaml single-binary, filesystem, 30-day retention @@ -33,7 +34,7 @@ snmp-exporter/ snmp.yaml generated, 14k lines, ${PLACEHOLDER} communities grafana/ provisioning/ datasources + dashboard provider - dashboards/*.json 5 dashboards, 79 panels + dashboards/*.json 5 dashboards, 84 panels ``` ## Things worth knowing before editing @@ -58,5 +59,6 @@ make validate ``` Runs `docker compose config`, `promtool check config`, `promtool check rules`, -`amtool check-config`, `alloy fmt --verify`, the dashboard checks, yamllint, -markdownlint, shellcheck and gitleaks. Same set CI runs. +`promtool test rules`, `amtool check-config`, `alloy fmt --verify`, the +dashboard checks, yamllint, markdownlint, shellcheck and gitleaks. Same set CI +runs. diff --git a/stacks/observability/prometheus/rules/containers.rules.yaml b/stacks/observability/prometheus/rules/containers.rules.yaml index 07a7fde..234497b 100644 --- a/stacks/observability/prometheus/rules/containers.rules.yaml +++ b/stacks/observability/prometheus/rules/containers.rules.yaml @@ -27,19 +27,34 @@ groups: summary: "Container {{ $labels.name }} was OOM-killed" - alert: ContainerHighMemory - # Only meaningful where a limit is actually set; the spec_memory_limit - # guard excludes unlimited containers, which report the host total. + # Measured against the host, not against a limit. The limit-based form + # this replaces could not fire: it divided by + # container_spec_memory_limit_bytes and guarded on that being > 0, but + # no service in compose.yaml sets a memory limit, so cAdvisor reports 0 + # for all six and the guard excluded every one of them. See #63. + # + # Do not "simplify" this to `/ machine_memory_bytes`. That form returns + # no series at all: machine_memory_bytes carries boot_id, machine_id and + # system_uuid, the container series carry name, id and image, so nothing + # matches and the rule silently goes unfireable again. The on(instance) + # join is what makes it work, and group_left is mandatory — six + # containers to one machine series is many-to-one, and Prometheus + # rejects that as an error unless the grouping is explicit. + # + # 0.40 of 7.6 GiB is ~3 GB in one container, against a whole-stack + # working set of ~790 MB. HostMemoryPressure in host.rules.yaml already + # catches the host running out; this one fires earlier and, unlike that + # rule, names which container is responsible. expr: | container_memory_working_set_bytes{name!=""} - / container_spec_memory_limit_bytes{name!=""} > 0.90 - and container_spec_memory_limit_bytes{name!=""} > 0 + / on(instance) group_left machine_memory_bytes > 0.40 for: 15m labels: component: containers severity: warning category: capacity annotations: - summary: "Container {{ $labels.name }} at {{ $value | humanizePercentage }} of its memory limit" + summary: "Container {{ $labels.name }} is using {{ $value | humanizePercentage }} of host memory" - alert: ContainerCpuThrottled expr: | diff --git a/stacks/observability/prometheus/tests/containers.test.yaml b/stacks/observability/prometheus/tests/containers.test.yaml new file mode 100644 index 0000000..b08b860 --- /dev/null +++ b/stacks/observability/prometheus/tests/containers.test.yaml @@ -0,0 +1,150 @@ +--- +# Unit tests for containers.rules.yaml. +# +# These exist because of #63. ContainerHighMemory sat in the rule file for +# months unable to fire for any input: it divided by +# container_spec_memory_limit_bytes and guarded on that being > 0, and nothing +# in compose.yaml sets a memory limit. `promtool check rules` passed the whole +# time — it parses PromQL, it does not ask whether an expression can ever be +# true. The rule showed as loaded and healthy on the Prometheus status page, +# which is the failure mode ups.rules.yaml warns about at length: a rule that +# cannot fire is worse than no rule. +# +# So the assertion that matters is not "the YAML is valid", it is "this +# expression produces an alert for a real input". Every case below is paired +# with a quiet case, because a test that only ever expects silence would have +# passed against the broken rule too. +# +# The firing fixture carries container_spec_memory_limit_bytes at 0 on purpose. +# That is what cAdvisor really reports for these containers, so the fixture is +# faithful to production — and the old expression still cannot fire against it, +# which is the whole finding rather than an artefact of a thin fixture. +# +# What these do NOT prove: the fixtures are synthetic, so nothing here notices +# if cAdvisor renames or drops a label. They guard the expression, not the data +# feeding it. A green run means the rule is still able to fire, not that it is +# still wired to anything. +rule_files: + - ../rules/containers.rules.yaml + +evaluation_interval: 1m + +tests: + # --- ContainerHighMemory: fires ------------------------------------------ + # 4097804288 / 8195608576 is exactly 0.5 of the host, held past for: 15m. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "4097804288x25" + - series: 'container_spec_memory_limit_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "0x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: + - exp_labels: + alertname: ContainerHighMemory + name: loki + instance: prometheus + host: prometheus + job: integrations/cadvisor + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Container loki is using 50% of host memory" + + # Not yet held long enough — for: 15m has not elapsed at 10m. + - eval_time: 10m + alertname: ContainerHighMemory + exp_alerts: [] + + # --- ContainerHighMemory: stays quiet at normal usage --------------------- + # 163912171 / 8195608576 is ~2%, which is what the stack actually runs at. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "163912171x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: [] + + # --- ContainerHighMemory: 0.40 is load-bearing ---------------------------- + # 3196287344 / 8195608576 is 39% — just under. With the 50% firing case above + # this brackets the threshold to (0.39, 0.50]. It does not pin it to exactly + # 0.40, but a drive-by retune can no longer land silently. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="prometheus",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "3196287344x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: [] + + # --- ContainerHighMemory: many containers, one machine -------------------- + # Two series on the left of the join and one on the right is many-to-one, + # which Prometheus refuses to evaluate without an explicit group_left. With a + # single container the join is 1:1 and a dropped group_left only quietly + # strips `name`; this case is what makes that error real. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="loki",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "4097804288x25" + - series: 'container_memory_working_set_bytes{name="grafana",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "163912171x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: + - exp_labels: + alertname: ContainerHighMemory + name: loki + instance: prometheus + host: prometheus + job: integrations/cadvisor + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Container loki is using 50% of host memory" + + # --- ContainerHighMemory: each host against its own total ----------------- + # config.alloy is written to run on every host, so this rule has to survive a + # second agent joining. Same container name on both, different host sizes: a + # join keyed on anything the two hosts share — `job`, or an absent `host` — + # collapses them into one match group and errors instead of alerting. + - interval: 1m + input_series: + - series: 'container_memory_working_set_bytes{name="alloy",instance="prometheus",host="prometheus",job="integrations/cadvisor"}' + values: "163912171x25" + - series: 'container_memory_working_set_bytes{name="alloy",instance="saruman",host="saruman",job="integrations/cadvisor"}' + values: "17179869184x25" + - series: 'machine_memory_bytes{instance="prometheus",host="prometheus",job="integrations/cadvisor",boot_id="081fcc5d",machine_id="62b6e52e"}' + values: "8195608576x25" + - series: 'machine_memory_bytes{instance="saruman",host="saruman",job="integrations/cadvisor",boot_id="4d1a9f70",machine_id="a3c2e011"}' + values: "34359738368x25" + alert_rule_test: + - eval_time: 20m + alertname: ContainerHighMemory + exp_alerts: + - exp_labels: + alertname: ContainerHighMemory + name: alloy + instance: saruman + host: saruman + job: integrations/cadvisor + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Container alloy is using 50% of host memory" From 7dbc96fc992f773cd448e3fa3d69a6d67b4e6e00 Mon Sep 17 00:00:00 2001 From: Your Name Date: Fri, 21 Aug 2026 21:44:58 +0000 Subject: [PATCH 2/2] docs(roadmap): track the memory limits as their own item #114 is the other half of #63's "two ways out". It goes in Monitoring rather than Infrastructure because it is about the observability stack, not the rack, and it records why it is not simply actionable: cAdvisor has only reported correctly since #62, so there are hours of history to size a limit from rather than the 30 days retention implies. The #63 paragraph now points at it, since "so should we set limits?" is the next question a reader has. --- docs/roadmap.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index 2695e27..4e3d8bb 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -44,6 +44,14 @@ issues intact. Nothing was summarised away. - **[#91](https://github.com/Gerrrt/HomeLab/issues/91) Add blackbox-exporter** for uptime and TLS-expiry on internal services. Until it exists, expiry is something you find out about from a browser warning. +- **[#114](https://github.com/Gerrrt/HomeLab/issues/114) Set memory limits on + the six services.** Nothing in `compose.yaml` bounds a leak, so one container + can take the host down — and the host has 8 GB soldered. Blocked on data + rather than on agreement: cAdvisor has only reported correctly since + [#62](https://github.com/Gerrrt/HomeLab/pull/62), so there are hours of + history to size from, not the 30 days retention implies, and `grafana` alone + swings 3.7x inside that window. A limit picked from it would be a guess at an + OOM kill. - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard screenshots.** `make screenshots` does four of the five; the Logs dashboard is deliberately excluded. → [`images/README.md`](images/README.md) @@ -74,6 +82,10 @@ an expression can be true — so the fix came with the first `promtool test rule unit tests in the repo, which fail if the rule stops being able to fire. They cover that one rule. The other 33 are still syntax-checked only, so the same class of fault could be sitting in any of them and would look just as healthy. +Setting the memory limits themselves is +[#114](https://github.com/Gerrrt/HomeLab/issues/114), deliberately separate: a +limit enforces, a rule detects, and making the second depend on the first is +what left this one unfireable for months. ## Infrastructure