diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index ecb2213..b4e1f4f 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -287,13 +287,16 @@ started on the result; §4 was run against it, including the negative assertions **What it found, which per-volume testing had not.** -- **Grafana will not start without internet access.** `GF_INSTALL_PLUGINS` makes - the background installer contact `grafana.com` on every start, and a failure - there is fatal — Grafana crash-loops, even though both plugins are already - present in the restored volume. On a host that has lost its uplink, which is a - perfectly ordinary disaster, the restore succeeds and Grafana still will not - come up. Clearing that variable is the workaround; the plugins in the volume - are used regardless. +- **Grafana would not start without internet access.** `GF_INSTALL_PLUGINS` + made the background installer contact `grafana.com` on every start, and a + failure there was fatal — Grafana crash-looped, even though both plugins were + already present in the restored volume. On a host that has lost its uplink, + which is a perfectly ordinary disaster, the restore succeeded and Grafana + still would not come up. Neither plugin was used by any dashboard, and one of + them was an Angular plugin this Grafana refuses to load anyway, so the + declaration was removed rather than repaired. Grafana's own bundled apps ship + inside the image and need no network, which is why an offline start works now + — verified on both fresh and restored volumes. - **Alloy replays, and now there is a number for it.** Restoring `alloy-data` put the log positions back to their offsets at the stamp, and Alloy re-read from there and re-shipped the lines with their original timestamps — about diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 83bd7f3..d966e2b 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -180,7 +180,28 @@ services: GF_AUTH_ANONYMOUS_ENABLED: "false" GF_ANALYTICS_REPORTING_ENABLED: "false" GF_ANALYTICS_CHECK_FOR_UPDATES: "false" - GF_INSTALL_PLUGINS: grafana-clock-panel,grafana-piechart-panel + # No plugins are installed here, deliberately. This was + # GF_INSTALL_PLUGINS: grafana-clock-panel,grafana-piechart-panel and both + # were dead weight — no dashboard under grafana/dashboards/ uses either + # panel type, and grafana-piechart-panel is an Angular plugin, which this + # Grafana refuses to initialise: it was downloaded and then rejected on + # every single start. + # + # It was also an availability bug, which is the reason it is gone rather + # than merely tidied. The background installer contacts grafana.com on + # every start, and a failure there is FATAL to the process — Grafana + # crash-looped on a host with no uplink, even though both plugins were + # already present in grafana-data. That turns an ordinary outage into a + # failed disaster recovery, and it was found by doing one: see the last + # section of docs/runbooks/restore-the-stack.md. + # + # Grafana still preinstalls its own bundled apps (pyroscope, exploretraces, + # metricsdrilldown, lokiexplore, elasticsearch). Those ship inside the + # image and need no network, which is what makes an offline start work. + # + # If a plugin is ever genuinely needed, add GF_PLUGINS_PREINSTALL — + # GF_INSTALL_PLUGINS is deprecated — and re-test the offline start before + # committing it. GF_PATHS_PROVISIONING: /etc/grafana/provisioning PROMETHEUS_URL: http://prometheus:9090 LOKI_URL: http://loki:3100